Add support for pip install dask[jobqueue]

[Andrew-S-Rosen]

It might be worthwhile to add support for doing pip install dask[jobqueue] by adding the dependency to [project.optional-dependencies] in pyproject.toml. Additionally, it could be nice to have this tested in some integration CI run.

[fjetter]

There are many different dask extensions that could be added this way and so far we chose not to do this. So far, the optional dependencies are only covering things that are related to dask core functionality and doesn't include any deployment specific optionals.
I'm inclined to not go down that route of adding all possible projects as there are too many.

---

Regarding testing, I see the benefits of having fully integrated CI runs but I'm concerned about the cost (mostly time and developer productivity).
Most of the test suite is actually not setup to run with a distributed cluster but are only using the trivial schedulers. To enable an integration test we'd have to migrate the entire test suite to a setup that is using a LocalCluster variant (which could then be replaced by dask-jobqueue et al with their respective clusters). This is a lot of work and would likely slow down CI considerably even without accounting for the additional test runs against jobqueue, kubernetes, gateway, mpi, ...

[fjetter]

cc @jacobtomlinson who is heavily involved in development and maintenance of the various OSS deployment tools

[Andrew-S-Rosen]

I think that's fair enough! Feel free to close if you wish.

[jacobtomlinson]

I think the amount of complexity in the CI of the deployment projects is enough to put me off doing something like this. Historically we've kept projects separate in Dask in order to give agency to the maintainers of each subproject.

I'm not especially +1/-1 on adding some more extras to the dask package. But it doesn't feel like it gives that much value over using package names directly. It would also mean we end up with a circular dependency as most subprojects depend on dask which I could imagine would cause a maintenance headache from time to time.

I'm always keen to discuss ideas like this. Anything we can do to make the user experience of Dask better is worth talking about. But I dont see any compelling reasoning in this particular case to motivate making a change like this. Given @fjetter is also not positive on the idea I'm going to close this out. But thanks for starting the discussion @Andrew-S-Rosen!