CDS with systemd spawner tries to launch jhsingle_native_proxy in the wrong env.

[mangecoeur]

Describe the bug
I followed the docs to configure CDS dashboards with the systemd spawner, but I ran into the problem that the dashboards wouldn't launch because it seems the Spawner tries to run:

/bin/bash -c cd /home/{USERNAME} && exec python3 -m jhsingle_native_proxy.main ...

without taking into account what the base environment of the hub is.

I have a setup following https://jupyterhub.readthedocs.io/en/1.2.2/installation-guide-hard.html , with a virtualenv containing jupyterhub/lab at /opt/jupyterhub/bin and a bunch of conda environments that the notebooks actually run in - ideally the command would run with the full path /opt/jupyterhub/bin/python3 instead.

The problem is also that the PATH of the jupyterhub environment is appended to $PATH in the spawner, I'm not sure why but it gives:
/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jupyterhub/bin

so the environment python gets overridden by the system python. I don't know where this is set. It's easy enough to work around by adding /opt/jupyterhub/bin to extra_paths but it's a bit ugly (adds the env path twice) and I think the approach is a bit fragile.

[danlester]

Sorry I don't have direct experience of the 'Hard Way JupyterHub' but Systemd spawner generally seems to work fine within The Littlest JupyterHub.

I would suggest seeing (or reporting) what happens for regular Jupyter Servers first. How does it seem to find the correct jupyterhub virtualenv in that case? Is a similar command line being executed?

I agree it's not clear why the path is being set back-to-front, and maybe SystemdSpawner would have some answers to that.