RFC: Add dev container configuration #12306

hoechenberger · 2023-12-18T16:06:44Z

👩‍💻 The experience I'm implementing here

Screen.Recording.2023-12-18.at.15.29.08.mov

📚 Background

In the past year, I moved almost all my development work into dev containers.

A development container (or dev container for short) allows you to use a container as a full-featured development environment. It can be used to run an application, to separate tools, libraries, or runtimes needed for working with a codebase, and to aid in continuous integration and testing. Dev containers can be run locally or remotely, in a private or public cloud, in a variety of supporting tools and editors.

Until recently, one of the few development workflows I had not yet transitioned to use dev containers was MNE-Python. Reason was that the interactive figures don't work well in a notebook or web browser, and I didn't want to run an X server on my host just to display the figures.

💡 Solution

I created a dev container configuration that deploys a remote resktop feature. It allows for a remote connection via a VNC client, or via the web browser. Browser-based access even works inside VS Code with the built-in Simple Browser.

The dev container runs on Debian 11, installs MNE-python via conda-forge, followed by a pip-based editable install of the version in the checked-out repository.

🎯 Scope & target audience

Provide a seamless, virtually setup-free development environment to any interested party on all supported platforms (macOS, Windows, Linux)
Allow for development via GH Codespaces.

🔮 Future plans

Use dev container as a basis to deliver containers of stable MNE releases to end-users.
Add a feature that installs FreeSurfer.
Add a feature that downloads the sample and testing datasets.

🏃‍♀️ Using the dev container

You need

Docker Desktop
VS Code with the Dev Containers extension
a copy of this repository branch

Open the repository in VS Code and click on Reopen in Container in the bottom right.

The Debian docker image and dev container features including MNE will be downloaded and installed. Lastly, an editable install of MNE will be carried out.

The VNC server will be exposed on localhost:5901 (you can connect to it with any VNC client, e.g. Screen Sharing.app on macOS), and the web server can be reached on http://localhost:6080. The password is vscode.

You can now start developing and using MNE-Python. Interactive figures will appear on the screen exposed via the VNC server.

All development dependencies are installed (incl. pre-commit hooks, pytest etc.), and the editor is set to auto-format code via Ruff on saving.

The host SSH agent's socket is transparently forwarded into the container, meaning you should be able to use SSH immediately without requiring any further configuration.

The host's git configuration is copied to the container as well, so your name and email address are set correctly, too.

✋ Limitations

I currently enforce running an AMD64 Debian image on all platforms, including Apple Silicon. The native ARM64 wouldn't work as MNE-Python cannot be installed from conda-forge on this platform, as the migration of aarch64dependencies is still ongoing. To avoid this, all deps could be potentially installed from PyPI via pip, but I haven't tried this yet.

❓Thoughts?

What do you all think?

cc @cbrnr @britta-wstnr @agramfort @sappelhoff

hoechenberger · 2023-12-18T16:07:43Z

.devcontainer/Dockerfile

@@ -0,0 +1,3 @@
+# This Dockerfile is currently only used to enforce use of x86 image on ARM64 CPUs (Apple Silicon)
+# See aarch64 migration on https://conda-forge.org/status/, which is currently blocking us.
+FROM --platform=amd64 mcr.microsoft.com/devcontainers/base:debian-11


enforcing AMD64 platform here

Why base:debian-11 instead of base:ubuntu-22.04? My undestanding is that it can be very painful to get some packages (latest or not) in Debian.

Ubuntu 22 and Debian 12 ship with OpenSSL 3, which dropped support for some older, insecure algorithms.

Our corporate firewall still injects some of these older certs in some places, so certain encrypted connections fail to be established. Sucks but that's how it is currently, not just for me.

All packages we need seem to be available in recent-enough versions. Is there anything particular you expect to be problematic? Remember we're not trying to provide an all-purpose dev environment here, only one specifically tailored for MNE development.

Many tools are available as dev container features too, so we're somewhat independent from the base OS anyway. And we can install binaries nova conda.

hoechenberger · 2023-12-18T16:08:05Z

.devcontainer/devcontainer.json

+      "packages": "mesa-utils,libegl1,^libxcb.*-dev,libx11-xcb-dev,libglu1-mesa-dev,libxrender-dev,libxi-dev,libxkbcommon-dev,libxkbcommon-x11-dev"
+    },


this could probably be trimmed down a little? not sure.

hoechenberger · 2023-12-18T17:01:05Z

The password for VNC access is vscode.

drammock

I like this idea / initiative! I've left mostly questions and a few comments, so I can understand better how the config works.

drammock · 2023-12-19T14:58:56Z

.devcontainer/devcontainer.json

+  "build": { "dockerfile": "Dockerfile" },
+  "containerEnv": {
+    "PYTHONNOUSERSITE": "true", // Make Python ignore the user's site-packages folder if it exists
+    "XDG_RUNTIME_DIR": "/home/vscode/.cache/xdgr"


I stumbled on this line, until I realized that "vscode" is being set as the username in the container. Will it still work if we change it to "mneuser" or similar (here and on line 131)? I.e., something that is more obviously an (arbitrary) username?

The base image I'm using sets up the vscode user by default. There's been discussions on changing this to something more generic like devcontainer, but this still remains to be implemented:

devcontainers/images#145

Changing it to a custom name like mneuser would require changes to the Dockerfile and potentially some of the features we're using; hence I'd like to refrain from doing that – there's no need to, you basically never deal with that username anyway outside of this very line here :)

I however changed the VNC password to mne in 54b159b

I may have found a workaround but I need to test it a bit first. It may make it harder for me to get the dev container to work behind our corporate firewall, where I need to inject our self-signed SSL certs via a custom feature first. I want to make sure this doesn't become even more difficult or impossible if I change the default user name here.

Ok but don't spend too much effort on it, it's a minor maintainer-facing improvement right (user will not notice)?

It's basically irrelevant to users of the container, yes

Username is now mne-user

.devcontainer/devcontainer.json

drammock · 2023-12-19T15:02:57Z

.devcontainer/devcontainer.json

+    // Zsh plugins
+    "ghcr.io/devcontainers-contrib/features/zsh-plugins:0": {
+      "plugins": "zsh-autosuggestions zsh-syntax-highlighting history-substring-search",
+      "omzPlugins": "https://github.com/zsh-users/zsh-autosuggestions https://github.com/zsh-users/zsh-syntax-highlighting"


why are these needed? Will the user ever access a terminal in the container (and if so, won't it be a BASH terminal by default?)

Yes, of course :) It's a complete development environment, and if you open a terminal, it will open a "remote" shell inside the Docker container.

The default shell is zsh with oh-my-zsh.

I'm trying to provide a nice and comfy shell experience through these plugins.

I'm trying to provide a nice and comfy shell experience through these plugins.

Another word for "comfy" is "familiar". I don't necessarily object to including zsh alongside bash but I'm dubious about making zsh the default here.

MacOS users have zsh as the default shell and zsh is very close to bash.

yes, I know that since 2019 macOS uses zsh as the default shell. And I've heard that they're similar. What I'm questioning is why tailor the experience in a linux container to feel familiar to macOS users. Sticking with BASH seems like the natural default choice, and deviation from that feels like it needs to be justified.

Both shells are first-class citizens in the MS dev containers. By default, both are shipped, hence I'm shipping both too. The plugins listed here make the zsh experience better and superior to the Bash experience. I have not yet decided which shell shall be the default one. However, I'm surprised this seems to be so controversial. VS Code clearly displays which shell is being used, and when spawning a new terminal, one can select the shell from a dropdown menu. Besides, I don't see how the choice of shell matters as long as one doesn't do shell programming; and this is an MNE dev container, no generic Linux Shell Programming dev container. Like I said, I haven't decided yet which default I think would be best; but It's quite obvious to me though that the zsh prompt looks better and the shell provides more help during interactive use (history substring search being one of those invaluable features; git support being another).

I have not yet decided which shell shall be the default one.

This is news to me; you originally set zsh as default (which I noticed at the time), then removed that setting in 7d5c6a2 after my earlier comments. It seems a bit disingenuous to now say "why is this so controversial, they're both included and I haven't chosen a default".

That said, I'm happy if both are included and it's easy (and documented) for users to switch the shell they get in their container. Ideally this would be a one-time config (not something that needs doing every time the container is launched) if that's possible

This is news to me; you originally set zsh as default (which I noticed at the time), then removed that setting in 7d5c6a2 after my earlier comments

Ah, I see!! No, this was actually a misunderstanding. The default login shell always was Bash, and the setting you're referring to controls which shell should be spawned by default when creating a new terminal without explicitly selecting a shell. This was in fact an accidental setting that I copied over from another dev container config I'm working with. But this setting never changed the (login) default shell to zsh, nor dropped bash or something :) I had however forgotten to properly set up mamba and conda for bash – something I realized and fixed after your comment / question.

That said, I'm happy if both are included and it's easy (and documented) for users to switch the shell they get in their container. Ideally this would be a one-time config (not something that needs doing every time the container is launched) if that's possible

Yep, I'm open for this! for now I'm really still trying to make things work and experimenting a bit! Let's save the heated debates for later! 😁🤗

Oh and, happy holidays to you all!!!

drammock · 2023-12-19T15:05:31Z

.devcontainer/devcontainer.json

+    // Configure properties specific to VS Code.
+    "vscode": {
+      "settings": {


seems like fertile ground for bikeshedding. Is there a way for the container to pick up the user's local settings? If not, then can you indicate which of these settings are truly necessary for the container to work, which are not necessary but clearly a good idea (e.g., don't restore port forwards?), and which are your preferences (perhaps "format on type" fits here)?

.devcontainer/devcontainer.json

drammock · 2023-12-19T15:11:47Z

.gitignore

@@ -97,6 +97,7 @@ cover
 .venv/
 venv/
 *.json
+!/.devcontainer/devcontainer.json


Suggested change

!/.devcontainer/devcontainer.json

!.devcontainer/devcontainer.json

I actually wanted to be explicit, hence the "full" path specification here

hoechenberger · 2023-12-19T16:40:43Z

Thanks for you feedback, @drammock
I will respond to your remaining comments later tonight or tomorrow

… via a feature

hoechenberger · 2024-04-09T13:53:06Z

would be good to get this "into" 1.7, but i'm not sure if i'll be able to finish it in time

i know it's devs-only but it would be nice to have a reference to this in the 1.7 stable docs

larsoner · 2024-04-09T13:55:02Z

would be good to get this "into" 1.7, but i'm not sure if i'll be able to finish it in time

In this case let's not mark it for 1.7 -- let's only mark the things that are blockers for release (which should happen this week ideally). If it gets merged in time though that's great (nothing stopping it if you can get it done)!

hoechenberger · 2024-04-09T14:11:58Z

ok I'm fine with that!

hoechenberger commented Dec 18, 2023

View reviewed changes

Add dev container configuration

edd8c9f

hoechenberger force-pushed the devcontainer branch from 9ef71dd to edd8c9f Compare December 18, 2023 16:31

hoechenberger marked this pull request as ready for review December 19, 2023 08:25

hoechenberger requested review from larsoner and drammock as code owners December 19, 2023 08:25

drammock reviewed Dec 19, 2023

View reviewed changes

hoechenberger added 12 commits December 20, 2023 12:38

Change VNC password to "mne"

54b159b

Fix bash support

7d5c6a2

Don't install git from PPA [ci skip]

533db90

Git is actually already included in the base image, no need to add it…

88fb767

… via a feature

Add more comments

c2ef194

Use custom username

4e7d48f

Add git and pip zsh plugins

c9fb83a

Merge branch 'main' into devcontainer

1d475dc

Better comments [ci skip]

7732c1b

Add comment [ci skip]

c2da3f2

Simplify formatting rules and disbale format-on-type

42714c8

Add git blame ignore revs

f758f91

hoechenberger added this to the 1.7 milestone Apr 9, 2024

larsoner modified the milestones: 1.7, 1.8 Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Add dev container configuration #12306

RFC: Add dev container configuration #12306

hoechenberger commented Dec 18, 2023 •

edited

hoechenberger Dec 18, 2023

lucasfcnunes Dec 21, 2023 •

edited

hoechenberger Dec 21, 2023 •

edited

hoechenberger Dec 18, 2023

hoechenberger commented Dec 18, 2023

drammock left a comment

drammock Dec 19, 2023

hoechenberger Dec 20, 2023 •

edited

hoechenberger Dec 21, 2023 •

edited

drammock Dec 21, 2023

hoechenberger Dec 21, 2023

hoechenberger Dec 22, 2023

drammock Dec 19, 2023

hoechenberger Dec 19, 2023 •

edited

drammock Dec 19, 2023

lucasfcnunes Dec 21, 2023 •

edited

drammock Dec 21, 2023

hoechenberger Dec 22, 2023 •

edited

drammock Dec 22, 2023

hoechenberger Dec 22, 2023 •

edited

drammock Dec 19, 2023

drammock Dec 19, 2023

hoechenberger Dec 19, 2023

hoechenberger commented Dec 19, 2023

hoechenberger commented Apr 9, 2024

larsoner commented Apr 9, 2024

hoechenberger commented Apr 9, 2024

		"packages": "mesa-utils,libegl1,^libxcb.*-dev,libx11-xcb-dev,libglu1-mesa-dev,libxrender-dev,libxi-dev,libxkbcommon-dev,libxkbcommon-x11-dev"
		},

	!/.devcontainer/devcontainer.json
	!.devcontainer/devcontainer.json

RFC: Add dev container configuration #12306

Are you sure you want to change the base?

RFC: Add dev container configuration #12306

Conversation

hoechenberger commented Dec 18, 2023 • edited

👩‍💻 The experience I'm implementing here

📚 Background

💡 Solution

🎯 Scope & target audience

🔮 Future plans

🏃‍♀️ Using the dev container

✋ Limitations

❓Thoughts?

Choose a reason for hiding this comment

lucasfcnunes Dec 21, 2023 • edited

Choose a reason for hiding this comment

hoechenberger Dec 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hoechenberger commented Dec 18, 2023

drammock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hoechenberger Dec 20, 2023 • edited

Choose a reason for hiding this comment

hoechenberger Dec 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hoechenberger Dec 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucasfcnunes Dec 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hoechenberger Dec 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hoechenberger Dec 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hoechenberger commented Dec 19, 2023

hoechenberger commented Apr 9, 2024

larsoner commented Apr 9, 2024

hoechenberger commented Apr 9, 2024

hoechenberger commented Dec 18, 2023 •

edited

lucasfcnunes Dec 21, 2023 •

edited

hoechenberger Dec 21, 2023 •

edited

hoechenberger Dec 20, 2023 •

edited

hoechenberger Dec 21, 2023 •

edited

hoechenberger Dec 19, 2023 •

edited

lucasfcnunes Dec 21, 2023 •

edited

hoechenberger Dec 22, 2023 •

edited

hoechenberger Dec 22, 2023 •

edited