Bugs (UI and feature access) from incorrectly set config paths (macOS homebrew?)

[defjaf]

Description

On homebrew python installations on macOS, with pip installation of packages:

• "Open in..." menu, extension manager and extension UI all don't show up correctly.
• Seems to be related to locations of config files.

There are also related bugs specific to:

• notebook 7 — see this issue (This has a full list of all related symptoms.)
• nbclassic — see this issue

Basically, with homebrew python, jupyter --paths should have:

• /opt/homebrew/etc/jupyterin "config" and
• /opt/homebrew/share/jupyter in "data"

This can be done by hand with JUPYTER_PATH=/opt/homebrew/share/jupyter JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter.
(or homebrew should somehow populate the default directories with the contents that it places in those locations).

Note also that the config homebrew directories that does appear in jupyter --paths — /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/etc/jupyter — does not exist. The data dir .../share/jupyter exists (perhaps only after jupyter build?) but is different from /opt/homebrew/share/jupyter.

(I don't know if this is unique to macOS — there is also linux homebrew — or indeed if this could be a broader issue, but I've only seen it for this one case.)

Reproduce

In the "base" homebrew python environment (i.e., no venv, and also not the system python), with jupyter lab:

• the "Open in..." menu does not appear
• the "Installed" extension list is empty and, for example, there is no sidebar entry for ipyparallel even when installed:

The problem disappears with JUPYTER_PATH=/opt/homebrew/share/jupyter JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter.

Expected behavior

When run with those variables set, or in a venv, these issues go away. In particular:

Context

• Operating System and version: macOS 13.5 with homebrew.
• Browser and version: Safari 16.6
• JupyterLab version: 4.0.3
• Notebook version: 7

`jupyter --paths`

config:
    /Users/jaffe/.jupyter
    /Users/jaffe/Library/Python/3.11/etc/jupyter
    /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /Users/jaffe/Library/Jupyter
    /Users/jaffe/Library/Python/3.11/share/jupyter
    /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /Users/jaffe/Library/Jupyter/runtime

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[defjaf]

Ping? This and other related issues/PRs (jupyter/notebook#6974, jupyter/notebook#7006) seem stalled. I am very happy to help, but would be need some guidance (lots of python experience, but zero experience with jupyter internals).

[defjaf]

Yet more information (but still no solutions) at jupyter/notebook#6974 (comment).

TL;DR: the "schemas" issue is due to a call with the incorrect schemas_dir, but locally fixing that still leaves the other issues unsolved unless you still set either

• JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter JUPYTER_PATH=/opt/homebrew/share/jupyter or
• JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter JUPYTER_PLATFORM_DIRS=1

[defjaf]

FWIW, and as expected, all of these errors and misbehaviours vanish for the python.org framework build (of 3.12, but I assume more generally), without requiring any special environment variables or other changes. This still lets me install packages by pip and not be wedded to using a venv, so I think I am moving back to this infrastructure from homebrew, at least for now.

[sunilshah]

Hi @defjaf

Homebrew default python3 is now python@3.12. I tried using venv as recommended on the homebrew formula page. The config paths are still problematic. Using separate virtual environments for x86_64 (Rosetta 2) and for arm64 and rebuilding JupyterLab as well as ipykernel in the appropriate environment works for jupyter lab. It still picks up system homebrew python@3.11 for Jupyter lab build. However, in the rebuilt kernel, it shows correct versions when running the notebooks.

I am considering moving to Python.org framework. For moving to python.org framework build from homebrew, what specific changes did you have to go through?

Thanks in advance.