Installation issues in a conda environment: ModuleNotFoundError: No module named 'mesonpy' #238

kbarnhart · 2023-11-09T16:03:20Z

Following the instructions here yields the following error:

ModuleNotFoundError: No module named 'mesonpy'

Summary of outcome:

The issue was that the current hashes of each submodule do not point to the most up to date meson build system. When each submodule is updated to the current master branch, the problem resolves. Therefore this will be resolved when clawpack submodule management updates to more recent git hashes.

What I did:

Create a clean conda environment and attempt to install.

(base) USER@COMPUTER clawpack % git clone https://github.com/clawpack/clawpack.git
(base) USER@COMPUTER clawpack % cd clawpack
(base) USER@COMPUTER clawpack % git submodule init
(base) USER@COMPUTER clawpack % git submodule update
(base) USER@COMPUTER clawpack % conda create -n clawpack python=3.11
(base) USER@COMPUTER clawpack % conda activate clawpack
(clawpack) USER@COMPUTER clawpack % pip install --no-build-isolation -e ./

Obtaining file://PATH/clawpack
  Checking if build backend supports build_editable ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 513, in collect_root_requirements
    reqs = list(
           ^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 474, in _make_requirements_from_install_req
    cand = self._make_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 190, in _make_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
                                           ^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 318, in __init__
    super().__init__(
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 328, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 696, in prepare_editable_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 67, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/req/req_install.py", line 569, in prepare_metadata
    and self.supports_pyproject_editable()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_internal/req/req_install.py", line 247, in supports_pyproject_editable
    return "build_editable" in self.pep517_backend._supported_features()
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 153, in _supported_features
    return self._call_hook('_supported_features', {})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/clawpack/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mesonpy'

I guessed the name for a pip install and did not succeed.

(clawpack) USER@COMPUTER clawpack % pip install mesonpy
ERROR: Could not find a version that satisfies the requirement mesonpy (from versions: none)
ERROR: No matching distribution found for mesonpy

However, a quick goole indicates that I should have installed meson-python https://pypi.org/project/meson-python/

(clawpack) USER@COMPUTER clawpack % pip install pip install meson-python

Trying the install again still yields an error:

(clawpack) USER@COMPUTER clawpack % pip install --no-build-isolation -e ./

Obtaining file://PATH/clawpack
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]

      ('\x1b[31m',)meson-python: error: Could not find ninja version 1.8.2 or newer.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

So I installed ninja https://pypi.org/project/ninja/1.8.2/

(clawpack) USER@COMPUTER clawpack % pip install ninja

And tried again:

(clawpack) USER@COMPUTER clawpack % pip install --no-build-isolation -e ./

Obtaining file:/PATH/clawpack
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      + meson setup PATH/clawpack /Users/krbarnhart/krbarnhart/source/clawpack/build/cp311 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=PATH/clawpack/build/cp311/meson-python-native-file.ini
      The Meson build system
      Version: 1.2.3
      Source dir: PATH/clawpack
      Build dir: PATH/clawpack/build/cp311
      Build type: native build
      Program python3 found: YES (/opt/anaconda3/envs/clawpack/bin/python3)
      Project name: clawpack
      Project version: 5.9.2

      ../../meson.build:1:0: ERROR: Compiler gfortran cannot compile programs.

      A full log can be found at PATH/clawpack/build/cp311/meson-logs/meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Discussed with @mandli and found that the issue was that the meson build (which should know how to install meson-python and ninja) was not behaving as expected.

At this time the submodules were in the following state:

(clawpack) USER@COMPUTER clawpack % git submodule status
 4b50c2661a955639e634102fe80c38f77883aebc amrclaw (v5.3.1-847-g4b50c26)
 a27a4951a42f392cd8d1b36ed296b84f53e888e8 classic (v5.3.1-51-ga27a495)
 5aaee229da179c97ea09c00ece86b38d3d6c2b65 clawutil (v5.3.1-162-g5aaee22)
 222676970c6f12fefcc3a715eda7ac00b3c22b89 geoclaw (v5.3.1-1607-g22267697)
 c2b04786c08376258ca1b593854cdeea61b1cf4a pyclaw (v5.3.1-331-gc2b04786)
 c7a9ed0105f7d7a6817276d8d89c7c40b94fce06 riemann (v5.3.1-170-gc7a9ed0)
 32e257c83c6810fd7881c44c06956bae7b82fa9b visclaw (v5.3.1-282-g32e257c)

We fetched origin and checked out master

(clawpack) USER@COMPUTER clawpack % git submodule foreach 'git fetch origin'
(clawpack) USER@COMPUTER clawpack % git submodule foreach 'git checkout master'

At that point, I had no issues with mesonpy or ninja but still produced the same fortran compiler error.

      ../../meson.build:1:0: ERROR: Compiler gfortran cannot compile programs.

which fortran indicates I was using system gfortran. I then updated to gfortran distributed by conda forge

conda install gfortran -c conda-forge

At that point

pip install -e .

yielded success.

ketch · 2023-11-12T07:04:38Z

@dalcinl do you know what is the best way to make sure that these dependencies (meson-python and ninja) are installed automatically?

dalcinl · 2023-11-12T07:37:10Z

If this is about editable installs, then there is no automatic way, that how pip --no-build-isolation is supposed to work, it assumes the build requirements are already installed.

For the time being, I would suggest the following minor enhancements.

Add a requirements-dev.txt file at the top level with the following lines

meson-python
numpy

Change the editable install instructions to

pip install -r requirements-dev.txt
pip install --user --no-build-isolation -e .

This way, if the build requirements ever change you update both pyproject.toml and requirements-dev.txt. You could also add more stuff to requirements-dev.txt, for example spin and pytest, because developers are supposed to benefit from these tools. After all, occasional users should not be cloning repos and installing in editable mode, and just pip install clawpack from PyPI instead.

dalcinl · 2023-11-12T07:50:11Z

cf. pypa/pip#9314

It would be nice if pip learned an additional option --install-build-requirements that would install first the packages required by the build backend. This way there is no need to list build requirements in pyproject.toml and requirements.txt files.

As suggested here: clawpack#238 (comment) This will install meson-python and ninja which are needed for editable installs, as well as spin and pytest, which are useful for developers.

ketch · 2023-11-12T07:55:17Z

Thanks; I opened a PR with the changes to the code. There has been some discussion around whether to encourage non-developers to use an editable install, so I haven't updated the docs in any way yet.

dalcinl · 2023-11-12T07:56:55Z

@kbarnhart To your specific issue, the following updated pip instructions should solve your problem.

pip install meson-python ninja numpy
pip install --no-build-isolation -e .

@ketch You may need to add ninja to the requirement.txt file.

As suggested here: clawpack#238 (comment) This will install meson-python and ninja which are needed for editable installs, as well as spin and pytest, which are useful for developers.

kbarnhart · 2023-11-13T18:18:18Z

@dalcinl @ketch Thanks for the recommendations. Will try and report back in the next few days.

ketch mentioned this issue Nov 12, 2023

Add developer requirements file. #240

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation issues in a conda environment: ModuleNotFoundError: No module named 'mesonpy' #238

Installation issues in a conda environment: ModuleNotFoundError: No module named 'mesonpy' #238

kbarnhart commented Nov 9, 2023

ketch commented Nov 12, 2023

dalcinl commented Nov 12, 2023

dalcinl commented Nov 12, 2023

ketch commented Nov 12, 2023

dalcinl commented Nov 12, 2023

kbarnhart commented Nov 13, 2023

Installation issues in a conda environment: ModuleNotFoundError: No module named 'mesonpy' #238

Installation issues in a conda environment: ModuleNotFoundError: No module named 'mesonpy' #238

Comments

kbarnhart commented Nov 9, 2023

Summary of outcome:

What I did:

ketch commented Nov 12, 2023

dalcinl commented Nov 12, 2023

dalcinl commented Nov 12, 2023

ketch commented Nov 12, 2023

dalcinl commented Nov 12, 2023

kbarnhart commented Nov 13, 2023