Error installing clawpack

[saeedmohanna]

I tried installing clawpack using the pip command: pip install --src=$HOME/clawpack_src --user -e
git+https://github.com/clawpack/clawpack.git@v5.8.0#egg=clawpack-v5.8.0

and I get the attached error, what do I do?

When I tried installing by using the tar file method, I get the following:

I want to use geoclaw so I cannot install using the conda method.

Not sure what to do here.

[mandli]

There has been an issue with the first method you tried. Seeing as you are on a mac the second problem has to do with a weird incompatibility between clang and Python that causes an error to occur when compiling a part of PyClaw. To get around these problems I would suggest the following:

$ git clone git://github.com/clawpack/clawpack.git
$ cd clawpack
$ git submodule update
$ git submodule foreach 'git checkout master'
$ pip install -e .

I think this should work but I may have gotten a detail wrong so try it out and let us know.

[saeedmohanna]

Thanks for the prompt reply, just tried that and unfortunately the same error from the tar file method showed up.

[mandli]

Did all those steps work without error? It's important that at least the pyclaw repository is checked out to the master version.

[saeedmohanna]

every step except the "pip install -e ." command worked

[mandli]

There should be a log file that was created, can you attach that?

[saeedmohanna]

install.log

[mandli]

The best I can tell is that anaconda seems to have added two arguments:

clang-11: error: unknown argument: '-mavx5124fmaps'
clang-11: error: unknown argument: '-mavx5124vnniw'

These are GCC and x86 specific commands. We unfortunately do not have control over what gets compiled on the C side of things. The best thing I can think of doing is either trying to explicitly use GCC (install it via homebrew and make sure that the compiler is first on the path). This may not work though given that you are running through anaconda. Just to make sure, you are not on an M1 mac right?

[saeedmohanna]

I'll try that, thanks! I am not on an M1 mac.

[mandli]

Let us know if it worked!

[ketch]

You should be able to avoid the first error above by using this command line:

 pip install --src=$HOME/clawpack_src --user -e
git+https://github.com/clawpack/clawpack.git@v5.8.0#egg=clawpack-v5.8.0 --use-deprecated=legacy-resolver

[rjleveque]

My latest problem when trying to make a docker image for v5.8.2rc: I can make an image without clawpack and then when I start a container and try to use this command:

pip install --src=/${HOME}/ --user -e git+https://github.com/rjleveque/clawpack.git@v5.8.2rc#egg=clawpack

it clones everything fine but then dies with errors similar to those shown below.

The geoclaw examples compile and run just fine (after I set $PYTHONPATH appropriately), but there's some issue when trying to do a pip install (with or without the legacy-resolver)...

jovyan $ cd ~/clawpack
jovyan $ pip install -e . --use-deprecated=legacy-resolver
Obtaining file:///home/jovyan/clawpack
Requirement already satisfied: numpy>=1.14 in /opt/conda/lib/python3.7/site-packages (from clawpack==0.0.0) (1.21.1)
Requirement already satisfied: matplotlib>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from clawpack==0.0.0) (3.4.2)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=1.0.1->clawpack==0.0.0) (2.8.2)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=1.0.1->clawpack==0.0.0) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=1.0.1->clawpack==0.0.0) (1.3.1)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=1.0.1->clawpack==0.0.0) (2.4.7)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=1.0.1->clawpack==0.0.0) (8.2.0)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil>=2.7->matplotlib>=1.0.1->clawpack==0.0.0) (1.16.0)
Installing collected packages: clawpack
  Attempting uninstall: clawpack
    Found existing installation: clawpack 5.8.2
    Can't uninstall 'clawpack'. No files were found to uninstall.
  Running setup.py develop for clawpack
    ERROR: Command errored out with exit status 1:
     command: /opt/conda/bin/python3.7 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/jovyan/clawpack/setup.py'"'"'; __file__='"'"'/home/jovyan/clawpack/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/jovyan/clawpack/
    Complete output (4 lines):
    /opt/conda/compiler_compat/ld: cannot find /lib64/libpthread.so.0
    /opt/conda/compiler_compat/ld: cannot find /usr/lib64/libpthread_nonshared.a
    collect2: error: ld returned 1 exit status
    error: Command "gcc -pthread -shared -B /opt/conda/compiler_compat -L/opt/conda/lib -Wl,-rpath=/opt/conda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/pyclaw/src/pyclaw/limiters/weno/reconstruct.o -o pyclaw/src/pyclaw/limiters/weno/reconstruct.cpython-37m-x86_64-linux-gnu.so" failed with exit status 1
    ----------------------------------------
  ERROR: Can't roll back clawpack; was not uninstalled
ERROR: Command errored out with exit status 1: /opt/conda/bin/python3.7 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/jovyan/clawpack/setup.py'"'"'; __file__='"'"'/home/jovyan/clawpack/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

[mandli]

Looks like some weird incompatibility between compilers as libpthread is a pretty low level library.

[rjleveque]

Maybe in f2py?

[mandli]

f2py uses the compiler chain that the Python was built with along with whatever gfortan compiler you provide. If these compilers do not match this is the type of thing that comes up. The more puzzling thing though is that conda should be consistent about that but if there's some way that a compiler is being overridden then there could be an incompatibility.

That all being said, not finding this library is a bit crazy given the importance and low-level aspect of the library.

[ngam]

I managed to get this to install fine on m1 macs outside of conda. (Btw, I have started the process adding clawpack to conda-forge: see #210.) However, it is a little tricky to get other things to work smoothy like h5py, etc.

Eventually I got stuck on an error related to PETSC it seems:

Last login: Mon Mar  7 09:48:25 on ttys003
~$ python3
Python 3.10.2 (v3.10.2:a58ebcc701, Jan 13 2022, 14:50:16) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import clawpack
>>> clawpack.__version__
'5.8.2'
>>> clawpack.__all__
['clawutil', 'riemann', 'visclaw', 'pyclaw', 'petclaw', 'forestclaw', 'classic', 'amrclaw', 'geoclaw']
>>> import clawpack.pyclaw
>>> from clawpack.pyclaw import examples
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal memory access
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
Abort(59) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

Initially I thought this might be due to MPI issues, note MPI_Abort at the end. So I tried a full new installation of mpich and mpi4y. I then reinstalled petsc, petsc4py, and clawpack... but I still get the same error above.

~$ python3
Python 3.10.2 (v3.10.2:a58ebcc701, Jan 13 2022, 14:50:16) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mpi4py
>>> mpi4py.__version__
'3.1.3'
>>> mpi4py.get_config()
{'mpicc': '/usr/local/bin/mpicc', 'mpicxx': '/usr/local/bin/mpicxx', 'mpifort': '/usr/local/bin/mpifort', 'mpif90': '/usr/local/bin/mpif90', 'mpif77': '/usr/local/bin/mpif77'}
>>> import petsc4py
>>> petsc4py.__version__
'3.16.1'
>>> petsc4py.get_config()
{'PETSC_DIR': '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/petsc', 'PETSC_ARCH': ''}
>>> petsc4py.init()
>>> petsc4py.PETSc
<module 'petsc4py.PETSc' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/petsc4py/lib/PETSc.cpython-310-darwin.so)>

fyi fortran compiler:

~$ gfortran --version
GNU Fortran (GCC) 11.0.0 20201128 (experimental)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[ngam]

clawpack is now available on conda-forge and I have just added support for m1 macs (still not working correctly for m1)

• https://github.com/conda-forge/clawpack-feedstock
• https://anaconda.org/conda-forge/clawpack

To install: mamba create -n claw clawpack -c conda-forge or conda create -n claw clawpack -c conda-forge.

Please give it a try and test it; please and do let me know if anything could be improved.

[mandli]

I am a bit curious what the issue is as I do have an M1 and have not seen any issues in direct installation beyond a testing bug.

[ngam]

I am a bit curious what the issue is as I do have an M1 and have not seen any issues in direct installation beyond a testing bug.

Do you get this error as well? Or do you have a custom installation of petsc/petsc4py?

>>> from clawpack.pyclaw import examples
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 10 BUS: Bus Error, possibly illegal memory access
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
Abort(59) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

[mandli]

No, the only errors I get are of the form

verification function
        def advection_nu_verify(claw):
            q0=claw.frames[0].state.get_q_global()
            qfinal=claw.frames[claw.num_output_times].state.get_q_global()
            if q0 is not None and qfinal is not None:
                dx=claw.solution.domain.grid.delta[0]
                grid1d = claw.frames[0].state.grid
                grid1d.mapc2p = mapc2p_nonunif
                nx = 100
                aux = np.zeros((1,nx))
                aux[0,:] = np.diff(grid1d.p_nodes)/np.diff(grid1d.x.nodes)
                test = abs(np.sum(dx*aux[0,:]*(qfinal-q0)))
                return check_diff(expected, test, reltol=1e-4)
            else:
                return

args                 : {'use_petsc': True, 'kernel_language': 'Python', 'solver_type': 'classic', 'outdir': None}
norm of expected data: 7.817097663620487e-17
norm of test data    : 7.686993402922226e-17
test error           : 1.3010426069826053e-18
reltol  : 0.0001

This is of course not an issue and just implies we need to be more careful about testing tolerances. PETSc works fine for me using OpenMPI (I have not tested mpich). I should mention though that I am not using anything from anaconda in my setup with a preference towards homebrew and strictly what is available via pip with the Python form homebrew.

[ngam]

Interesting for two reasons:

1. I had a major headache enabling OpenMPI for the conda build --- it keeps erroring either during compilation or during a basic test.
2. I tried compiling PETSc natively and things didn't work, so I must've done something wrong in that compilation...

However, I now can say that the clawpack from conda-forge seems to be working correctly even on M1 Mac. Again, only mpich for now. In conda-forge, petsc4py isn't available for M1 Macs so I disabled it and included the appropriate compilers in case someone does pip install petsc4py so that it has a higher chance of successful compatibility.

I will try to expand the testing, potentially running the whole suite when I manage to have more time to work through the code :)

Feel free to join me as maintainers of the conda-forge redistribution of clawpack.

[fabian-kutschera]

Dear clawpack Team,

I am having problems installing clawpack. Therefore I think it makes sense to use this open issue about installation errors.

If I follow your instructions on GitHub (https://github.com/clawpack/clawpack#installing-the-python-clawpack-tools) using pip install clawpack, I obtain the following error message:

Collecting clawpack
Downloading clawpack-5.9.0.tar.gz (5.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 47.7 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Discarding https://files.pythonhosted.org/packages/7b/3a/6b727ea26bf2945946e474ce93988722bdadffe6b9c1c1ec3935ed0806ec/clawpack-5.9.0.tar.gz (from https://pypi.org/simple/clawpack/): Requested clawpack from https://files.pythonhosted.org/packages/7b/3a/6b727ea26bf2945946e474ce93988722bdadffe6b9c1c1ec3935ed0806ec/clawpack-5.9.0.tar.gz has inconsistent version: expected '5.9.0', but metadata has '0.0.0'
Downloading clawpack-5.8.2.tar.gz (5.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 66.2 MB/s eta 0:00:00
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
error: Multiple top-level packages discovered in a flat-layout: ['pyclaw', 'classic', 'visclaw', 'geoclaw', 'amrclaw', 'riemann', 'clawutil', 'clawpack'].

  To avoid accidental inclusion of unwanted files or directories,
  setuptools will not proceed with this build.
  
  If you are trying to create a single distribution with multiple packages
  on purpose, you should not rely on automatic discovery.
  Instead, consider the following options:
  
  1. set up custom discovery (`find` directive with `include` or `exclude`)
  2. use a `src-layout`
  3. explicitly set `py_modules` or `packages` with a list of names
  
  To find more information, look for "package discovery" on setuptools docs.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

If I follow your installation guide from https://www.clawpack.org/installing_pip.html#quick-installation-of-all-packages-with-pip with the command pip install --user -e \ git+https://github.com/clawpack/clawpack.git@v5.9.0#egg=clawpack-v5.9.0 \ --use-deprecated=legacy-resolver it seems to be successful first, but returns an error message after Running setup.py develop for clawpack (cf. error.md).

However, a new folder src is now visible which also has two log files included (cf. log.md (renamed from log.err due to GitHub not supporting the other file type) and install.log).

Could you give me a hint what might go wrong?

Best regards
Fabian
error.md
install.log
log.md

Edit: Check your .bashrc and make sure you do not import any intel modules/ compilers by default.