Cleaner install

[ketch]

This replaces #140. This version does not move the fortran code in riemann, so it does not require any changes to amrclaw, geoclaw, or classic. It does have all the benefits of #140.

To test, do the following:

mkdir claw_test
cd claw_test
git clone https://github.com/clawpack/clawpack.git
cd clawpack
git remote add ketch https://github.com/ketch/clawpack.git
git fetch ketch
git checkout ketch/cleaner_install
git submodule init
git submodule update
pip install -e .

[ketch]

@mandli @rjleveque I'd like to discuss this and decide on merging it within the next week or so, so we don't forget about it. Let me know if you don't have time to review it in that timeframe.

I believe that there is nothing backward-incompatible here. It would be wise when people update for them to remove the symlinks from their old installation, or just create a new install. But as far as I can tell, leaving the old symlinks in place won't actually break anything.

[mandli]

@ketch This all makes a lot of sense to me but I have not had a chance yet to exhaustively test it.

[rjleveque]

I tried this and got as far as...

$ git submodule update
Cloning into '/Users/rjl/Downloads/claw_test_cleaner_install/clawpack/amrclaw'...
Cloning into '/Users/rjl/Downloads/claw_test_cleaner_install/clawpack/classic'...
Cloning into '/Users/rjl/Downloads/claw_test_cleaner_install/clawpack/clawutil'...
Cloning into '/Users/rjl/Downloads/claw_test_cleaner_install/clawpack/geoclaw'...
Cloning into '/Users/rjl/Downloads/claw_test_cleaner_install/clawpack/pyclaw'...
Cloning into '/Users/rjl/Downloads/claw_test_cleaner_install/clawpack/riemann'...
Cloning into '/Users/rjl/Downloads/claw_test_cleaner_install/clawpack/visclaw'...
Submodule path 'amrclaw': checked out '727d98d243c521267c927f6fe107ba6f1155597b'
Submodule path 'classic': checked out '5f5f7600f8f007dad9f7359df2033e6aa801cd1f'
Submodule path 'clawutil': checked out '4854ba1e158d9744986e5128941a062baee5ba9b'
Submodule path 'geoclaw': checked out '4fa16c9df56b515c7e085c588de6143458ebd066'

error: no such remote ref bb4488d3680471ca9f88d09c7c069cb82442e800
Fetched in submodule path 'pyclaw', but it did not contain 
bb4488d3680471ca9f88d09c7c069cb82442e800. Direct fetching of that commit failed.

[ketch]

@rjleveque Hmmmm, I tested this again in a virtualenv and couldn't reproduce that. Also, the instructions up to that point are essentially identical to those of #139, which didn't give you this error. Could you double-check that you're typing everything exactly (or just copy-paste it)?

Has anyone else had this problem?

[mandli]

I tried this and it seemed to work for me.

[rjleveque]

I tried this again from scratch and got the same error.

So I added a remote ketch in pyclaw and did git fetch ketch and then it could find that commit bb4488d3, and the git submodule update got past pyclaw and threw a similar error in riemann. Again fetching ketch solved the problem.

But the git submodule update did not seem to update visclaw, which was still an empty directory, so when I tried to pip install I got this error:

    Git development environment initialized for: amrclaw
    Git development environment initialized for: clawutil
    Git development environment initialized for: geoclaw
    Git development environment initialized for: classic
    Git development environment initialized for: pyclaw
    Git development environment initialized for: riemann
    Git development environment initialized for: visclaw
    non-existing path in 'clawpack': 'visclaw/src/python/visclaw'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/rjl/Downloads/claw_test/clawpack/setup.py", line 247, in <module>
        setup_package(setup_dict, SUBPACKAGES)
      File "/Users/rjl/Downloads/claw_test/clawpack/setup.py", line 210, in setup_package
        setup(configuration=configuration, **setup_dict)
      File "/usr/local/lib/python2.7/site-packages/numpy/distutils/core.py", line 135, in setup
        config = configuration()
      File "/Users/rjl/Downloads/claw_test/clawpack/setup.py", line 146, in configuration
        config.add_subpackage('clawpack')
      File "/usr/local/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 1001, in add_subpackage
        caller_level = 2)
      File "/usr/local/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 970, in get_subpackage
        caller_level = caller_level + 1)
      File "/usr/local/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 907, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "clawpack/setup.py", line 7, in configuration
        config.add_subpackage('visclaw',  subpackage_path='visclaw/src/python/visclaw')
      File "/usr/local/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 1001, in add_subpackage
        caller_level = 2)
      File "/usr/local/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 963, in get_subpackage
        caller_level = caller_level+1)
      File "/usr/local/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 745, in __init__
        raise ValueError("%r is not a directory" % (package_path,))
    ValueError: 'visclaw/src/python/visclaw' is not a directory

[ketch]

So I added a remote ketch

I'm glad that fixed it. I don't understand this since I've done this in a virtualenv with a fresh clone, where nothing was set up except the instructions above. In any case, this won't be an issue once this is merged.

But the git submodule update did not seem to update visclaw

I'm having a hard time imagining any way this could be related to the PR (we certainly don't mess with the submodule structure here). Maybe a flaky internet connection that timed out? What happens if you rerun git submodule init and git submodule update?

[rjleveque]

No change to the empty visclaw directory when I redo the submodule init/update.

I also tried from scratch in a new conda environment with the same behavior. I'll send you the output.

[ketch]

@rjleveque could you try this:

git clone https://github.com/clawpack/clawpack.git
cd clawpack
git remote add ketch https://github.com/ketch/clawpack.git
git fetch ketch
git checkout ketch/cleaner_install
git submodule init
cd pyclaw
git remote add ketch https://github.com/ketch/pyclaw
git fetch ketch
cd ../riemann
git remote add ketch https://github.com/ketch/riemann
git fetch ketch
cd ..
git submodule update
pip install -e .

[rjleveque]

Strange, but after doing

git clone https://github.com/clawpack/clawpack.git
cd clawpack
git remote add ketch https://github.com/ketch/clawpack.git
git fetch ketch
git checkout ketch/cleaner_install
git submodule init
cd pyclaw

the ketch remote in pyclaw already exists but points to the wrong repository (as does origin, I think):

$ pwd
/Users/rjl/D/clawpack_cleaner_install_test/clawpack/pyclaw

$ git remote -v
ketch	https://github.com/ketch/clawpack.git (fetch)
ketch	https://github.com/ketch/clawpack.git (push)
origin	https://github.com/clawpack/clawpack.git (fetch)
origin	https://github.com/clawpack/clawpack.git (push)

Shouldn't these point to pyclaw.git rather than clawpack.git.

Ditto in riemann,

$ cd ../riemann
$ pwd
/Users/rjl/D/clawpack_cleaner_install_test/clawpack/riemann

$ git remote -v
ketch	https://github.com/ketch/clawpack.git (fetch)
ketch	https://github.com/ketch/clawpack.git (push)
origin	https://github.com/clawpack/clawpack.git (fetch)
origin	https://github.com/clawpack/clawpack.git (push)

[ketch]

@rjleveque I have no idea why that would happen. It's annoying that these issues have nothing to do with this PR and seem to be problems with navigating submodules and multiple github forks.

I have pushed all three branches (in clawpack, pyclaw, and riemann) to the main repos so now you shouldn't need to add the ketch remote at all. Please try:

git clone https://github.com/clawpack/clawpack.git
cd clawpack
git checkout cleaner_install
git submodule init
git submodule update
pip install -e .

[rjleveque]

So now these commands work in the sense that no errors are generated. But the output below has me a bit mystified...

Why does it install to
/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
when I was doing all this in a conda environment in hopes that it would not mess with my system outside of this environment?

(clawpack_cleaner_install) [clawpack] como $ git submodule update
Cloning into '/Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack/amrclaw'...
Cloning into '/Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack/classic'...
Cloning into '/Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack/clawutil'...
Cloning into '/Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack/geoclaw'...
Cloning into '/Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack/pyclaw'...
Cloning into '/Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack/riemann'...
Cloning into '/Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack/visclaw'...
Submodule path 'amrclaw': checked out '727d98d243c521267c927f6fe107ba6f1155597b'
Submodule path 'classic': checked out '5f5f7600f8f007dad9f7359df2033e6aa801cd1f'
Submodule path 'clawutil': checked out '4854ba1e158d9744986e5128941a062baee5ba9b'
Submodule path 'geoclaw': checked out '4fa16c9df56b515c7e085c588de6143458ebd066'
Submodule path 'pyclaw': checked out 'bb4488d3680471ca9f88d09c7c069cb82442e800'
Submodule path 'riemann': checked out '517e72515cea46dd62f26551df332018c70c2791'
Submodule path 'visclaw': checked out '13af9565aa22574288243bf76def7e59bdfd4bd0'
(clawpack_cleaner_install) [clawpack] como $ pip install -e .
Obtaining file:///Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack
Requirement already satisfied: numpy>=1.6 in /usr/local/lib/python2.7/site-packages (from clawpack==5.5.0) (1.12.1)
Requirement already satisfied: matplotlib>=1.0.1 in /usr/local/lib/python2.7/site-packages (from clawpack==5.5.0) (2.0.0)
Requirement already satisfied: six>=1.10 in /usr/local/lib/python2.7/site-packages (from matplotlib>=1.0.1->clawpack==5.5.0) (1.10.0)
Requirement already satisfied: pytz in /usr/local/lib/python2.7/site-packages (from matplotlib>=1.0.1->clawpack==5.5.0) (2016.10)
Requirement already satisfied: pyparsing!=2.0.0,!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in /usr/local/lib/python2.7/site-packages (from matplotlib>=1.0.1->clawpack==5.5.0) (2.2.0)
Requirement already satisfied: python-dateutil in /usr/local/lib/python2.7/site-packages (from matplotlib>=1.0.1->clawpack==5.5.0) (2.6.0)
Requirement already satisfied: subprocess32 in /usr/local/lib/python2.7/site-packages (from matplotlib>=1.0.1->clawpack==5.5.0) (3.2.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python2.7/site-packages (from matplotlib>=1.0.1->clawpack==5.5.0) (0.10.0)
Requirement already satisfied: functools32 in /usr/local/lib/python2.7/site-packages (from matplotlib>=1.0.1->clawpack==5.5.0) (3.2.3.post2)
Installing collected packages: clawpack
  Found existing installation: clawpack 5.5.0
    Uninstalling clawpack-5.5.0:
      Successfully uninstalled clawpack-5.5.0
  Running setup.py develop for clawpack
Successfully installed clawpack
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.



(clawpack_cleaner_install) [clawpack] como $ whichclaw

`import clawpack` imports from:
    /Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack

The CLAW environment variable is not set
The PYTHONPATH environment variable is not set

The following directories on sys.path might contain clawpack,
and are searched in this order:
    /Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack

The following easy-install.pth files list clawpack:
    /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages 
        (points to /Users/rjl/Downloads/clawpack_cleaner_install_test/clawpack)
(clawpack_cleaner_install) [clawpack] como $ 

[ketch]

@rjleveque I haven't used conda environments much, but I don't think this is anything special about Clawpack. If you do an editable install of, say, numpy, where does it put things?

I'm guessing it's an issue with your conda environment setup, but let me know if I'm wrong.

I'm using virtualenv and this installation puts things only in the site-packages specific to my virtualenv.

[rjleveque]

Sorry for not getting back to this sooner. I think it's fine to merge but I'll let @mandli do it since there are now merge conflicts with clawpack/pyclaw#612.

I just tested this PR on an AWS EC2 instance to avoid some of the issues on my laptop. It worked fine, after installing git, numpy, etc. The one thing I had to change was the pip install command to

    pip install --user -e .

This is something we should mention in general in the docs for people installing on any machine they don't have root access to (although on AWS I probably could have also done sudo pip install -e .).

Before long we should update the dev branch of the docs with these new instructions.

I can add the full set of installation instructions for getting things running on AWS too.

All the unit tests ran in all repositories except the problem I raised in clawpack/riemann#146 that is unrelated to this PR.

[xinshengqin]

I have used pip install --user -e . on several different machines and it always works fine for me.

[ketch]

Hooray!

Suggesting that people use --user is a good idea. I've been meaning to write a PR for #138, so I'll just add that to it.

Before long we should update the dev branch of the docs with these new instructions.

I think the only change to the instructions is that there is no longer a symlink-only option, right?

[mandli]

That looks right to me.

[ketch]

I don't understand why there are still conflicts here. I merged the recent commits to pyclaw and riemann (for the HLLE solvers) into these branches. I don't know what the conflicts are or how to "resolve" them.

[ketch]

I did the following on my machine:

cd clawpack
git checkout master
git checkout -b temp
git merge cleaner_install

and I got

Auto-merging riemann
Auto-merging pyclaw
Merge made by the 'recursive' strategy.
 .f2py_f2cmap         |   1 +
 .travis.yml          |   1 -
 clawpack/__init__.py |  23 ++++++++++-
 clawpack/setup.py    |  18 ++++-----
 pyclaw               |   2 +-
 riemann              |   2 +-
 setup.py             | 143 ++++++++++++++----------------------------------------------------
 7 files changed, 63 insertions(+), 127 deletions(-)

So I'm really puzzled as to why github thinks there are conflicts. Should I just push to master?

[rjleveque]

I get the same, and gitk doesn't show any potential conflicts. So yes, seems ok to me to push back to master.