Allow scripted modules to declare and lazily install pip requirements #7707
Labels
make-simple-things-simple
"Make simple things simple and complex things possible" - to improve Slicer usability
Continuation of #7697; tightly coupled to #7171 and #6913.
After discussion and feedback at the weekly hangout Apr 16 (see #7697 (comment) and #7697 (comment)) I think my first issue conflated three separate tasks:
I'm interested in tackling only the first of these, and since CLI have questionable value anyway it doesn't make sense to require CLI usage to get the benefit here. Really, what we want is a way for scripted module's logic to depend on pypi packages, and only install those packages when they are needed. Doing this purely in the Python layer allows all scripted modules to benefit.
Draft implementation and features
I have a Python 3.9 draft implementation of a
LazyImportGroup
context manager, based on theLazyLoader
andlazy_import
recipe in the standard libraryimportlib
documentation.https://github.com/allemangD/lazyloader - see the doctests for
LazyImportGroup
for more details.The important principles here are:
import
statements at top-level, all dependencies inrequirements.txt
.These were not mentioned during the discussion last week, but I think they are important:
Usage requires the extension developer place a
requirements.txt
in a python package, installed viaRESOURCES
argument of the existing CMake macros. The anchor and path for thatrequirements.txt
resource are given as an argument toLazyImportGroup
context manager, wrapping top-level imports.These imports are real import statements that are detected by static analysis, and run-time magic only occurs with imports within the context manager. These imports produce proxy modules that defer install and real import until first usage:
Modules imported within a
LazyImportGroup
can safely have bare top-level imports of their dependencies. This is probably recommended usage for this tool:Above,
__init__.py
andlogic.py
can safely include their dependencies at top level, as long asCompute.py
imports them in a lazy group:Only the first attribute access of a module in a group will trigger the installation for all imports in that group. There may be multiple groups with different requirements to denote multiple feature sets, and only the used features will be installed. Each group is installed at most a single time.
The first attribute access of X or Y will install
training-requirements.txt
, and the first attribute access of Z will installinference-requirements.txt
.A fast
pip install --no-deps --dry-run --report
check is first used to ensure all the items inrequirements.txt
are satisfied. If so, no action is taken, the module is imported, and the proxy module is updated. If not, the requirements are first installed withpip install -r
.The dry-run report is used to produce a summary of packages to be installed. The Slicer implementation should show this to the user with
slicer.util.confirmOkCancelDisplay
.Not implemented in the draft, but the report and install steps could accept a
-c constraints.txt
argument to pin Slicer libraries likenumpy
,SimpleITK
, etc. Also the requirements of other groups could be inspected to detect incompatibilities with messages like: "Module X wants to install package Y which is incompatible with module Z. Proceed?"I've tested these proxy objects on plain python packages, loadable packages, and even
itk
's complex lazy-loaded metapackage. It works on all of these, although there are some caveats on object identity and safe import order. See the doctests for details.Outstanding questions
slicer.LazyImportGroup
.From prior discussions: cc @jcfr @pieper @lassoan @ebrahimebrahim
The text was updated successfully, but these errors were encountered: