Upcoming high-impact changes

[tgamblin]

This discussion is meant to track upcoming changes to Spack that are likely to impact users. We'll put links to issues here that will either break existing workflows or cause major disruptions when you pull from develop.

develop is the main development branch of Spack, but many users like to stay at the bleeding edge, and this is intended to provide advance warning for those users. Note that we still recommend using either a release (more here)or a fixed commit from develop for production/large site deployments.

High-impact changes after v0.22

• Developer Certificate of Origin (DCO): Spack is becoming a Linux Foundation project and as part of this, sometime soon after v0.22, we are going to start requiring DCO signoff on Spack commits. DCO was created for the Linux Kernel, and it's used by nearly all Linux Foundation projects. Basically what that means is that your commits need a signoff trailer like this at the end:

  Signed-off-by: Full Name <email>
  

  This is an attestation by you (not your company -- just you) that you have the right to contribute the code you're committing. The full text of the certificate is at developercertificate.org.

  In practice, it is fairly simple to comply with this. GitHub has support for automatically signing off commits made in the GitHub web UI, and git commits made on the command line just need an extra git commit --signoff or git commit -s argument. We will also be adding spackbot support so that you can @spackbot signoff if you forget.

• specs: include source provenance in spec.json and package hash #32312: This updates the spec JSON and will change all new hashes, but doesn't strictly require a version bump.

• Deprecate Cray platform, based on PE modules #43796: support for platform=cray is discontinued, and any mention of platform=cray has been removed from core and packages.

High-impact changes for v0.22

• Compilers as dependencies: We are in the process of making compilers proper dependencies in Spack, and a number of changes in v0.22 will support that effort:

  • gcc-runtime: add separate package for gcc runtime libs #41104: Packages compiled with %gcc on Linux, macOS and FreeBSD now depend on a new package gcc-runtime, which contains a copy of the shared compiler runtime libraries. This fixes various issues: gcc runtime libraries can be installed and relocated when using a build cache, and when building minimal Spack-generated container images it is no longer necessary to install libgfortran, libgomp etc using the system package manager.
  • Add intel-oneapi-runtime, allow injecting virtual dependencies #42062: Packages compiled with %oneapi now depend on a new package intel-oneapi-runtime. The change is similar to gcc-runtime: add separate package for gcc runtime libs #41104, except now the runtimes can provide virtuals and compilers can inject dependencies on virtuals into compiled packages. This allows to model libraries soname compatibility and allows compilers like %oneapi to provide virtuals like sycl (that can also be provided by standalone libraries).
  • Improve hit-rate on buildcaches #43272: changes to the optimization criteria of the solver improved the hit-rate of buildcaches by a fair amount. Now the solver won't try to strictly match compilers or targets of reused specs, but has more relaxed compatibility rules. To get back the previous strict behavior users could write proper requirements, imposing the same compiler or target to all nodes. Also, to avoid reusing broken packages, Spack will discard all the %gcc and %oneapi specs that don't have the corresponding runtime as a dependency.
  • Allow reusing specs built with compilers not in config #43539: allows reusing specs built with compilers that are not in the local configuration. This is now possible, since specs will bring in their runtime. This will improve binary reuse of buildcaches, and will help with distro updates that e.g. bump the compiler version.
  • libc runtime #43190: binary compatibility on linux is now based on libc, instead of on the os tag. For each linux machine, allowed libcs are detected from the available compilers and used as an implicit external spec. Binaries with a libc having the same name and a version lesser or equal than a detected libc are considered compatible with it. Nothing changes for macOS or Windows.
  • External package detection for compilers #43464: each package that can provide a compiler is now detectable using spack external find. External packages defining compiler paths are effectively used as compilers, and spack external find -t compiler can be used as a substitute for spack compiler find.

• Improve parsing of quoted flags and variants in specs #41529: We are making some breaking changes to how Spack parses specs on the CLI in order to respect shell quoting instead of trying to fight it. If you had to write something like this on the command line (ugh):

  spack install zlib cflags=\"-O2 -g\"
  

  That will now result in an error, but you can now write what you probably expected to work in the first place:

  spack install zlib cflags="-O2 -g"
  

  Quoted can also now include special characters, so you can supply flags like:

  spack intall zlib ldflags='-Wl,-rpath=$ORIGIN/_libs'
  

  To reduce ambiguity in parsing, we now require that you not put spaces around = and == when for flags or variants.
  So, this will now result in an error where it wouldn't have before:

  spack install zlib cflags = "-O2 -g"
  

• python: always use a venv #40773: Spack has unique requirements for Python because it a) installs every package in its own independent directory and b) allows users to register external python installations that may contain their own installed packages. Some distributions like Ubuntu and Debian even change the system Python's sysconfig in ways that change the installation layout of Python packages when they are installed (e.g. with the addition of a /local prefix on Debian or Ubuntu). To isolate Spack from these and other issues, we now insert a small python-venv package in between python and packages that need to install Python code. This resolves a large number of issues we've had with Python installations.

• concretizer: update reuse: default to True #41302: We made a mistake when we set the default reuse policy to --reuse-deps instead of --reuse - we're changing it back for reasons outlined in this PR. You will need to use --reuse-deps if you want the old behavior.

• Only reuse externals when configured #41707: This ensures that external entries that make it into the database are only reused if they represent an external still configured in packages.yaml. This will ensure that changes to the configured externals are reflected immediately, rather than requiring additional steps to ensure the artifacts are no longer in the database.

• binary_distribution.py: list parent dirs in binary tarball #41773: Build cache entries created with Spack 0.22 are (forward) incompatible with Spack 0.21.0 (they cause errors on install), but are compatible with Spack 0.21.1. This is due to a backwards compatible change to the structure of the tarballs.

• spack load: remove --only argument #42120: removed the --only argument, which was deprecated in v0.21

Past changes

High-impact changes for v0.21

• zlib-api: use zlib-ng +compat by default #39358: Possibly high impact, probably not: switching from zlib to the much faster zlib-ng+compat as our default zlib-api provider. If you notice issues, please comment on the PR.
• Remove deprecated versions and packages #37895: Removal of a large number of deprecated versions and packages, especially python and geospatial packages.
• Forbid using env: as a top level environment attribute #38199: Using env: as a top-level attribute in environments was deprecated in v0.20, and is removed in this PR.
• Add virtual information on DAG edges #34821: The spec JSON format (and DB format) will be bumped, and older spack versions won't be able to read it if you've installed things with this change. Similarly, older Spack versions will not be able to use newer build caches (this has never been a guaranteed, but older versions could at least read newer caches).
• Separate concretization of build dependencies (coupled) #38447 and Make "minimal" the default duplicate strategy #39621: Now Spack can duplicate build dependencies, in cases where this is necessary (e.g. conflicting versions of cython or setuptools in a DAG)
• ASP: targets, compilers and providers soft-preferences are only global #31261: preferences for targets, compilers and providers are only global, preferences for versions are only local to packages.
• Change container labeling so that "latest" is the latest tag #40593: nightly Docker images from the develop branch will be tagged as :develop, while :latest is reserved for the latest release.

High-impact changes for v0.20

• Remove support for running with Python 2.7 #33063: Spack develop no longer runs on Python 2.7! It's been a long journey to get to this point, but we finally said goodbye.
• Python: drop EOL versions #33898: Spack will no longer build EOL versions of Python. We now only support building 3.7 and later.
• Refer to mirrors by name, path, or url #34891: spack buildcache create is migrating to use more flexible positional arguments, and use of --directory. --mirror-url, and --mirror-name is deprecated.
• Disable module generation by default #35564: tcl module file generation is turned off by default. Users should now turn it on explicitly:

  spack config add modules:default:enable:[tcl]
  

• simplify spack install behavior #35206: makes the behavior of spack install less "surprising", when an environment is active. This is not backward compatible for commands like spack install x y z or spack install --add x y z (though the previous behavior is hard to justify with some rationale).
• cmake build system: change default build type to "Release" #36679 modifies the default value of the build_type variant in CMakePackage from RelWithDebInfo to Release. This will provide both performance and space savings. The PR and linked issue include longer discussions of the trade-offs involved.
• Improve version, version range, and version list syntax and behavior #36273: the version syntax and semantic have been extended. Now an expression like @X.Y always represents a range @X.Y:X.Y, no matter the context in which it appears. The new syntax @=X.Y has been introduced to specify a single version. This change solves issues we had for a long time with selecting a specific version, but is backward incompatible in a few pathological cases.
• concretizer: don't change concrete environments without --force #37438: Spack will not change the previously concretized specs, when new specs are added to an environment already in use and Spack is asked to concretize it. To reconcretize as if there was no previous lockfile, users have to pass the --force flag explicitly to spack concretize.
• Stand-alone testing: Make recipe support and processing spack-/pytest-like #34236: the way to encode stand-alone tests has changed, and now it is more "pytest-like". Details are in the documentation. Using the old way of encoding tests is deprecated in v0.20 (but still supported with a warning). Support for running test encoded in the old way will be removed in v0.21.

High-impact changes for v0.19

• remove activate/deactivate support in favor of environments #29317: global activation/deactivation of extensions have been removed in favor of using environment views
• remove deprecated top-level module config #33828: the top level module configuration, already deprecated in v0.18, will be removed in v0.19
• remove deprecated concretization environment key #33774: the concretization key in Spack environment, already deprecated in v0.18, will be removed in v0.19
• Deprecate spack bootstrap trust/untrust #33600: the spack bootstrap trust/untrust commands are deprecated as of v0.19, in favor of spack bootstrap enable/disable
• Deprecate old YAML format for buildcaches #33707: the YAML format for buildcaches is deprecated as of v0.19 and support for it will be removed in v0.20
• "spack uninstall": don't modify env #33711: spack install in an environment will no longer add to the specs: list; you'll need to either use spack add <spec> or spack install --add <spec>. Similarly, spack uninstall will not remove specs from your environment's specs: list; you will need to use spack remove or spack uninstall --remove. This is to minimize the number of commands that modify spack.yaml, and to make it easier to install/uninstall specific specs from environments stored in repositories.
• Cray support: use linux platform for newer craype versions #29392: Spack will take advantage of new features in the CrayPE to treat Cray EX systems at newer versions like linux builds. This will reduce the likelihood of cray-specific bugs and reduce the maintenance burden going forward, and make package development for cray systems much simpler.
• Don't set LD_LIBRARY_PATH on package load #28354: spack load and spack-generated modules will no longer set LD_LIBRARY_PATH by default. You can get the old behavior in your environment or modules by running:

  spack config add modules:prefix_inspections:lib64:[LD_LIBRARY_PATH]
  spack config add modules:prefix_inspections:lib:[LD_LIBRARY_PATH]

• Deprecate Python 2 support #32615: Spack v.0.19 will be deprecating Python 2 support, and Python 2 support will be removed from Spack develop shortly after v0.19 is released.
• Enable/Disable Propagation for Compiler Flags & Variants #29761: Compiler flags will no longer propagate to dependencies by default (you will need cflags==-O3 instead of cflags=-O3)
• Format all of Spack with black #24718: This touches a lot of code and may conflict your PRs
• deprecate blacklist/whitelist in favor of include/exclude #31569: We're changing module configuration syntax slightly. You can use spack config update to automatically fix your configuration files.
• Allow packages with multiple build-systems (separate classes) #30738: The format for package recipes will change to support packages that change build-systems across versions or platforms. In the future people will need to code more than one class in a package.py. Backward compatibility with the previous style is kept as much as possible.

High-impact changes for v0.18

• Reuse by default in Spack #30396: We will be making --reuse the default concretizer behavior. This will be configurable so you can revert back using Add a top-level concretizer config scope and --fresh option #28468.
• Switch to full hash everywhere #28504: After --reuse is default, we will be switching to a "full hash" as the default package id in Spack. Spack has long used a coarser package hash that it probably should, in order to prevent redundant rebuilds. Specifically, we omit build dependencies, the hash of package.py, and the hashes of resources (tarballs, etc.) from our build identifiers. This was mainly to prevent every update of cmake or other build tools from triggering a rebuild of the full stack. With --reuse as the default, we don't have this concern any more, and we will prioritize detailed provenance for package builds. The full hash will provide this and will also avoid hash collisions that we currently see in CI.
• Allow for multiple dependencies/dependents from the same package #28673: This PR allows a given spec to depend on multiple instances of the same package, differentiated by dependency type; recording the dependency type will change the underlying storage representation
• PythonPackage: install packages with pip #27798: This PR completely changes how we install Python packages to run pip install instead of python setup.py install. It also improves bootstrapping of frontend tools (pip/build/installer) and backend tools (setuptools/flit/poetry).
• Revert "Deprecate Python 2 installations" #28411: We got some volunteers and will be reverting the change below. Spack will continue to support Python 2.7 and 3.5 for reproducibility.
• Deprecate Python 2 installations #28003: This PR deprecates versions of Python that have reached EOL (Python 2.7-3.5) and packages that require them. Note that this PR does not affect which versions of Python can be used to run Spack, it only affects which versions of Python Spack can install.

High-impact changes for v0.17

• specs: move to new spec.json format with build provenance #22845: This PR changes the spec.yaml format that Spack uses for hashes to spec.json. We're doing this for performance -- YAML is too slow for large operations, like reindexing the Spack database or creating indexes for large build caches. We expect this to speed up Spack in many ways, but it will cause the hashing algorithm to change.
• Make clingo the default solver #25502: This PR makes clingo the default solver, in preparation for 0.17.0. You may see different concretizations for some packages, and the new concretizer will need to be bootstrapped (see https://spack.io/changes-spack-v017/). You can switch back by setting concretizer: original in config.yaml, but note that the original concretizer will be deprecated in 0.17 and removed in a later version.
• Remove support for Python 2.6 #27256: Python 2.6 was deprecated in Spack v0.17.0 and support has been removed in develop -- it will not be supported in v0.18.