Package caches are not regenerated when they should, and lead to spurious failures

[e4t]

Steps to reproduce

Prepare system

$ podman run -ti --rm registry.opensuse.org/opensuse/bci/bci-init:latest
# zypper -n install wget python3-clingo patchelf patch tar gzip bzip2 lsb-release unzip update-alternatives xz zstd gpg2 gcc gcc-fortran gcc-c++
# wget -P${HOME} https://github.com/spack/spack/releases/download/v0.21.2/spack-0.21.2.tar.gz
# wget -P${HOME} https://github.com/spack/spack/releases/download/v0.22.0/spack-0.22.0.tar.gz
# cd /opt
# tar -xvzf ~/spack-0.21.2.tar.gz
# tar -xvzf ~/spack-0.22.0.tar.gz

Run 0.21.2:

# cd spack-0.21.2
# source share/spack/setup-env.sh
# spack compiler find --scope=system
# spack --debug spec zsh
==> [2024-05-17-06:43:59.419174] Reading config from file /opt/spack-0.21.2/etc/spack/defaults/config.yaml
==> [2024-05-17-06:43:59.513340] Imported spec from built-in commands
==> [2024-05-17-06:43:59.514642] Imported spec from built-in commands
==> [2024-05-17-06:43:59.517096] DATABASE LOCK TIMEOUT: 60s
==> [2024-05-17-06:43:59.517200] PACKAGE LOCK TIMEOUT: No timeout
==> [2024-05-17-06:43:59.529306] Reading config from file /opt/spack-0.21.2/etc/spack/defaults/concretizer.yaml
==> [2024-05-17-06:43:59.554707] Reading config from file /opt/spack-0.21.2/etc/spack/defaults/repos.yaml
==> [2024-05-17-06:43:59.835884] Reading config from file /opt/spack-0.21.2/etc/spack/defaults/packages.yaml
==> [2024-05-17-06:44:56.004307] Reading config from file /opt/spack-0.21.2/etc/spack/defaults/mirrors.yaml
==> [2024-05-17-06:44:56.059307] Reading config from file /etc/spack/compilers.yaml
==> [2024-05-17-06:45:03.458013] UNKNOWN SYMBOL: attr("virtual_on_incoming_edges", NodeArgument(id='0', pkg='pkgconf'), pkgconfig)
Input spec
--------------------------------
 -   zsh

Concretized
[...]

Switch to 0.22.0 and observe error (full error message below)

# cd ../cd spack-0.22.0/
# source share/spack/setup-env.sh
# spack spec zsh
[... see below]
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error
    zsh is unsatisfiable

Remove cache and observe it working again:

# rm ~/.spack/cache/ -rf
# spack spec zsh
Input spec
--------------------------------
 -   zsh
[...]

Error message

# spack --debug spec zsh
==> [2024-05-17-06:45:50.294066] Reading config from file /opt/spack-0.22.0/etc/spack/defaults/config.yaml
==> [2024-05-17-06:45:50.385941] Imported spec from built-in commands
==> [2024-05-17-06:45:50.387029] Imported spec from built-in commands
==> [2024-05-17-06:45:50.388516] DATABASE LOCK TIMEOUT: 60s
==> [2024-05-17-06:45:50.388598] PACKAGE LOCK TIMEOUT: No timeout
==> [2024-05-17-06:45:50.389554] DATABASE LOCK TIMEOUT: 60s
==> [2024-05-17-06:45:50.389607] PACKAGE LOCK TIMEOUT: No timeout
==> [2024-05-17-06:45:50.390124] Reading config from file /opt/spack-0.22.0/etc/spack/defaults/concretizer.yaml
==> [2024-05-17-06:45:50.410438] Reading config from file /opt/spack-0.22.0/etc/spack/defaults/packages.yaml
==> [2024-05-17-06:45:50.507473] Reading config from file /opt/spack-0.22.0/etc/spack/defaults/linux/packages.yaml
==> [2024-05-17-06:45:50.516329] Reading config from file /opt/spack-0.22.0/etc/spack/defaults/repos.yaml
==> [2024-05-17-06:45:50.742941] Reading config from file /etc/spack/compilers.yaml
==> [2024-05-17-06:45:50.760516] '/usr/bin/gcc' '-v' '/tmp/spack-implicit-link-info5_33icp7/main.c' '-o' '/tmp/spack-implicit-link-info5_33icp7/output'
==> [2024-05-17-06:45:51.004901] DATABASE LOCK TIMEOUT: 60s
==> [2024-05-17-06:45:51.005185] PACKAGE LOCK TIMEOUT: No timeout
==> [2024-05-17-06:45:51.006795] Reading config from file /opt/spack-0.22.0/etc/spack/defaults/mirrors.yaml
==> [2024-05-17-06:45:58.165155] SolverError: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    zsh is unsatisfiable
==> [2024-05-17-06:45:58.166547] Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    zsh is unsatisfiable
Traceback (most recent call last):
  File "/opt/spack-0.22.0/lib/spack/spack/main.py", line 1069, in main
    return _main(argv)
  File "/opt/spack-0.22.0/lib/spack/spack/main.py", line 1022, in _main
    return finish_parse_and_run(parser, cmd_name, args, env_format_error)
  File "/opt/spack-0.22.0/lib/spack/spack/main.py", line 1052, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/opt/spack-0.22.0/lib/spack/spack/main.py", line 649, in _invoke_command
    return_val = command(parser, args)
  File "/opt/spack-0.22.0/lib/spack/spack/cmd/spec.py", line 101, in spec
    concretized_specs = spack.cmd.parse_specs(args.specs, concretize=True)
  File "/opt/spack-0.22.0/lib/spack/spack/cmd/__init__.py", line 177, in parse_specs
    spec.concretize(tests=tests)
  File "/opt/spack-0.22.0/lib/spack/spack/spec.py", line 2995, in concretize
    self._new_concretize(tests)
  File "/opt/spack-0.22.0/lib/spack/spack/spec.py", line 2968, in _new_concretize
    result = solver.solve([self], tests=tests, allow_deprecated=allow_deprecated)
  File "/opt/spack-0.22.0/lib/spack/spack/solver/asp.py", line 3834, in solve
    setup, specs, reuse=reusable_specs, output=output, allow_deprecated=allow_deprecated
  File "/opt/spack-0.22.0/lib/spack/spack/solver/asp.py", line 925, in solve
    result.raise_if_unsat()
  File "/opt/spack-0.22.0/lib/spack/spack/solver/asp.py", line 468, in raise_if_unsat
    raise SolverError(constraints, conflicts=conflicts)
spack.solver.asp.SolverError: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    zsh is unsatisfiable

Information on your system

When run from within 0.21.2:

• Spack: 0.21.2
• Python: 3.6.15
• Platform: linux-opensuse15-skylake
• Concretizer: clingo

When run from within 0.22.0:

• Spack: 0.22.0
• Python: 3.6.15
• Platform: linux-opensuse15-skylake
• Concretizer: clingo

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[haampie]

either we put cache in a subdir $cache_dir/v0.22, which has the same issue, just at a different granularity.

or we use content-addressed caching of all package.py files, which is slightly more expensive.