v0.17.0 (2021-11-05)

v0.17.0 is a major feature release.

Major features in this release

1. New concretizer is now default
   The new concretizer introduced as an experimental feature in v0.16.0
   is now the default (#25502). The new concretizer is based on the
   clingo logic programming system,
   and it enables us to do much higher quality and faster dependency solving
   The old concretizer is still available via the concretizer: original
   setting, but it is deprecated and will be removed in v0.18.0.

2. Binary Bootstrapping
   To make it easier to use the new concretizer and binary packages,
   Spack now bootstraps clingo and GnuPG from public binaries. If it
   is not able to bootstrap them from binaries, it installs them from
   source code. With these changes, you should still be able to clone Spack
   and start using it almost immediately. (#21446, #22354, #22489, #22606,
   #22720, #22720, #23677, #23946, #24003, #25138, #25607, #25964, #26029,
   #26399, #26599).

3. Reuse existing packages (experimental)
   The most wanted feature from our
   2020 user survey and
   the most wanted Spack feature of all time (#25310). spack install,
   spack spec, and spack concretize now have a --reuse option, which
   causes Spack to minimize the number of rebuilds it does. The --reuse
   option will try to find existing installations and binary packages locally
   and in registered mirrors, and will prefer to use them over building new
   versions. This will allow users to build from source far less than in
   prior versions of Spack. This feature will continue to be improved, with
   configuration options and better CLI expected in v0.17.1. It will become
   the default concretization mode in v0.18.0.

4. Better error messages
   We have improved the error messages generated by the new concretizer by
   using unsatisfiable cores. Spack will now print a summary of the types
   of constraints that were violated to make a spec unsatisfiable (#26719).

5. Conditional variants
   Variants can now have a when="<spec>" clause, allowing them to be
   conditional based on the version or other attributes of a package (#24858).

6. Git commit versions
   In an environment and on the command-line, you can now provide a full,
   40-character git commit as a version for any package with a top-level
   git URL. e.g., spack install hdf5@45bb27f58240a8da7ebb4efc821a1a964d7712a8.
   Spack will compare the commit to tags in the git repository to understand
   what versions it is ahead of or behind.

7. Override local config and cache directories
   You can now set SPACK_DISABLE_LOCAL_CONFIG to disable the ~/.spack and
   /etc/spack configuration scopes. SPACK_USER_CACHE_PATH allows you to
   move caches out of ~/.spack, as well (#27022, #26735). This addresses
   common problems where users could not isolate CI environments from local
   configuration.

8. Improvements to Spack Containerize
   For added reproducibility, you can now pin the Spack version used by
   spack containerize (#21910). The container build will only build
   with the Spack version pinned at build recipe creation instead of the
   latest Spack version.

9. New commands for dealing with tags
   The spack tags command allows you to list tags on packages (#26136), and you
   can list tests and filter tags with spack test list (#26842).

Other new features of note

• Copy and relocate environment views as stand-alone installations (#24832)
• spack diff command can diff two installed specs (#22283, #25169)
• spack -c <config> can set one-off config parameters on CLI (#22251)
• spack load --list is an alias for spack find --loaded (#27184)
• spack gpg can export private key with --secret (#22557)
• spack style automatically bootstraps dependencies (#24819)
• spack style --fix automatically invokes isort (#24071)
• build dependencies can be installed from build caches with --include-build-deps (#19955)
• spack audit command for checking package constraints (#23053)
• Spack can now fetch from CVS repositories (yep, really) (#23212)
• spack monitor lets you upload analysis about installations to a
  spack monitor server (#23804, #24321,
  #23777, #25928))
• spack python --path shows which python Spack is using (#22006)
• spack env activate --temp can create temporary environments (#25388)
• --preferred and --latest options for spack checksum (#25830)
• cc is now pure posix and runs on Alpine (#26259)
• SPACK_PYTHON environment variable sets which python spack uses (#21222)
• SPACK_SKIP_MODULES lets you source setup-env.sh faster if you don't need modules (#24545)

Major internal refactors

• spec.yaml files are now spec.json, yielding a large speed improvement (#22845)
• Splicing allows Spack specs to store mixed build provenance (#20262)
• More extensive hooks API for installations (#21930)
• New internal API for getting the active environment (#25439)

Performance Improvements

• Parallelize separate concretization in environments; Previously 55 min E4S solve
  now takes 2.5 min (#26264)
• Drastically improve YamlFilesystemView file removal performance via batching (#24355)
• Speed up spec comparison (#21618)
• Speed up environment activation (#25633)

Archspec improvements

• support for new generic x86_64_v2, x86_64_v3, x86_64_v4 targets
  (see archspec#31)
• spack arch --generic lets you get the best generic architecture for
  your node (#27061)
• added support for aocc (#20124), arm compiler on graviton2 (#24904)
  and on a64fx (#24524),

Infrastructure, buildcaches, and services

• Add support for GCS Bucket Mirrors (#26382)
• Add spackbot to help package maintainers with notifications. See
  spack.github.io/spackbot
• Reproducible pipeline builds with spack ci rebuild (#22887)
• Removed redundant concretizations from GitLab pipeline generation (#26622)
• Spack CI no longer generates jobs for unbuilt specs (#20435)
• Every pull request pipeline has its own buildcache (#25529)
• --no-add installs only specified specs and only if already present in… (#22657)
• Add environment-aware spack buildcache sync command (#25470)
• Binary cache installation speedups and improvements (#19690, #20768)

Deprecations and Removals

• 0.17.x is the last Spack release series that will support Python 2.6. We will no longer retain 2.6 compatibility in develop and it will be unsupported in 0.18.
• spack setup was deprecated in v0.16.0, and has now been removed.
  Use spack develop and spack dev-build.
• Remove unused --dependencies flag from spack load (#25731)
• Remove stubs for spack module [refresh|find|rm|loads], all of which
  were deprecated in 2018.

Notable Bugfixes

• Deactivate previous env before activating new one (#25409)
• Many fixes to error codes from spack install (#21319, #27012, #25314)
• config add: infer type based on JSON schema validation errors (#27035)
• spack config edit now works even if spack.yaml is broken (#24689)

Packages

• Allow non-empty version ranges like 1.1.0:1.1 (#26402)
• Remove .99's from many version ranges (#26422)
• Python: use platform-specific site packages dir (#25998)
• CachedCMakePackage for using *.cmake initial config files (#19316)
• lua-lang allows swapping lua and luajit (#22492)
• Better support for ld.gold and ld.lld (#25626)
• build times are now stored as metadata in $prefix/.spack (#21179)
• post-install tests can be reused in smoke tests (#20298)
• Packages can use pypi attribute to infer homepage/url/list_url (#17587)
• Use gnuconfig package for config.guess file replacement (#26035)
• patches: make re-applied patches idempotent (#26784)

Spack community stats

• 5969 total packages, 920 new since v0.16.0
  • 358 new Python packages, 175 new R packages
• 513 people contributed to this release
  • 490 committers to packages
  • 105 committers to core
• Lots of GPU updates:
  • ~77 CUDA-related commits
  • ~66 AMD-related updates
  • ~27 OneAPI-related commits
  • 30 commits from AMD toolchain support
• spack test usage in packages is increasing
  • 1669 packages with tests (mostly generic python tests)
  • 93 packages with their own tests