2.19.x Release Series

Pants 2 is a fast, scalable, user-friendly build system for codebases of all sizes. It's currently focused on Python, Go, Java, Scala, Kotlin, Shell, and Docker, with support for other languages and frameworks coming soon.

Individuals and companies can now sponsor Pants financially.

Pants is an open-source project that is not owned or controlled by any one company or organization, and does incur some expenses. These expenses are managed by Pants Build, a non-profit that was established for this purpose. This non-profit's only source of revenue is sponsorship by individuals and companies that use Pants.

We offer formal sponsorship tiers for companies, as well as individual sponsorships via GitHub.

What's New

Highlights

• Test retries for flaky Python tests.
• Using the parametrize to set multiple fields at once when generating new groups of targets.
• buildx support in the Docker backend.
• New backends for running semgrep and openapi-format.

Keep reading to see the details and what's also included.

Overall

The .pids/ internal directory is now under .pants.d/ by default, rather that adjacent to it (this can be controlled via the [GLOBAL].pants_subprocessdir option).

The new [system-binaries] subsystem allows controlling where Pants searches for binaries like bash, zip, tar that is uses as part of normal operation. Set the system_binaries_paths as appropriate, potentially using the <PATH> special value. This can also be set on specific environments using the system_binaries_system_binary_paths field.

Additional features and fixes:

• New versions of the get-pants.sh script install to ~/.local/bin, not ~/bin.
• Environment variables that aren't valid UTF-8 no longer cause Pants to crash (note, PANTS_... environment variables that Pants needs to read but aren't UTF-8 will be ignored).
• Reference documentation for subsystem options includes how to set them in pants.toml, in addition to the environment variable and CLI arguments.
• Environments are now more resilient to being (partially) defined via macros, although this is not recommended.
• Docker environments can now pull public images without credentials, and can now handle docker images without Python.
• The fix goal now partitions files across multiple processes in a way that's more similar to the partitioning of the lint goal, which can work around some spurious linting failures.

BUILD files

The parametrize helper now supports parametrizing multiple fields together, in lock step. For instance:

# Creates two targets:
#
#    example:tests@parametrize=py2
#    example:tests@parametrize=py3

python_test(
    name="tests",
    source="tests.py",
    **parametrize("py2", interpreter_constraints=["==2.7.*"], resolve="lock-a"),
    **parametrize("py3", interpreter_constraints=[">=3.6,<3.7"], resolve="lock-b"),
)

In addition, parametrize can now be used with a single value (field=parametrize("for-consistency")), and now works on target generations in more cases (particularly those with "plugin" fields).

Performance

Some processes are run with more consistent "color" configuration which may result in more cache hits: Pytest, Helm unittest and shunit2 processes now always run with colored output enabled. If [GLOBAL].colors is not enabled, Pants removes the coloring before displaying.

Remote caching/execution

Large files that need to be stored to a remote cache are now streamed to it directly from disk, in a way that should reduce overhead and increase concurrency (#19711).

The deprecation has expired for the [GLOBAL].remote_store_chunk_upload_timeout_seconds and [GLOBAL].remote_cache_read_timeout_millis options. Use [GLOBAL].remote_store_rpc_timeout_millis and [GLOBAL].remote_cache_rpc_timeout_millis instead.

Backends

Docker

The docker backend now has explicit support for using buildx (and thus BuildKit), via the new use_buildx option. This also allows adding few additional fields for docker_image targets, for buildx-specific functionality:

• cache_from and cache_to for using external cache storage backends
• build_platform for configuring multi-platform builds
• output to control how/where buildx outputs an image when running pants package ::

JVM

JVM lock-files are more reliable, handling some cases of Coursier not including dependencies when expected.

Using shading_rules is now supported on deploy_jar targets defined in directories of the build root.

Kotlin

The Kotlin analyzer now runs using the Zulu JDK, to be able to run natively on Apple Silicon (arm64 macOS).

Java

Dependency inference for Java now supports code using recently added syntax like sealed class & permits, switch expressions and yield.

Scala

Scala 3 is now supported when using the BSP in an IDE.

OpenAPI

The new pants.backend.experimental.openapi.lint.openapi_format backend supports running openapi-format to format openapi_... targets.

Protobuf

The pants.backend.codegen.protobuf.python backend now supports:

• protobuf_sources targets that are part of multiple Python resolves, like python_resolve=parametrize("first-resolve", "second-resolve").
• [python-infer].ambiguity_resolution = "by_source_root" to infer dependencies without warnings or errors in more cases.

Version 24.4 of the protoc compiler is included as a known version (but is not the default). It can be selected by setting [protoc].version = "24.4". This version has native support for Apple Silicon (arm64 macOS).

Buf (pants.backend.codegen.protobuf.lint.buf) now correctly ignores buf.work.yaml, buf.lock, buf.gen.lock when running as a linter.

Python

Deprecation: Using the platforms field on pex_binary is now deprecated. It is not a recommended way to build for platforms other than the local one, as it is a lossy abbreviation of the target platform and often leads to binaries that don't work. To replace this, either use complete_platforms to continue cross-compiling or use an environment matching the target platform.

Python tests can now be attempted multiple times, if they fail. This can help with limiting the impact of flaky tests. For instance, to require tests to fail three times before the overall pants test goal fails, set [test].attempts_default = 3. In this case, a very flaky test might fail on the first and second runs but finally pass on the third run, and thus the overall Pants invocation would pass.

Pants now uses version 2.1.148 of the PEX CLI by default, which includes support for Python 3.12. To use Python 3.12, you will need to set [python].pip_version to 23.2 or newer (or latest), and ensure all of your tools use resolves generated with this new version (Pants' built-in ones are not, yet).

Python interpreters provided by ASDF are now searched by default, in addition those provided by pyenv and on the system PATH, by being included in the [python-bootstrap].search_path option.

Many more "groups" of PyPI dependencies have default module mappings that guide how an import statement in code matches particular python_requirements. All packages from the azure-..., django-..., google-cloud-..., opentelemetry-instrumentation-..., oslo-... ecosystems are now matched to modules matching their usual conventions. In addition, packages starting with python-... now have that python- prefix removed by default: for instance, a package like python-example-name will be implicitly assumed to be imported like import example_name. The value of any module_mapping field can thus be tweaked (hopefully simplified!).

The [generate-lockfiles].diff option is now True by default, so running pants generate-lockfiles now pretty-prints any differences in the dependencies and their versions.

Additional features and fixes:

• Additional arguments can be passed to the IPython REPL, like pants repl --shell=ipython :: -- -i helloworld/main.py.
• Additional arguments can be passed to the underlying PEX invocations when building FaaS artifacts (AWS Lambda, Google Cloud Functions) via the new pex3_venv_create_extra_args field. For instance, if dependencies have packaged files in unexpected locations, passing pex3_venv_create_extra_args=["--collisions-ok"] can side-step collision errors.
• The pip_version field no longer restricts the values, and so allows more flexibility when upgrading the PEX CLI if it supports newer Pip versions.
• Running tests or pants run a source file now make binaries provided by python_requirement available for execution as normal processes (for instance, using subprocess.run(["name-of-binary", ...])).
• The PATH environment variable can now be set using [subprocess-environment].env_vars
• Import statements within a with contextlib.suppress(ImportError) context are now weak, and so no longer warn if not provided by a dependency.
• Computing an aggregate coverage sandbox report is now more reliable in the presence of name collisions in files created via code-generation (including relocated_files).
• Pyright (pants.backend.experimental.python.typecheck.pyright) now runs with an appropriately initialised environment and so runs more reliably (especially in transient CI environments, where "named caches" might not be preserved).
• Ruff (pants.backend.experimental.python.lint.ruff) now properly respects any exclude configuration from ruff.toml or pyproject.toml.

The following code has been removed, because deprecations have expired:

• Support for building FaaS artifacts (AWS Lambda, Google Cloud Functions) via Lambdex has been fully replaced with the new zip layout: remove the [lambdex] section and its layout setting from pants.toml.
• Passing both runtime and complete_platforms to a FaaS target (python_aws_lambda_function, python_google_cloud_function) is now longer supported, as complete_platforms fully overrides runtime.
• Dependency inference now only supports running with the new Rust-based parser. That is, the [python-infer].use_rust_parser option cannot be disabled and is now unnecessary.

NEW: Semgrep

The new pants.backend.experimental.tools.semgrep backend supports running Semgrep as a local linter on all files that Pants knows about (any target with a source or sources field). Configuration files are automatically discovered.

Shell

The experimental_test_shell_command target now obeys its environment field, and thus runs the command within the specified environment instead of the host.

Terraform

The check goal runs terraform validate on terraform_modules targets by default, again. For modules for which this doesn't work and/or isn't desirable, use the new skip_terraform_validate field. (This restores the behaviour from Pants 2.17 and earlier, with the addition of the skip field, based on feedback about Pants 2.18.0 which switched to only validating terraform_deployments.)

terraform_deployment's var_files field now supports files in a different directory to the root_module.

Transitive dependencies are now automatically included, in addition to direct dependencies.

Plugin API changes

Processes, especially tests, can now be run with retries on failure using ProcessWithRetries (#20378). In addition, to support tests that have been retried multiple times, TestResult.from_fallible_process_result now accepts a tuple of multiple FallibleProcessResults, instead of just one. (#19760).

Some @rules may now need to pass canonical_name_suffix in preparation for "call by name". (#19755)

The new TransitivelyExcludeDependenciesRequest can be used to emulate transitive excludes (!!<address>). (#20080)

The TestResult.stdout and TestResult.stderr fields are now removed, in favour of stdout_bytes and stderr_bytes. (#19768)

Full Changelog

For the full changelog, see the individual GitHub Releases for this series: https://github.com/pantsbuild/pants/releases