open-policy-agent/opa v1.21.0

on GitHub

This release contains a mix of new features and bug fixes. Notably:

• Improved rule indexing
• Improved rule recursion check
• YAML is parsed against the 1.2 core schema (breaking change)

Rules with general refs no longer collide in the recursion check (#6813)

Before, this was a recursion error:

package play

p[x].foo.bar if {
	x := "a"
	not p[x].foo.baz
}

p[x].foo.baz if {
	x := "a"
	false
}

Rules with a variable in their head are all stored at the ground prefix of their ref, so
p[x].foo.bar and p[x].foo.baz looked like dependencies of each other. The compiler is
now less conservative and compares the ref parts past the prefix. Genuine cycles are still
reported.

The IR and Wasm targets however still return an error: they plan one function per ground
path prefix, and cannot evaluate part of a function that is still being planned.

Authored by @sspaink, reported by @tsandall

Data and Query APIs can return rule labels in the response (#9211)

# METADATA labels for evaluated rules were only available in decision log
events. The Data API (GET/POST /v1/data) and Query API (GET/POST /v1/query) now accept a rule_labels query parameter to include the same
merged labels in the response payload, under a rule_labels key.

Authored by @srenatus

Behavior change: response gzip compression now bounds its buffer to min_length (#9205)

The server's gzip response compression (server.encoding.gzip) buffered an entire
incoming Write call before deciding whether to compress, so a single large write could
grow the buffer well past min_length before that decision was made. The handler is now
built on klauspost/compress/gzhttp
instead of a hand-rolled buffer and gzip.Writer pool, which caps what it buffers to
min_length (floored at 512 bytes) before streaming the remainder through the chosen
path. min_length and compression_level behave the same as before; only gzip is
negotiated, not zstd.

Authored by @srenatus

YAML is now parsed against the 1.2 core schema (#5754, #6598)

OPA parsed YAML with a library pinned to go-yaml v2, which implements YAML 1.1. Under
1.1, the bare words y, n, yes, no, on and off resolve to booleans, so a
GitHub Actions workflow loaded with --data came back with true where it should have
had on:

on: push

{ "true": "push" }

These words are now plain strings, as the YAML 1.2 core schema specifies. true and
false are unaffected. This applies everywhere OPA reads YAML: --data, bundles,
config files, and the yaml.unmarshal builtin.

If you were relying on yes/no/on/off being read as booleans, quote the value and
use true/false instead.

Authored by @sspaink, reported by @scnewma and @johnc-c

Empty composite literals are now typed as empty (#7275)

The type checker used to give the empty object literal {} the type
object[any: any], the empty array literal [] the type array[any], and the
empty set literal set() the type set[any], i.e. the types of a collection
that may hold anything. Every other literal is typed by its contents, so
referencing a key that isn't there is caught at compile time — but only for
non-empty literals:

obj := {"foo": "bar"}
obj.bar # rego_type_error: undefined ref: obj.bar

obj := {}
obj.bar # compiles

Empty literals are now typed as what they are: an object with no properties, an
array with no items, and a set with no members. Both examples above now fail to
compile, and so does every other way of selecting from an empty literal,
including iterating one (some x in []).

Comparing an empty object or array literal against a value whose type says it
can't be empty ({"foo": "bar"} == {}) is now a match error too, the same way
{"foo": "bar"} == {"bar": "foo"} already was. Use count(x) == 0 to test a
collection for emptiness without asserting its type. Sets are unaffected here:
set[string] describes any set of strings, the empty one included, so
{"foo"} == set() still compiles.

Authored by @sspaink, reported by @disaverio

Rule indexing improvements

The rule indexer now excludes rules from more kinds of expression, and builds a smaller
trie to do it with. See Use indexed statements
for what is indexed.

• startswith, endswith, strings.any_prefix_match and strings.any_suffix_match
  are indexed when the base strings are known at compile time.
• A reference that reads a key out of the object at its ground prefix in base data
  (data.groups.admins.members[input.subject]) is indexed by asking that object for the
  key, where such a ruleset used to leave every rule a candidate.
• References rooted at a local variable (x := input; x.foo == "a") are indexed the
  same as input.foo == "a", and a chain of assignments no longer drops the constraint
  at the end of it.
• A rule's path through the trie stops at the last level it constrains, and a reference
  reached by several values no longer leaves the rest of the rule unindexed.
• Candidates come back in declaration order, which the indexer documented but did not
  do. A complete rules must not produce multiple outputs error now points at the first
  of the conflicting definitions rather than the second, and partial evaluation names
  and orders the generated locals of its support rules differently. What a policy
  evaluates to is unaffected.

Changes

• ast: Count a ref once when an index entry replaces its var entry (#9257) authored by @srenatus, reported by @vlsi
• ast: Index a lookup into a collection in base data (#9235) authored by @srenatus
• ast: Index refs rooted at a local variable (#9081) authored by @srenatus
• ast: Let concrete index values supersede leftover "any" entries (#9081) authored by @srenatus
• ast: Number the references an index is built on (#9190) authored by @srenatus
• ast: Number the refs a rule requires (#9244) authored by @srenatus
• ast: Number the rules an index holds (#9190) authored by @srenatus
• ast: Keep refs no rule constrains to a value out of the trie (#9190) authored by @srenatus
• ast: Stop a rule index path at its last constrained level (#9108) authored by @srenatus
• ast: Walk a rule tree's children in a stable order (#9190) authored by @srenatus
• docs: Document indexing of refs rooted at a local variable (#9081) authored by @srenatus
• index: Also index suffix matching (endswith, strings.any_suffix_match) + some tweaks (#9164) authored by @srenatus
• perf: Add startswith and strings.any_prefix_match indexing (#9161) authored by @tsandall
• rego: Benchmark index lookups at every match position (#9190) authored by @srenatus

Runtime, SDK, Tooling

• compile: don't panic on non-string table/column mappings (#9241) authored by @sspaink
• debug: Adding query stack-trace framing mode (#9128) authored by @johanfylling
• dependencies: Include else bodies and unused ref bindings (#4814) authored by @sspaink, reported by @anderseknert
• download: Fix Trigger() racing a cancelled context into a false success (#9233) authored by @srenatus
• download: Fix ignored OCI downloader settings (#9113) authored by @sspaink
• download: Note that ociTarget.Exists is dead code (#9233) authored by @srenatus
• download: Resolve OCI bundles behind an image index (#7461) authored by @sspaink, reported by @zscott
• download: Stop BundleRequest timer on OCI early returns (#9233) authored by @srenatus
• fix: runner.CapturePrintOutput setting never read (#9104) authored by @anderseknert
• format: don't drop comments after an inline if body (#9109) authored by @sspaink, reported by @anderseknert
• metricsexport: Support custom headers on the OTLP exporter (#9234) authored by @srenatus
• plugins/logs: Add trace_id, span_id and request_context to the event AST (#9193) authored by @sspaink
• plugins/logs: Fix data race on the cached mask and drop queries (#9189) authored by @sspaink
• plugins/logs: Make BenchmarkMaskingRuleCountsNop vary the rule count (#9222) authored by @srenatus
• plugins/logs: Report upload failures when retrying requeued chunks (#9186) authored by @sspaink
• plugins/rest: Remove unused azureSigningAuthPlugin.host field (#9215) authored by @sspaink
• repl: Honor DisableUndefinedOutput setting (#9185) authored by @anderseknert
• repl: Recall multi-line statements as one history entry (#4939) authored by @sspaink, reported by @stobias123
• rest: Fix SSO cache path written to wrong field (#9233) authored by @srenatus
• rest: Remove stray debug print in Azure KeyVault signing (#9233) authored by @srenatus
• runtime: Only log diagnostic API access at DEBUG (#8419) authored by @srenatus, reported by @msahmi
• runtime: Reload the config file on change when --watch is set (#9184) authored by @sspaink, reported by @charlieegan3
• runtime: Return the listener error instead of exiting the process (#9240) authored by @srenatus
• runtime: Revert config file reload on --watch (#9219) authored by @sspaink
• server: Build the middleware stack in one place (#9233) authored by @srenatus
• storage/disk: Split large bundle writes across transactions (#9202) authored by @sspaink
• tracing: Add distributed_tracing.exclude_paths (#7494) authored by @sspaink, reported by @srenatus
• yaml: Reject documents with unreachable content (#6854) authored by @sspaink, reported by @kishorviswanathan

Compiler, Topdown and Rego

• ast: Avoid pointer escape in GenericTransformer (#9148) authored by @anderseknert
• ast: Build package exports in a single pass (#9162) authored by @srenatus
• ast: Clear shared output buffer in outputVarsForExprEq (#8302) authored by @zanarellidev, reported by @johanfylling
• ast: Clear the term cache when a brace operand guess is abandoned (#9140) authored by @sspaink
• ast: Collect a lookup's candidates in a bitset (#9190) authored by @srenatus
• ast: Emphasize top-most differing types in type errors (#499) authored by @sspaink, reported by @tsandall
• ast: Fix panic comparing a decimal zero with a non-integral number (#9098) reported and authored by @kmadan
• ast: Fix type errors from allowed undefined function calls (#6946) authored by @sspaink, reported by @nikpivkin
• ast: Hint at missing future keyword imports (#4619) authored by @sspaink, reported by @anderseknert
• ast: Mark JSON schema builtins nondeterministic (#8998) authored by @ARMeeru, reported by @charlesdaniels
• ast: Only compute template string scopes for rules that have one (#9248) authored by @srenatus
• ast: Point object parse errors at the offending token (#6714) authored by @sspaink, reported by @charlesdaniels
• ast: Report keywords used as rule names (#6652) authored by @sspaink, reported by @johanfylling
• ast: Report violations from multiple compiler stages (#5815) authored by @sspaink, reported by @anderseknert
• ast: Resolve local ref heads in the ground-prefix path too (#9081) authored by @srenatus
• ast: Skip the reordered body's output vars where no closure reads them (#9248) authored by @srenatus
• ast: Type check the in operator against the collection's types (#5658) authored by @sspaink, reported by @anderseknert
• ast: Type empty object and array literals by their contents (#7275) authored by @sspaink, reported by @anderseknert
• ast: don't box a slice header on the way out of Transform (#9248) authored by @srenatus
• ast: don't rebuild modules that have nothing to rewrite (#9248) authored by @srenatus
• builtins: Reject leading zeroes and empty pre-release/build in semver built-ins (#9004) authored by @sueun-dev
• compiler: HasherMap returned by getExports never used (#9149) authored by @anderseknert
• rego: Fix EvalDisableInlining always being overridden (#9233) authored by @srenatus
• rego: Pass Time and Seed through Partial() like Eval() does (#9233) authored by @srenatus
• rego: don't run leaktest checks in a parallel test (#9176) authored by @sspaink
• topdown: Add stack traces to evaluation errors (#555) authored by @sspaink, reported by @tsandall
• topdown: Fix regex cache leak (#9087) reported and authored by @charlesdaniels
• topdown: Fix sprintf formatting of floats with zero fraction (#9187) authored by @sspaink, reported by @rothenes
• topdown: Hoist enumerate callbacks out of the loop (#9147) authored by @srenatus
• topdown: Iterate known keys and save unknown ones during PE (#9139) reported and authored by @srenatus
• topdown: Record evaluated rules during partial evaluation (#9163) authored by @srenatus
• wasm: Match topdown semantics in strings.replace_n (#9216) authored by @andreaTP

Docs, Website, Ecosystem

• docs: Add Evolith to the OPA ecosystem (#9196) authored by @beyondnetPeru
• docs: Address a number of broken links in blog (#9117) authored by @charlieegan3, reported by @github-actions
• docs: Address incorrect package name in example (#9250) authored by @charlieegan3
• docs: Ecosystem entry for Agent Evidence Admission (#9213) authored by @astrogilda
• docs: Remove word from missed review to #9172 (#9183) authored by @charlieegan3
• docs: Report builtin availability in other interpreters (#8228) authored by @sspaink, reported by @srenatus
• docs: Update Agent Evidence Admission repo links (#9255) authored by @astrogilda
• docs: Update builtin availability in other interpreters (#9157) authored by @johanfylling
• docs: Updates to AI guidelines (#9172) authored by @charlieegan3
• website: Implement local search based on Pagefind (#9249) authored by @charlieegan3
• website: Use new kapa attr to hide AI chat button (#9239) authored by @charlieegan3

Miscellaneous

• ast: Add util.MapKeys helper (#9158) authored by @anderseknert
• ast: Enable more gocritic linters (#9154) authored by @anderseknert
• ast: Enable unparam linter (#9223) authored by @anderseknert
• ast: More niceties, less allocs, less code (#9228) authored by @anderseknert
• ast: Pin BenchmarkObjectConstruction shuffle seed (#9222) authored by @srenatus
• ast: Update remaining errors.As call sites to use errors.AsType (#9106) authored by @anderseknert
• ast: Use modern Go in place of custom compare code (#9151) authored by @anderseknert
• ast: Where have all the allocs gone? (#9137) authored by @anderseknert
• build: Add bench-nightly, a three-arm benchlab experiment runner (#9118) authored by @srenatus
• build: Pin pigeon in build/tools instead of go run pkg@version (#9160) authored by @sspaink
• bundle: Avoid allocation in getdepth (#9199) authored by @srenatus
• bundle: Remove unused writeModules helper (#9199) authored by @srenatus
• bundle: Reuse encoder buffer while hashing (#9199) authored by @srenatus
• bundle: deep-copy bundle data natively instead of via JSON round-trip (#9199) authored by @srenatus
• check: Avoid allocating in checkExprEq (#9150) authored by @anderseknert
• ci: Publish benchmark trend on a schedule instead of per-push (#9119) authored by @srenatus
• ci: Run the nightly benchlab experiment (#9118) authored by @srenatus
• cmd: Stop binding a fixed port in the run tests (#9240) authored by @srenatus
• github: Drop python from the CodeQL language matrix (#9097) authored by @sspaink
• nightly: Fix go get smoke test (#9245) authored by @srenatus
• perf: Cheaper custom function calls (#9167) authored by @anderseknert
• perf: Fix linear runtime for Array.set due to rehashing (#9161) authored by @tsandall
• perf: General performance improvements in compiler (#9170) authored by @anderseknert
• style: Some more functional niceties (#9152) authored by @anderseknert
• test/e2e: Wait for diagnostic listeners before running tests (#9134) authored by @sspaink
• tests: Expanded testing for and/or keywords (#9115) authored by @johanfylling
• topdown: Fix BulkStartsWith benchmark input generation (#9222) authored by @srenatus
• topdown: Enable more revive linters (#9181) authored by @anderseknert
• topdown: Fix flaky TestRegexBuiltinCache (#9254) authored by @sspaink
• topdown: Fix linter issues (#9231) authored by @srenatus
• util: Add RoundTripFast (#9199) authored by @srenatus
  • plugins/logs: Use util.RoundTripFast for decision-log event conversion (#9199) authored by @srenatus
  • rego: Use util.RoundTripFast for input parsing (#9199) authored by @srenatus
  • storage: Use util.RoundTripFast for write round-tripping (#9199) authored by @srenatus
• util: Decode RoundTrip's fallback into a fresh value (#9206) authored by @srenatus
• workflows: Check the nightly go-get job for retractions via the proxy (#9240) authored by @srenatus
• workflows: Remove benchmarks from nightly (#9182) authored by @srenatus
• workflows: Run Regal's do.rq in the nightly compatibility check (#9209) authored by @sspaink
• workflows: Use OCP@main in nightly (#9171) authored by @srenatus
• Collapse v0 shim packages into a single file each (#8976) authored by @sspaink, reported by @anderseknert
• Makefile: Add a benchlab target (#9222) authored by @srenatus
• Remove retired go report (#9112) authored by @sspaink
• Dependency updates:
  • build(go): Bump to 1.27.1 (#9136) authored by @srenatus
  • build(deps): Bump github.com/dgraph-io/badger/v4 from 4.9.5 to 4.9.6
  • build(deps): Bump github.com/gobwas/glob to v1.0.0 (#9114) authored by @sspaink, reported by @ghmer
  • build(deps): Bump github.com/huandu/go-sqlbuilder from 1.42.1 to 1.43.0
  • build(deps): Bump github.com/lestrrat-go/jwx/v3 from 3.1.1 to 3.3.0
  • build(deps): Bump github.com/olekukonko/tablewriter from 1.1.4 to 1.1.5
  • build(deps): Bump github.com/prometheus/client_model from 0.6.2 to 0.6.3
  • build(deps): Bump github.com/santhosh-tekuri/jsonschema/v6 from 6.0.2 to 6.0.3
  • build(deps): Bump github.com/sirupsen/logrus from 1.9.4 to 1.10.2
  • build(deps): Bump github.com/vektah/gqlparser/v2 from 2.5.36 to 2.5.37
  • build(deps): Bump go.opentelemetry.io/contrib/bridges/prometheus from 0.69.0 to 0.71.0
  • build(deps): Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from 0.69.0 to 0.71.0
  • build(deps): Bump go.opentelemetry.io/otel from 1.44.0 to 1.46.0
  • build(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc from 1.44.0 to 1.46.0
  • build(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp from 1.44.0 to 1.46.0
  • build(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace from 1.44.0 to 1.46.0
  • build(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.44.0 to 1.46.0
  • build(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.44.0 to 1.46.0
  • build(deps): Bump go.opentelemetry.io/otel/sdk from 1.44.0 to 1.46.0
  • build(deps): Bump go.opentelemetry.io/otel/sdk/metric from 1.44.0 to 1.46.0
  • build(deps): Bump go.yaml.in/yaml/v3 from 3.0.4 to 3.0.5
  • build(deps): Bump golang.org/x/sync from 0.22.0 to 0.23.0
  • build(deps): Bump golang.org/x/term from 0.45.0 to 0.46.0
  • build(deps): Bump golang.org/x/text from 0.40.0 to 0.42.0
  • build(deps): Bump golang.org/x/time from 0.15.0 to 0.16.0
  • build(deps): Bump google.golang.org/grpc from 1.82.1 to 1.83.2
  • build(deps): Bump google.golang.org/protobuf from 1.36.11 to 1.36.12
  • build(deps): Drop sigs.k8s.io/yaml (was 1.6.0)