kgateway-dev/kgateway v2.4.0-rc.1

on GitHub

🎉 Welcome to the v2.4.0-rc.1 release of the kgateway project!

Release Notes

Changes since v2.3.0

Breaking Changes

• Breaking: Using APIs to override image tags clears any digest. Non-breaking: Using APIs to change image digests now clears the image tag by default. Kubernetes did not use the tag when a digest was provided, and you can still provide both if you choose.

  This break applies only to users who have GatewayParameters that use a non-empty sha256 image digest. (Default parameters only use image tags, never image digests.)
  (#13953)

• remove deprecated perConnectionBufferLimitBytes gateway annotation support. Use ListenerPolicy instead. (#14156)

New Features

• Add headersToClient and headersToClientOnSuccess fields to AuthorizationResponse in ExtHttpService, mapping to Envoy's allowed_client_headers and allowed_client_headers_on_success. This enables redirect-based OAuth2 flows by forwarding auth service response headers (e.g. Location) to the downstream client. (#13592)
• Add support for gateway-level TrafficPolicy application to routes. TrafficPolicies attached to Gateways now apply to child HTTPRoutes, with route-level policies taking precedence over gateway-level ones. (#13596)
• Envoy routes generated by kgateway now carry a io.kgateway.route_source
  filter metadata entry with the originating xRoute's kind, group, name,
  namespace, and rule name — enabling tracing correlation and access-log
  filtering against route origin.
  (#13620)
• Add filterStage configuration to ExtProc GatewayExtension, allowing control over filter chain positioning via stage, predicate, and weight fields. (#13845)
• Now storing release artifacts for each commit to make it easier to try out cutting-edge releases. Previously we only stored vX.Y.Z-main, a moving tag. Also fixes a minor race that could cause that tag to point to an older commit than desired. (#13876)
• Add GatewayParameters.spec.kube.envoyContainer.bootstrap.staticListenerProxyProtocol to enable the Envoy PROXY protocol listener filter on the readiness listener port. (#13879)
• TrafficPolicy headerModifiers now supports secret-backed header values via secretRef on set/add entries. name and key are both optional: each defaults to the other when only one is specified, and omitting both injects all entries from the secret as headers. The secret namespace defaults to the policy namespace. Cross-namespace references require a ReferenceGrant. (#13880)
• Add maxRequestsPerConnection to ListenerPolicy.httpSettings to cap downstream keepalive connections per client. Useful for rebalancing long-lived HTTP/2 and gRPC connections across gateway pods when using L4 load balancers such as AWS NLB. (#13922)
• Added forwardClientCertDetails to httpSettings on ListenerPolicy for forwarding selected fields of the downstream client certificate to upstream backends via the x-forwarded-client-cert (XFCC) header. (#13925)
• Added AWS EC2 backend support behind the controller.enableAwsEc2Discovery setting, allowing Backend resources to dynamically discover tagged EC2 instances and route to them through Envoy EDS. Lambda backends now prefer spec.aws.lambda.accountId, while the existing top-level spec.aws.accountId remains supported for backward compatibility. (#13961)
• Added BackendConfigPolicy zone-aware routing with native Envoy prefer-local and force-local support, including bootstrap locality wiring for Envoy proxies. (#13978)
• Added http2ProtocolOptions support to ListenerPolicy and HTTPListenerPolicy for configuring downstream HTTP/2 settings on Envoy listeners. (#13991)
• Add downstream TCP keepalive configuration to ListenerPolicy. (#13994)
• Add kgateway_routing_replacements_total counter metric. (#14019)
• Users can now customize the readiness and startup probes (#14020)
• Added stripHostPortMode to ListenerPolicy HTTPSettings, providing the option to strip ports
  from Host/authority headers before forwarding to upstream backends.
  (#14031)
• Add merge semantics for BackendConfigPolicy and define precedence between BackendConfigPolicy and BackendTLSPolicy when both target the same backend. (#14043)
• Add transportSocketConnectTimeout field to ListenerPolicy (#14058)
• Add an rbac.create value to disable the creation of the ClusterRole and ClusterRoleBinding (#14105)
• Add requestAttributes field to GatewayExtension ext_proc config, allowing
  Envoy attributes (e.g. source.address) to be forwarded to ext_proc servers.
  (#14109)
• Add an opt-in admin bind address setting for the kgateway control-plane admin server. (#14135)
• Added Envoy local reply configuration to ListenerPolicy (#14146)
• Add optional jwksBackendRef field to OAuth2JWTConfig in GatewayExtension, allowing JWKS endpoint to be fetched through a different backend than the OAuth2 token endpoint. (#14147)
• Added an AssumeRole AWS auth type to the Backend API (spec.aws.auth.assumeRole) for per-Backend STS role chaining, used for both Lambda request signing and EC2 instance discovery. The previous spec.aws.ec2.roleArn field is replaced by spec.aws.auth.assumeRole.roleArn. (#14148)
• EC2 backends now report an EndpointsDiscovered status condition reflecting whether runtime endpoint discovery succeeded, including credential, authorization, and zero-match failures. (#14173)
• added max_headers_count setting to ListenerPolicy (#14188)
• added reference grant mode (#14209)
• Added asyncFetch and retryPolicy options to the JWT provider remote JWKS configuration, enabling asynchronous JWKS fetching/caching and retries with exponential backoff. (#14211)
• Added support for GatewayHTTPListenerIsolation conformance behavior for HTTP listeners. (#14234)
• Add support for Gateway API 1.6.0 (#14245)
• Add internal redirect feature in TrafficPolicy (#14261)
• TrafficPolicy response compression now supports selecting and negotiating multiple codecs via compression.responseCompression.libraries (Gzip, Brotli, Zstd, in preference order). Envoy negotiates the codec from the request's Accept-Encoding header. Defaults to Gzip, preserving existing behavior. (#14275)
• Allow configuring supported TLS signature algorithms in Gateway and BackendConfigPolicy (#14282)
• EC2 backends now expose Prometheus metrics for endpoint discovery: kgateway_ec2_discovery_poll_total, kgateway_ec2_discovery_endpoints_active, kgateway_ec2_discovery_error_state, and kgateway_ec2_discovery_poll_duration_seconds. (#14318)
• Added allowConnect to listener http2ProtocolOptions (ListenerPolicy and HTTPListenerPolicy), mapping to Envoy's HCM http2_protocol_options.allow_connect, to support WebSocket-over-HTTP/2 (RFC 8441 Extended CONNECT). (#14323)
• Added support for excluding Istio ServiceEntries from kgateway ServiceEntry backend and endpoint discovery using Kubernetes label selectors. (#14328)
• TrafficPolicy retry configuration is now applied at the route level instead of the virtual host level, making retry policy precedence between TrafficPolicy and the builtin HTTPRouteRetry policy explicit and consistent. (#14360)
• Added connectionKeepalive to BackendConfigPolicy.spec.http2ProtocolOptions, enabling Envoy HTTP/2 keepalive PINGs for active half-dead upstream connection detection. (#14367)
• Added default-off support for ordered ADS delivery via KGW_ENABLE_ORDERED_ADS=true, preserving CDS, EDS, LDS, and RDS response order for same-snapshot busy-stream additions. (#14372)
• Added a new priorityGroups backend type to the Backend API, enabling active/passive failover between static backends. Each group in the ordered list becomes an Envoy priority level; traffic fails over to the next group when the preceding groups' endpoints are unhealthy, and recovers automatically. Combine with an active health check via BackendConfigPolicy to drive failover. (#14379)
• added get cookie functions to transformation template funcion (#14385)
• Added an opt-in statPrefix field to TrafficPolicy that sets the Envoy route stat_prefix, enabling per-route statistics. The value supports {{route_name}}, {{route_namespace}}, and {{rule_name}} template variables and is only honored for HTTPRoute and GRPCRoute targets. (#14409)
• Added a libraries field to TrafficPolicy.compression.requestDecompression to decompress request bodies encoded with gzip, brotli, or zstd. (#14431)

Bug Fixes

• Fix a reconnect-time xDS race where Envoy could briefly receive routes and listeners before all referenced backend clusters were present, causing transient NC/500 responses during controller restart. (#13868)
• The controller Helm chart omits Prometheus annotations only if explicitly configured to do so, whereas before any ServiceMonitor usage would cause their omission. (#13981)
• implemented http-acl policy merge logic (#13996)
• Fixes strict validation on delegating routes (#14000)
• Fixed Gateway deployment apply operations to respect controller cancellation, preventing in-flight resource patches from continuing after the controller is shutting down. (#14013)
• Fix a bug that can lead to multiple leaders during upgrades (#14017)
• Fix STRICT validation for BackendConfigPolicy when tls.wellKnownCACertificates is set to System. (#14037)
• Fix xDS TLS env var name in Helm chart: KGW_XDS_TLS_ENABLED is renamed to KGW_XDS_TLS to match the runtime settings key. Users who set controller.xds.tls.enabled: true will now have xDS TLS correctly enabled without requiring a manual extraEnv workaround. (#14050)
• Fixed BackendTLSPolicy status reporting for TCPRoute and terminated TLSRoute backends. (#14071)
• Fixed HTTPRoute RequestRedirect handling so redirect Location headers no longer include default ports (:80 for HTTP, :443 for HTTPS). (#14086)
• Backend translation errors are now surfaced on the Backend's Accepted status condition. (#14113)
• bundled dynamic module in controller image to fix strict validation (#14119)
• Always set Path attribute to / for cookie-based session persistence.
  Note: This intentionally differs from the behavior that GEP-1619 describes. If you wish to mimic the per-matched-path session persistence specified there, use multiple rules with different cookie names.
  (#14124)
• kgateway no longer overwrites an existing Kubernetes Service for a Gateway unless the Service has a matching Gateway ownerReference or kgateway ownership metadata. (#14145)
• fix: Allows all X.509 certificates that Envoy itself allows, regardless of the sign of the serial number. (#14154)
• Fix global rate limit descriptor translation: multiple descriptors now correctly produce separate Envoy RateLimit actions instead of being merged into a single combined action. (#14157)
• Register the xDS TLS certificate watcher with the controller manager so that failures to start it are surfaced as fatal errors rather than silently disabling certificate rotation. (#14187)
• Fixed a bug where a BackendConfigPolicy health check host (HTTP host or gRPC authority) was ignored for Static backends because the endpoint-level hostname overrode it, causing health checks to use the wrong Host header. (#14201)
• Fixed route Hostname/ServiceEntry backendRefs to the requested port on multi-port hosts (#14212)
• Advertise support for the Gateway API BackendTLSPolicySANValidation conformance feature. (#14214)
• ServiceEntry clusters with workloadSelector-backed pod endpoints now respect pod readiness: NotReady pods are excluded from routing, matching the EndpointSlice-based Service behavior and enabling locality failover when all locally selected pods are NotReady. WorkloadEntry and inline ServiceEntry endpoints are unaffected. (#14222)
• Fixed EC2 backend discovery serving endpoints resolved under an outdated config (e.g. the old port) for up to a refresh interval after a Backend spec change. Spec changes and newly created EC2 backends now trigger an immediate discovery refresh, and a credential rotation combined with a transient AWS API failure no longer drops healthy endpoints. (#14228)
• Fix XListenerSet status patch failures caused by missing required port field since v2.3.0. (#14231)
• fix: resolve FrontendTLS CA certificate references in the Gateway's namespace when listeners are contributed by a ListenerSet, rather than incorrectly looking in the ListenerSet's namespace. (#14232)
• Strict validation (KGW_VALIDATION_MODE=STRICT) now caches validation verdicts by config content
  hash, eliminating redundant envoy invocations across per-client translation and recomputes. New
  settings: KGW_VALIDATOR_MODE (CACHE [default] | BINARY) and KGW_VALIDATOR_CACHE_SIZE (default
  4096).
  (#14253)
• Fix ListenerPolicy with clientCertificateValidation not being marked as Attached if there are other policies applied to the same target (#14278)
• Fix excessive DNS queries from gateway pods by rendering the xDS cluster address as a rooted FQDN (trailing dot), preventing DNS search-domain expansion under the default ndots:5 resolver config. (#14291)
• Fix route status reporting and ensure stale status is cleared for GRPCRoute, TCPRoute, and TLSRoute resources. (#14294)
• Fixed a bug where a Gateway, Route, Backend, or ListenerSet status observedGeneration could intermittently freeze at a stale value after a spec change, due to a skew between the translation cache and the status syncer's cache. (#14302)
• Fixed two HTTPListenerPolicy/DirectResponse cases where an IR field could be compared by pointer identity instead of value, causing spurious re-translations. (#14331)
• Fixed north-south traffic to a global service continuing to route to a local pod for the duration of its termination grace period. kgateway now drains a terminating pod (deletionTimestamp set) from ServiceEntry-derived endpoints and fails over to a healthy peer-cluster endpoint, consistent with the east-west (ztunnel) path and the Kubernetes Service backend path. (#14332)
• Fixed a bug where a TrafficPolicy timeout (and other route-level settings) targeting a Gateway HTTP listener section was silently ignored. (#14374)
• TrafficPolicy retry may now target a whole Gateway (without a sectionName), consistent with timeouts. TrafficPolicy urlRewrite is now validated to only target HTTPRoute resources, matching its documented behavior. (#14378)
• Fix a bug where per-client xDS snapshots could be withheld indefinitely (reverts #13868/#13958), stranding running gateway pods on stale endpoints and starving newly created pods of any configuration until they crashloop. The control plane now publishes snapshots unconditionally and instead delays each newly connected client's first xDS response by a short grace period (default 1s, configurable via KGW_XDS_FIRST_CONNECT_DELAY; 0 disables) so per-client translation can converge before the first snapshot is served. Trade-off: in environments where per-client translation exceeds the grace period, a controller restart can once again cause a brief, self-healing burst of upstream-connect (NC) errors on reconnecting proxies; raise KGW_XDS_FIRST_CONNECT_DELAY if you observe this. (#14380)
• Fix 503s for ingress-use-waypoint routes to auto-allocated ServiceEntries. (#14391)
• Fixed a ~15 second delay before freshly started gateway proxies receive their initial xDS configuration. (#14433)
• Fixed a regression where a TrafficPolicy using urlRewrite, autoHostRewrite, or tracing without targetRefs/targetSelectors (e.g. attached via an HTTPRoute ExtensionRef filter) was rejected by CRD validation. (#14435)

Cleanup

• Added kgateway validation metrics for Envoy validation calls, cache behavior, results, and duration by validation caller. (#14026)
• bump alpine base image used to build images (#14051)
• Reduced controller memory usage by interning policy ref ID strings retained in policy merge tracking. (#14217)
• Fixed GatewayExtension equality to include the object source, and Listener equality to ignore parent object metadata churn (e.g. resourceVersion bumps from status writes), preventing missed updates and spurious recomputation. (#14248)
• Improved strict HTTPRoute validation performance by batching full-route Envoy validation per virtual host. (#14269)
• Refactor: replace sort package with slices package for cleaner sorting code. (#14356)

Dependency Updates

• upgraded envoy to 1.38.1 (#14004)
• Default Istio proxy version bumped. (#14056)
• Because v2.3.0 has been released, v2.4.0-main is now the rolling tag for the very latest main prerelease artifacts. (#14099)
• Updated dependencies. (#14103)
• upgrade envoy to v1.38.3 (#14314)
• Kgateway now supports Gateway API v1.6.1 (#14426)

Contributors

Thanks to all the contributors who made this release possible: