github theagentrouter/agent-router v1.3.0

2 hours ago

Agent Router v1.3.0

Plain-markdown copy of the v1.3.0 release notes, suitable for pasting into the GitHub release body. The full rendered version lives at https://theagentrouter.ai/release-notes/v1.3.

Agent Router v1.3.0 makes QuotaPolicy a complete token-budget system. Per-model quotas are now charged accurately and applied without restarts. Limits can come from request metadata, windows can be weekly, monthly, or yearly, policies can target backends in other namespaces, and routes can fail over away from backends whose quota is exhausted. The release also adds the OpenAI Files API and the Decisions API, OpenAI workload identity federation for BackendSecurityPolicy, Claude Sonnet 5.5 between-tools thinking, and moves to Envoy Gateway v1.10.0 with Envoy Proxy v1.40.2. A few provider translations and the Go API changed, so read the breaking changes before you upgrade.

✨ New Features

QuotaPolicy, End to End

  • Accurate token charging — A request used to be charged one extra unit when it was admitted, on top of its real token cost when the response finished, so a 28-token request deducted 29. The admission check now charges zero, and only the token cost reported at the end of the response or stream is deducted. Budgets now match the usage your providers bill for.
  • Policy changes apply live — Creating, updating, or deleting a QuotaPolicy used to take effect only after the controller and the Envoy pods restarted. The controller now stamps a hash of the applicable policies on the generated HTTPRoute (aigateway.envoyproxy.io/quota-policy-hash), so Envoy Gateway re-translates and the new limits, models, and bucket rules apply within seconds. Deleting a policy also updates routes in other namespaces.
  • Per-tenant limits from request metadata with fromMetadata — Serve many tenants from one policy by letting an earlier filter, usually ext_authz, decide each tenant's budget. Set fromMetadata.namespace and fromMetadata.key on any QuotaValue, and have the filter write requests_per_unit and unit (for example HOUR) to dynamic metadata. The static limit and duration stay required and apply when the metadata is missing or malformed. A value of 0 suspends the tenant. Dynamic bucket limits
  • Weekly, monthly, and yearly windows — duration now accepts 1w, 1mo, and 1y in addition to 1s, 1m, 1h, and 1d, so you can express weekly team budgets or monthly spend caps directly. These are Envoy rate limit windows, not calendar-aligned billing periods. Your rate limit service image must support the WEEK, MONTH, and YEAR units (see Upgrade Guidance).
  • Cross-namespace quota targets — A platform team can keep budgets in its own namespace while backends live with the teams that own them. Set targetRefs[].namespace to point a QuotaPolicy at an AIServiceBackend in another namespace, and authorize it with a ReferenceGrant in the backend's namespace (from QuotaPolicy, to AIServiceBackend). Omitting the namespace keeps the previous same-namespace behavior and needs no grant.
  • Quota-aware backend failover (preview) — When a route has several prioritized backends and one runs out of quota, Agent Router can now route the request to the next backend that still has budget instead of returning 429. The quota rate limit service reports which (backend, model) groups still have quota, and Agent Router wraps quota-backed clusters in Envoy's host_allowlist load balancing policy so exhausted backends are skipped. A 429 is returned only when every quota group on the route is exhausted. This needs an Envoy build that includes host_allowlist, which Envoy v1.40.2 (shipped by Envoy Gateway v1.10.0) does not, and a rate limit service with RESPONSE_DYNAMIC_METADATA=true. On other data planes the route falls back to its normal load balancing and quotas are still enforced.

OpenAI APIs & Providers

  • OpenAI Files API — Upload, list, retrieve, download, and delete files through the gateway with /v1/files, so batch and fine-tuning workflows no longer need direct provider access. Uploads route on the multipart model field. The gateway never returns the upstream file ID. It returns an encrypted ID that records the owning backend and model, so later requests route back to the right place. Files work with OpenAI-schema backends and need exactly one backendRef on the matching rule. Files API
  • Decisions API on OpenAI and Azure OpenAI — POST /v1/decisions is passed through to OpenAI, and to Azure OpenAI at /openai/v1/decisions, with usage tracking, metrics, and tracing under the new decisions operation. Question instructions can be a string or a list of text and image parts. Streaming and file_id inputs are not supported.
  • Claude Sonnet 5.5 between_tools thinking — Send "thinking": {"type": "between_tools"} to skip up-front thinking while keeping short progress updates between tool calls. It works on /v1/messages and Chat Completions for Anthropic, GCP Anthropic, AWS Anthropic, and AWS Bedrock Converse backends. Messages requests to Bedrock Converse now also forward adaptive thinking.
  • Structured output and effort for Sonnet 5.5 and Haiku 5.5 — Chat Completions requests to sonnet-5-5 and haiku-5-5 now forward reasoning_effort as Anthropic output_config.effort on AWS and GCP Anthropic, and response_format JSON schemas as output_config.format on GCP Anthropic. These were silently dropped before.
  • reasoning_effort: high on Gemini 3 Pro — reasoning_effort: high now maps to thinking_level: high on every Gemini 3 model instead of returning 400 on non-Flash models. See Breaking Changes for the related change to medium.

Credentials & Security

  • OpenAI workload identity federation — Stop storing long-lived OpenAI API keys. A BackendSecurityPolicy of type: OpenAICredentials exchanges a SPIFFE JWT-SVID or an OIDC token for a short-lived OpenAI access token, rotates it before it expires, and sets the optional OpenAI-Organization and OpenAI-Project headers. Set AI_GATEWAY_OPENAI_PROXY_URL on the controller to send the exchange through a proxy. Every SPIFFE-based policy authenticates as the controller's identity, so only trusted administrators should be allowed to create them.
  • Configurable credential pre-rotation window — controller.credentialPreRotationWindow (default 5m) sets how long before expiry the controller refreshes AWS, Azure, GCP, and OpenAI tokens. Raise it when your identity provider is slow or rate-limits token requests.
  • MCP OAuth resource identifier derived from the request — protectedResourceMetadata.resource on an MCPRoute is now optional. Leave it out and the gateway builds the identifier from each request's scheme, host, and MCP path, so one route can serve several hostnames or environments. Restrict the accepted hosts with route or listener hostnames. Routes that set it keep exactly the same behavior.
  • Bring-your-own Secret for the MCP fallback seed — controller.mcp.sessionEncryption.fallback.existingSecret reads the fallback session encryption seed from a Secret you manage, so GitOps users can rotate seeds without the seed appearing in Helm values.

Inference & Operations

  • Weighted routing across multiple InferencePools — An HTTPRoute rule can now list several InferencePool backends with Gateway API weights, for example to shift traffic between model server fleets. Each pool keeps its own endpoint picker. This needs Envoy Gateway v1.11 or later. On Envoy Gateway v1.10, a rule with more than one InferencePool returns 500. AIGatewayRoute rules still accept one InferencePool.
  • One endpoint picker call per request — A pool reached by several route matches used to get one endpoint picker filter per match, so each request ran several scheduling passes. That broke stateful scorers such as prefix-cache affinity. The extension server now adds one filter per pool.
  • More reliable reconciliation — Finalizer writes use merge patches and are retried, so resources no longer end up without a finalizer or stuck terminating after a conflict. InferencePool status updates are retried too, so status.parents lists every Gateway.

🔗 API Updates

  • QuotaPolicy.spec.targetRefs[].namespace — targetRefs changed from LocalPolicyTargetReference to NamespacedPolicyTargetReference. The optional namespace defaults to the policy's namespace. A cross-namespace target needs a ReferenceGrant, and an ungranted target is reported with the new RefNotPermitted reason. v1alpha1 only.
  • QuotaValue.fromMetadata — Optional {namespace, key} (each 1-256 characters, ^[A-Za-z0-9._:-]+$) naming the dynamic metadata struct that overrides limit and duration per request. limit and duration are now explicitly required. fromMetadata under serviceQuota is accepted but has no effect.
  • QuotaValue.duration: 1w, 1mo, 1y — Three new enum values.
  • BackendSecurityPolicy type: OpenAICredentials — New openAICredentials field with optional organization and project and a required workloadIdentityFederationConfig (identityProviderID, serviceAccountID, and a subjectToken that is exactly one of spiffeJWTSVID or oidcExchangeToken). An OIDC client Secret in another namespace needs a ReferenceGrant. v1beta1 only.
  • MCPRoute protectedResourceMetadata.resource is optional — No longer required in v1alpha1 or v1beta1. When it is unset, the identifier is derived from the request.
  • Embedded Envoy Gateway v1.10 types — The CRDs pick up new optional fields from Envoy Gateway v1.10, such as OIDC cookieNames and codeVerifierTTL under oidcExchangeToken, and http3, enableTrailers, and failTrafficOnPanic in backend settings. CRD descriptions are now truncated to 200 characters to keep the CRD Helm release under the 1 MiB Secret limit. No schema types or validation changed.
  • New Helm values — controller.credentialPreRotationWindow, controller.fileIDEncryption.{seed, existingSecret, iterations, fallback.seed, fallback.iterations}, and controller.mcp.sessionEncryption.fallback.existingSecret.

Deprecations

  • cache_creation_input_tokens in translated OpenAI-format responses — Still returned alongside cache_write_tokens in v1.3. Read cache_write_tokens instead. The legacy field was scheduled for removal in v1.3.0, and removal has moved to v1.4.0.

⚠️ Breaking Changes

  • Legacy Gemini schema conversion removed — Chat Completions and tokenize requests to GCP Vertex AI now always send response_format schemas as responseJsonSchema and tool parameters as parametersJsonSchema, unchanged. Before, models whose name (or modelNameOverride) didn't contain gemini plus 2.5 or 3 got a converted OpenAPI-subset schema. If you route to Gemini through an alias without a modelNameOverride, or still use a pre-2.5 model, check that your schemas work as standard JSON Schema.
  • Gemini reasoning_effort: medium mapping — medium now maps to thinking_level: medium on every Gemini 3 model except gemini-3-pro* and Flash-Lite Image, which get high. Before, every non-Flash model got high, so models such as Gemini 3.1 Pro now think less at medium. Send high to keep the previous behavior.
  • gen_ai_backend metric label uses the backend's namespace — The backend metric attribute is now the AIServiceBackend's own <namespace>/<name>. Before, it used the route's namespace, so the value changes for routes that reference a backend in another namespace. Update dashboards and alerts that filter on the old value.
  • MCP OAuth protected resource metadata is served by the external processor — /.well-known/oauth-protected-resource/... is now answered by the MCP proxy in the external processor sidecar instead of an Envoy direct response. It returns 503 while the route isn't loaded, for example during an external processor restart. The controller deletes the old HTTPRouteFilter on upgrade. If a later release drops that cleanup, users upgrading directly from v1.2 or earlier will need to delete the stale filter by hand.
  • GCPWorkloadIdentityProvider Go type removed — The unused type was removed from the api/v1alpha1 and api/v1beta1 Go packages. No CRD field referenced it, so manifests are unaffected. Only Go code that imports the type needs to change.

🐛 Bug Fixes

  • GCP workload identity federation survives config reloads — With ADC credentials of type external_account (for example from EKS or AKS), every GCP request failed with context canceled after the first config reload until the external processor restarted. The token source now outlives the reload.
  • Anthropic overload errors arrive in the stream — When Anthropic sends overloaded_error mid-stream on a Chat Completions request to AWS or GCP Anthropic, the client now receives a final SSE error event, so it can retry. Before, it got a generic 500 or a truncated stream.
  • Bedrock ping events are forwarded — /v1/messages streams to AWS Anthropic backends now forward ping keepalive events, so clients such as Claude Code no longer hit idle timeouts during long tool calls.
  • Bedrock guardrail stops report content_filter — A Bedrock guardrail_intervened stop reason now maps to finish_reason: content_filter instead of stop, so clients can tell a guardrail block from a normal completion.
  • Anthropic passthrough keeps the query string — /v1/messages requests to an Anthropic-schema backend now forward the client's query string, for example ?beta=true.
  • More request bodies redacted in debug logs — With redaction enabled, debug logs now redact /v1/responses/input_tokens, /v1/messages/count_tokens, and tokenize chat_template_kwargs.

📖 Upgrade Guidance

Upgrading from v1.2 needs no CRD storage migration, and every new field is optional. Upgrade from v1.2.x, not directly from v1.1.x.

1. Upgrade Envoy Gateway first

Gateway API stays at v1.6.1. Upgrade the Envoy Gateway CRDs and controller to v1.10.0:

helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \
  --version v1.10.0 \
  --set crds.gatewayAPI.enabled=true \
  --set crds.envoyGateway.enabled=true \
  | kubectl apply --force-conflicts --server-side -f -

helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.10.0 \
  --namespace envoy-gateway-system \
  -f https://raw.githubusercontent.com/theagentrouter/agent-router/v1.3.0/manifests/envoy-gateway-values.yaml

Envoy Gateway v1.10 has its own breaking changes. Read its v1.10.0 release notes. The ones most likely to affect AI traffic:

  • Rate-limited 429 responses now include Retry-After by default. Set ClientTrafficPolicy.spec.headers.disableRetryAfterHeader: true to turn it off.
  • JWT requirement names change once on upgrade. On MCPRoutes with OAuth, requests on connections opened before the upgrade can get 403 Wrong requirement_name until those connections drain.
  • OIDC session cookies move to AES-256-GCM, so users with an active OIDC session sign in again once. HTTP OIDC issuer URLs are rejected.
  • Strict Lua validation now denies filesystem and environment access by default.

2. Keep Files API IDs stable

The chart now creates a random file ID encryption seed in the Secret <controller fullname>-file-id-encryption and reuses it on later helm upgrades. If you deploy with helm template or GitOps, the chart can't read back the seed it generated, so it generates a new one on every render and invalidates file IDs it has already issued. Supply your own Secret instead:

kubectl create secret generic file-id-seed -n envoy-ai-gateway-system \
  --from-literal=seed="$(openssl rand -hex 32)"

Then set controller.fileIDEncryption.existingSecret: file-id-seed. To rotate the seed later without breaking existing IDs, move the old seed to controller.fileIDEncryption.fallback.seed.

3. Update the quota rate limit service

If you use QuotaPolicy:

  • 1w, 1mo, and 1y windows need a rate limit service image that supports the WEEK, MONTH, and YEAR units. Agent Router's CI uses docker.io/envoyproxy/ratelimit:63c7e9cc. Older images, including 60d8e81b from the token_ratelimit example, don't support these units.
  • To try quota-aware failover, run that image with RESPONSE_DYNAMIC_METADATA=true, and use an Envoy build that includes the envoy.load_balancing_policies.host_allowlist load balancing policy. Envoy v1.40.2 doesn't include it. Without it, routes keep their normal load balancing and quotas are still enforced.
  • Quota counters are now charged exactly once per request. Clients that sat just under a limit get slightly more headroom.
  • serviceQuota is still not enforced. Use per-model quotas.

4. Check Gemini requests and dashboards

  • If you route to Gemini through an alias without modelNameOverride, or still use a pre-2.5 model, check that your response_format and tool schemas are valid standard JSON Schema.
  • If you send reasoning_effort: medium to Gemini models other than 3 Pro and Flash, send high to keep the previous behavior.
  • If a route references an AIServiceBackend in another namespace, update dashboards and alerts that filter on gen_ai_backend. The label now uses the backend's namespace.

5. Upgrade Agent Router

helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \
  --version v1.3.0 \
  --namespace envoy-ai-gateway-system

helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \
  --version v1.3.0 \
  --namespace envoy-ai-gateway-system

The controller pod restarts because its arguments and environment changed. If you're upgrading from v1.2.0 rather than v1.2.1, Gateway pods that serve MCPRoutes also roll once.

📦 Dependency Versions

Dependency Version
Go 1.27.3
Envoy Gateway v1.10.0
Envoy Proxy v1.40.2
Gateway API v1.6.1
Gateway API Inference Extension v1.6.2
MCP Go SDK v1.7.0

🙏 Acknowledgements

v1.3 came from 34 contributors. Special thanks to:

  • Hrushikesh Patil, Ayush Sawant, Fernando Escolar, and Gavrish Prabhu, who took QuotaPolicy end to end with dynamic limits, cross-namespace targets, long windows, live updates, accurate charging, and quota-aware failover.
  • Sivanantham, who built the OpenAI Files API.
  • hustxiayang and Mike Toldov, who added the Decisions API, Claude Sonnet 5.5 thinking, and a run of translator fixes.
  • junfeiliu531, who added OpenAI workload identity federation.
  • zirain and Gerrit van Doorn, who improved InferencePool routing.
  • Ignasi Barrera and Aishwarya Raimule, who continued hardening MCP and its OAuth support.
  • Everyone who reported bugs, reviewed PRs, and joined the community meetings.

🔮 What's Next

Work already in flight for upcoming releases:

  • Service-wide quotas, enforcing serviceQuota across all models on a backend.
  • Quota-aware failover on released Envoy versions, once host_allowlist ships in an Envoy Gateway release.
  • Weighted InferencePool routing with Envoy Gateway v1.11.
  • The stateless 2026-07-28 MCP specification, switched on end to end.
  • Removing cache_creation_input_tokens from translated OpenAI-format responses.

The roadmap is community-driven. Join us and help shape it.

Don't miss a new agent-router release

NewReleases is sending notifications on new releases.