envoyproxy/ai-gateway v1.1.0

on GitHub

Envoy AI Gateway v1.1.0 is the first minor release on the stable 1.x API. It adds token counting across providers, per-request upstream credentials, stream idle timeout with failover, MCP hostname routing and CEL backend selection, optional OpenTelemetry GenAI tracing, and HTTP CONNECT egress. Upgrading from v1.0 needs no CRD migrations; the only operator-facing change is Helm's restricted controller security context.

✨ New Features

Token Counting APIs

• vLLM-compatible /tokenize across providers — Count tokens for a prompt without generating a completion. Chat messages and completion prompts use the vLLM tokenize request shape against vLLM, Vertex AI Gemini, GCP Anthropic, AWS Bedrock Converse, and AWS Anthropic. Native Anthropic clients can use /anthropic/v1/messages/count_tokens; Responses API clients can use /v1/responses/input_tokens (OpenAI and Azure OpenAI).
• Anthropic-format /anthropic/v1/models — Claude clients that call Anthropic's model list endpoint now get the gateway's configured catalog in Anthropic's response shape, scoped by hostname the same way /v1/models is.

Per-Request Credentials & Egress

• Per-request upstream credentials via credentialOverride — A trusted filter can supply the backend credential per request instead of sharing one static key in BackendSecurityPolicy. Source from Envoy dynamic metadata (preferred) or a request header the gateway strips before the backend. API keys, Anthropic, Azure, GCP, and AWS SigV4 are supported.
• HTTP CONNECT forward proxy — GatewayConfig.spec.forwardProxy tunnels upstream LLM connections through an HTTP CONNECT proxy (host:port). Bedrock requests through PrivateLink now sign over the actual upstream host, so VPC endpoints no longer fail with a silent 403.

Streaming Resilience

• Stream idle timeout with failover — AIGatewayRouteRule.streamIdleTimeout bounds how long Envoy waits without upstream bytes on a streaming response. If it fires before the first token, a retry policy can fail over to the next backend; mid-stream it returns 504. Pair with a BackendTrafficPolicy whose retry covers reset.

MCP Gateway

• Hostname routing and backend selection — MCPRoute.spec.hostnames scopes an MCP endpoint to specific hosts. backendSelector restricts which backends a session fans out to via CEL rules evaluated at initialize (default Deny). Initialize now advertises merged backend capabilities, and the controller shares one Envoy Gateway Backend per namespace.

Observability

• OpenTelemetry GenAI conventions and Grafana dashboard — Set AI_GATEWAY_TRACING_SEMCONV=gen_ai on the ext-proc to emit gen_ai.* span attributes. An example dashboard for gen_ai_* Prometheus metrics ships in examples/monitoring/grafana-dashboard.json. Controller and ext-proc logs accept --logFormat=json; MCP resource reads populate mcp_resource_uri in access-log metadata.

Provider Translation & API Compatibility

• Structured output for Claude on Vertex AI — JSON-schema constrained decoding now works for Claude models on Vertex AI that advertise output_config. OpenAI reasoning_effort is forwarded to Bedrock as reasoning_config.
• Responses API tool_search, unknown tools, and Codex inputs — The Responses API accepts the built-in tool_search tool and additional_tools, preserves unknown tool types, and accepts Codex-style agent input items.

Helm & Operations

• Controller hardening and sharded config secrets — The chart can emit a PDB, honour topologySpreadConstraints, and attach extra podLabels. Filter configuration is split across multiple Secrets so large Gateway configs no longer hit Kubernetes' 1 MiB Secret size limit.

🔗 API Updates

• AIGatewayRouteRule.name — Optional unique name copied onto the generated HTTPRoute rule. route-not-found is reserved.
• AIGatewayRouteRule.streamIdleTimeout — Optional Gateway API duration; unset means no per-try idle timeout.
• BackendSecurityPolicy.spec.credentialOverride — Optional per-request credential source. Exactly one of fromRequestHeaders or fromDynamicMetadata. fallbackToConfigured defaults to true.
• GatewayConfig.spec.forwardProxy — Optional HTTP CONNECT proxy. address is required (host:port).
• MCPRoute.spec.hostnames — Optional list of hostnames (max 16).
• MCPRoute.spec.backendSelector — Optional CEL selector evaluated at session initialize. defaultAction defaults to Deny.

⚠️ Breaking Changes

• Helm controller security context defaults — The Helm chart now sets a restricted podSecurityContext and container securityContext on the controller (non-root UID/GID 65532, all capabilities dropped, no privilege escalation, RuntimeDefault seccomp). Previously both were empty. If you run a custom controller image that cannot use UID 65532, override controller.podSecurityContext and controller.securityContext in your values.

📖 Upgrade Guidance

Upgrading from v1.0 requires no CRD migrations. New fields are optional and backward compatible. Review the Helm security-context change if you override the controller image.

If the official controller image works for you, no action is needed. To restore empty security contexts:

controller:
  podSecurityContext: {}
  securityContext: {}

JSON logs: set controller.logFormat and extProc.logFormat to json. Enable a PDB with controller.podDisruptionBudget.enabled: true when you run more than one replica.

📦 Dependency Versions

🙏 Acknowledgements

Thank you to everyone who made v1.1 possible.