maximhq/bifrost core/v1.9.0

Core v1.9.0

on GitHub

Core Release v1.9.0

• [fix]: MCP tool schemas keep one property order across tool syncs @dougcalobrisi
• feat: WhiteList and BlackList entries that start with regex: are RE2 patterns. MatchEntry compiles them once as a case-insensitive full match, Validate refuses a pattern that is empty, * or does not compile, and the list-models pipeline never surfaces a pattern as a model. Plain entries keep their exact, case-insensitive match
• fix: fasthttp-level stale-connection retries no longer multiply max_retries. contextTransport.RoundTrip reports a failure before response headers on a freshly dialed socket with retry=false, so network.StaleConnectionRetryIfErr only walks past pooled keep-alive sockets the upstream closed while idle; an upstream that closes a fresh connection without answering now costs exactly one attempt of Bifrost's own retry budget instead of up to four. The retry backoff also ends as soon as the request context is cancelled, so a worker is freed immediately instead of after up to retry_backoff_max (#7035)
• [fix]: Gemini provider - preserve inline image and audio data in chat completions responses @Atharva-Kanherkar
• [fix]: route opencode-zen Responses calls through /v1/chat/completions, which its upstream does not serve on /v1/responses @miguelchico
• feat: the namespace-tool names a provider reserves for its own server tools (web, image_gen, browser, python on Bedrock and Bedrock Mantle) are now read from the datasheet row reserved_tool_namespaces (new ModelCapabilities field and ModelCaps.ReservedToolNamespaces) for the base provider and canonical model; a non-empty row replaces the hardcoded list, an absent or empty row keeps it, so any OpenAI-wire provider can reserve names through a row without a code change
• fix: Codex >= 0.147 wraps its default tools in a namespace literally named functions (openai/codex#37022), which Bedrock Mantle reserves and rejects with User-defined namespace 'functions' collides with an existing tool namespace. Core now unwraps that namespace to top-level tools, unprefixed, for every provider before dispatch; Codex treats the bare name and the explicit functions namespace as the same tool, so no mapping back is needed; a description on the functions namespace is prepended to each hoisted member, as flattening does
• fix: Responses namespace tools are now flattened in core for every provider whose wire does not support the type (Anthropic, Gemini, Vertex, Bedrock Converse, DeepSeek, and every OpenAI-compatible third party), with nested functions renamed to <namespace>__<function> so two namespaces that share a function name no longer collide into an upstream Tool names must be unique 400; returned function_call items are mapped back to the bare name plus namespace (unary and streaming), prior-turn calls carrying namespace and tool_choice names are re-aliased to match, and a name that is still duplicated after flattening or a tool_choice that matches several namespaces is rejected with a clear 400 before reaching the provider. Bedrock answers namespace support from its own surface resolver via the new optional ResponsesNamespaceToolProvider interface. A datasheet row supports_namespace_tools (new ModelCapabilities field and ModelCaps.SupportsNamespaceTools) overrides the per-provider default for a (provider, model) pair; with no row the default applies. A row can only narrow within what the wire can carry: the Anthropic Messages API (Anthropic, Claude on Azure or Bedrock Mantle), the Gemini API (Gemini, Vertex) and Bedrock Converse have no namespace container, so they answer false regardless of the row and always flatten. Flattened names honour the target wire's documented tool-name limit, 64 characters of [A-Za-z0-9_-] for OpenAI-compatible wires, Bedrock Converse and Fireworks, 128 for Anthropic, and 128 with . and : allowed for Gemini and Vertex, overridable per model through the datasheet row tool_name_max_length (a row below 10 cannot hold the hashed form and is ignored); a longer alias becomes an 8-hex hash prefix plus the function name, deterministically, so history and tool_choice re-alias to the same string. The alias map travels on the prepared request (BifrostResponsesRequest.NamespaceToolAliases) and is applied to that attempt's response, unary and streaming; nothing is kept on the request context or in process-wide state (#7048)
• fix: drop namespace tools whose name Amazon Bedrock reserves (web, image_gen, browser, python) on the Bedrock and Bedrock Mantle Responses paths instead of forwarding them into a tools.namespace collision 400; a dropped Codex web namespace becomes the hosted web_search tool on Bedrock Mantle
• fix: Bedrock Mantle chat streaming no longer drops the usage-only chunk that arrives after finish_reason; ProviderSendsDoneMarker now treats bedrock_mantle (and the legacy Mantle route under the bedrock key) as sending [DONE], so streamed usage and cost are recorded (#7065)
• fix: default_request_timeout_in_seconds now bounds the wait for response headers on streaming requests, and cancelling a request now closes the upstream socket. Every fasthttp client is driven through a Bifrost-owned RoundTripper (core/providers/utils/roundtripper.go) that applies the client's read/write timeouts and the request context to the request write and the header wait, then lifts the socket deadline once headers are parsed so stream_idle_timeout_in_seconds remains the only bound on the body. BuildStreamingClient and BuildLargeResponseClient no longer zero the timeouts. An upstream that accepts the connection and never answers now fails with 504 RequestTimedOut after the request timeout and the request's fallbacks are used, instead of pinning the provider worker until the upstream closes the socket; MakeRequestWithContext no longer leaves a background client.Do running until ReadTimeout after its context is cancelled (#7034)
• fix: unary large-response downloads (FinalizeResponseWithLargeDetection) now bound every body read with stream_idle_timeout_in_seconds and close the upstream socket on request cancellation, mirroring the streamed passthrough path; a stall during the prefetch surfaces as 504 RequestTimedOut instead of pinning the provider worker, and a stall while the transport drains the LargeResponseReader fails the read with the idle-timeout error instead of blocking the writer indefinitely (#7104)
• fix: gzip-encoded unary responses are now classified for large-response mode by decompressed size; Content-Length describes the compressed bytes, so a body that was small on the wire but large once decompressed used to be materialized in full past large_response_threshold (#7104)
• fix: an OpenAI-compatible upstream that omits [DONE] and then goes silent after finish_reason no longer fails the stream when stream_idle_timeout_in_seconds fires. The chat and text completion read loops now treat an idle timeout after a terminal signal as a parked upstream, mark the stream parked so the connection is abandoned rather than drained, and synthesize the final chunk with the buffered finish_reason; a stall before finish_reason still surfaces as the idle-timeout error (#7108)
• fix: a streamed upstream that drops the connection mid-body is again reported as the retryable 502 provider closed the stream before sending a completion marker error instead of a generic Error reading stream: unexpected EOF. The Bifrost round tripper's chunked decoder surfaced the drop as io.ErrUnexpectedEOF, which no provider read loop treats as end of stream; it now reports the plain io.EOF fasthttp always did and discards the half-read connection (#7104 follow-up)
• fix: fallbacks for image edit, image variation and video edit requests now reach the configured fallback provider and model. prepareFallbackRequest had no arm for those three types, so the shallow request copy kept the primary's sub-request pointer and the "fallback" attempt was routed back to the primary while RoutingInfo, the x-bifrost-routing-info-* headers, the log row and the fallback_index metric label reported it as a fallback. The helper now also verifies the prepared request targets the fallback provider and model and skips the fallback with a warning otherwise, so a future request type added without an arm fails loudly instead of silently re-running the primary (#6966)
• fix: route Bedrock Claude requests that carry a compact_20260112 edit to InvokeModel / InvokeModelWithResponseStream with the native Anthropic Messages body, so server-side compaction works on bedrock/ models, including keys that pin an inference-profile ARN. AWS documents compaction as unsupported on the Converse API, which previously received the edit and silently ignored it (#6825)
• feat: serve Anthropic tool search (tool_search_tool_*, defer_loading) on bedrock/ Claude models by routing those requests to InvokeModel / InvokeModelWithResponseStream, the only Bedrock API AWS allows it on; CountTokens counts such requests with the same InvokeModel body via the invokeModel input
• fix: a reasoning.context value the target model does not accept is now dropped on the OpenAI and Azure Responses path, so all_turns on the original gpt-5 family incl. gpt-5-pro, gpt-5.1 to gpt-5.3, and the o-series runs under the model's own current_turn default instead of failing with Unsupported value: 'all_turns' is not supported with the 'gpt-5-pro' model; gpt-5.4, gpt-5.5 and gpt-5.6 keep it. The accepted values come from the datasheet row supported_reasoning_contexts (new ModelCapabilities field and ModelCaps.SupportedReasoningContexts), falling back to a name-based default of auto and current_turn everywhere plus all_turns on gpt-5.4+
• fix: mid-conversation role: "system" messages are now inlined in place as <system-reminder> user turns for every model family on every converter that has a top-level system field: Bedrock Converse (Responses and Chat Completions), Gemini Chat Completions (systemInstruction), and the Anthropic wire shape used by DeepSeek, Fireworks and SGL. Previously only Claude on Bedrock/Anthropic inlined; everything else hoisted each reminder into the top-level system block. Claude Code appends a trailing <total_tokens> system reminder after every turn, so the hoisted block grew the front of the prompt each turn and prefix-based caches (Bedrock implicit caching for global.openai.gpt-5.6-luna on Converse, Gemini implicit caching, DeepSeek context caching) reported a full cache write and zero cache reads on every turn
• [fix]: vLLM key selection resolves the requested alias through key.Aliases per key before comparing it with vllm_key_config.model_name, so a valid key is no longer rejected when the request used an alias; the same public alias can map to different physical model IDs across vLLM instances, and allow and block checks keep matching the original user-facing alias @Constantine3
• [fix]: Gemini Responses requests keep top_k, frequency_penalty, presence_penalty, stop_sequences and media_resolution across retries and fallbacks. convertParamsToGenerationConfigResponses deleted those keys from ExtraParams while mapping them into generationConfig, and the conversion runs once per attempt on the same request, so every attempt after the first reached the provider without them; the outbound ExtraParams is now built through a filtered copy instead of mutating the request @VictorRequenaMaisa
• feat: wait_for_usage on custom_provider_config keeps the OpenAI-compatible chat and text completion read loops open past finish_reason until the trailing usage-only frame arrives, so a custom provider that also sets does_not_send_done_marker no longer synthesizes a zero-usage terminal chunk and records the request at zero tokens and zero cost. WaitForUsage is carried on BifrostContextKeyWaitForUsage with the same set-or-clear discipline as DoesNotSendDoneMarker, so it never leaks onto a fallback provider that did not declare it, and termination stays bounded by the usage chunk, two consecutive post-finish heartbeat comments, EOF, or stream_idle_timeout_in_seconds (#7143)
• fix: ExtraFields.RawResponse now carries role-only, finish-only and usage-only SSE frames on the OpenAI-compatible chat stream. Those frames never entered the chunk-forwarding branch, so their bytes were discarded from the reconstructed raw_response, which made the captured audit trail irreconcilable against a provider invoice because the usage frame carries the token counts Bifrost bills from. A pendingRawFrames buffer accumulates them and drains onto the next forwarded chunk or the synthetic terminal chunk; the Responses-over-Chat fallback no longer stamps one upstream frame onto every derived event, and delta.refusal and delta.annotations are added to the forwarding predicate so refusals and streamed URL citations are no longer dropped entirely (#7144)
• fix: Anthropic server-side tool search survives the Bedrock-native invoke ingress end to end. tool_search_tool_* was dropped during the Converse-shaped intermediate conversion, leaving the egress predicate blind to the tool and routing the request to Converse where server-side tool search is unavailable; the tool is now carried as an ingress-only marker (json:"-", never reaching a Converse body) and the neutral tool is rebuilt in ToBifrostResponsesRequest along with defer_loading. Results are returned as a server_tool_use plus tool_search_tool_result pair rather than a client tool_use, which Anthropic rejects when the caller echoes a tool_result for the srvtoolu_ ID on the next turn, on both the unary and streaming paths; BedrockContentBlock.UnmarshalJSON gained server_tool_use and tool_search_tool_result cases so replayed turns round-trip unchanged instead of falling through to empty structs; and the Anthropic-native response path carries the same blocks (#7155)
• fix: native Gemini generateContent image requests routed through /genai no longer return HTTP 200 while silently ignoring part of the request. isImageEditRequest only inspected contents[0].parts[0], so a request with the prompt text before the inlineData part, the ordering in Google's own REST edit sample, was misclassified as image generation: the image never reached Vertex and the model invented a picture from the prompt alone. Detection now scans every part across every content. imageConfig.aspectRatio is preserved as a typed aspect_ratio param that both outbound converters prefer, instead of being folded into a WxH size string that collapsed any ratio outside 1:1, 3:4, 4:3, 9:16 and 16:9 back to square (#7173)
• fix: Gemini Part gained mediaResolution, the per-part override for generationConfig.mediaResolution. Part.UnmarshalJSON decodes into a closed alias, so the key was discarded before any conversion ran and per-part image and PDF tokenization fell back to the model default; an ULTRA_HIGH image billed about 21k prompt tokens through /genai instead of about 22.1k direct. Both the mediaResolution and media_resolution spellings are parsed, the value round-trips through ResponsesMessageContentBlock and is reconstructed on the outbound part, and it is stripped on the OpenAI wire path where the field is unknown (#7156)
• fix: image blocks inside a Bedrock Converse toolResult are hoisted out and re-inserted immediately after the last tool result in the same message, with a placeholder text block left behind so the now image-free result is not rejected for being empty. Some Bedrock-hosted models, notably the OpenAI and Grok families, reject nested tool-result images on Converse even though they accept images in tool output via the Responses API. BedrockModelSupportsToolResultImages supplies the name-based default and the datasheet field supports_converse_tool_result_images (new ModelCapabilities field and ModelCaps.SupportsConverseToolResultImages) overrides it per row (#7150)
• feat: bifrost_error_requests_total gains a normalized error_type label drawn from a closed, prefix-structured vocabulary (caller_*, policy_*, provider_*, bifrost_*, _OTHER), so a 429 from a governance rate limit is distinguishable from a 429 from an upstream provider, a 403 from a blocked model from one from a bad key, and a 503 Bifrost shed under queue pressure from an upstream overload. ClassifyErrorType resolves a declared ExtraFields.ErrorType first, then Bifrost's own string markers, then the upstream status code; it deliberately ignores the provider's own error.type and error.code strings, which disagree across providers for the same condition. Values are prefixed by fault domain so an alarm expression can match a family with a single regex (#7141)
• fix: Azure Foundry deployments of Fireworks-hosted models are no longer silently capped at 4096 output tokens on /openai/v1/responses, where Microsoft routes those models through chat completions internally. The Azure provider checks the model's datasheet supported_endpoints and, when /v1/responses is absent, transparently serves both Responses and ResponsesStream through /openai/v1/chat/completions and converts the result back to the Responses shape; a model with an explicit /v1/responses entry or no datasheet row at all is unchanged. Separately, ToAnthropicResponsesResponse now follows StopReason > IncompleteDetails > tool_use inference > end_turn, so a turn truncated by the output-token cap on any OpenAI-shaped Responses provider reports stop_reason: max_tokens instead of hiding the truncation as end_turn (#6782)
• fix: DeepSeek chat requests move max_completion_tokens into max_tokens on the wire. DeepSeek's chat-completions endpoint only recognizes the legacy field and silently ignores max_completion_tokens, so the limit had no effect; this matches the behaviour already in place for Opencode and Ollama (#7131)
• feat: guardrail redaction gains an identity-based transformation path for provider-native request and response bodies, so exact provider-managed rewrites such as Model Armor and Bedrock land in the correct native JSON field even when the same text value appears in several fields. TextTargetID, TextTargetIDForIndex and TextRewrite identify each guardrail-visible field by a stable ID rather than by content value; RawRequestBodyTextTransformer and RawResponseTextTransformer let integrations register handlers on the request context; and rewriteRawJSONTextTargets validates target uniqueness, verifies each Original value before patching and re-reads the written value after, so stale or mismatched normalized text cannot silently rewrite the wrong field (#7049)
• feat: guardrail redaction covers LLM tool-call arguments, Chat function arguments, Responses function arguments and custom-tool input, across the Anthropic streaming and non-streaming paths. input_json_delta events are inspected and rewritten alongside text_delta, reading and writing delta.partial_json; collectAnthropicArgumentStringPaths recursively collects string-valued paths inside a tool_use block's input without touching tool names, IDs or definitions; BifrostResponsesStreamResponse gained Input to carry the full custom-tool payload on custom_tool_call_input.done; and DeepCopyResponsesMessage deep-copies ResponsesCustomToolCall so a copied message cannot mutate the original (#6977)
• fix: an abandoned non-streaming request, one whose caller context was already cancelled by the time the upstream finished, is billed and logged deterministically instead of about half the time. The worker's delivery select had two simultaneously ready cases, a send into a cap-1 channel and ctx.Done(), and Go picks uniformly among ready cases, so the terminal post-hooks that record billing and finalize the log row were skipped roughly 50% of the time. requestWorker now checks req.Context.Err() before the select on both the error and success paths and calls billAbandonedTerminal directly, keeping the 5-second timer guard inside the select for a caller that leaves between the check and the send (#6972)
• fix: provider response headers are filtered by classifier, not only by a fixed map of 28 exact names, so a credential-named header outside that map is no longer re-served to the inference caller in the HTTP response and extra_fields.provider_response_headers. The extractors now also consult schemas.IsSensitiveHeader, which matches credential names by substring and suffix and already recognized cf-access-* and x-amzn-oidc-*; a fixed list cannot enumerate the space when network_config.extra_headers exists to carry custom authentication headers and some upstreams echo request headers back (#7120) @Atharva-Kanherkar
• [fix]: a plugin that returns an incomplete BifrostError whose nested Error field is nil no longer panics the request worker. The fallback decision helper nil-checks fallbackErr, guards access to Error.Type and reads the message through the nil-safe GetErrorString() helper, continuing to the next fallback when allowed; behaviour for cancelled requests and AllowFallbacks == false is unchanged (#6967) @Constantine3
• [fix]: clearCtxForFallback now clears BifrostContextKeyProviderResponseHeaders. Providers set that key from their own HTTP response before the status check so error paths can forward it, and when a fallback attempt failed pre-flight, key selection failing for the fallback provider, a plugin short-circuiting it, or the queue retiring, nothing overwrote the key and the primary's headers survived onto the fallback's error response. A client could receive a response attributed to provider B carrying provider A's Retry-After and x-ratelimit-remaining-*, and wait according to a limit belonging to a provider that never served the request (#6973) @Huang-404-Q
• [fix]: untitled document blocks get unique names on the Bedrock Converse path. Every untitled block was named the literal document, and Converse rejects duplicate document names, so any request carrying two or more untitled documents failed unconditionally with ValidationException: Messages can't contain duplicate document names. A per-request namer threaded through the content-block conversion disambiguates with numeric suffixes (document, document-2, document-3) and suffixes explicitly titled documents only on an actual collision; the Responses input replay path gets the same treatment scoped to its content-block list (#7003) @Huang-404-Q
• [fix]: BifrostResponsesStreamResponse.Item is omitempty, so Responses stream events that carry no item payload no longer serialize "item": null. Strict OpenAI Responses clients reject those as invalid stream frames, which broke streamed /v1/responses usage entirely; response.output_item.added and response.output_item.done still emit the item object @ReStranger
• feat: SecretVar.RedactedIfSecret() returns a plain clone when the value is a literal and delegates to Redacted() when it is env-var or vault-backed, so regions and service URLs (Azure endpoint, Vertex/Bedrock/Bedrock Mantle region, vLLM/Ollama/SGL/Databricks URL, MCP connection string) are readable in the UI instead of being masked as if they were credentials. It always returns a fresh pointer, so the redacted copy never aliases the live config, and it replaces the ad-hoc IsFromSecret() guard blocks that tried to replicate this inline (#7085)
• feat: GA realtime transcription is served for OpenAI and Azure over both WebSocket and WebRTC with normal Bifrost authentication, routing, governance, guardrails, logging and transcription-aware pricing. Unlike a normal realtime session, which names its routing model in the connect URL, a GA transcription session carries only intent=transcription and delivers the model later in session.update under audio.input.transcription.model (or in the initial multipart /v1/realtime/calls request on WebRTC), so routing now resolves from the nested transcription model while realtime connection and turn semantics are preserved (#7089)
• fix: Claude Code replaying a prior assistant message through POST /anthropic/v1/messages at a Bedrock Mantle model is no longer rejected with hundreds of validation errors. The Bedrock-grouped ingress converter tagged user and system input text as output_text, though only input_text is valid on input messages, and omitted the required status on replayed assistant output messages, so Mantle's strict OpenAI-compatible validator refused the whole request. Replayed assistant messages now carry Status: "completed" and input text blocks are tagged input_text. Bedrock requests with no explicit max_tokens also populate it from the model's known capacity instead of truncating silently on large context windows (#7074)
• fix: Anthropic server tools such as web_search_20250305 are stripped before a request reaches Fireworks, vLLM or SGLang on their Anthropic-compatible endpoints, which return 400 for tools executed by Anthropic-operated infrastructure that does not exist on third-party hosts; clients whose built-in web search is always on, such as Codex, hit this on every request. ProviderFeatures entries for Fireworks, VLLM and SGL declare all server-tool flags off so the existing validators strip them, StripUnsupportedServerToolsFromRawBody mirrors that on the raw-body passthrough path, the caller's function tools are kept, and the drops are reported on the response's DroppedUnsupportedTools instead of failing the call (#7090)
• fix: guardrailConfig reaches Bedrock's OpenAI-compatible chat/completions and responses surfaces, streaming and non-streaming, which apply guardrails through request headers rather than the body field Converse uses, so a configured guardrail was previously ignored there with no error. withGuardrailHeaders maps guardrailIdentifier, guardrailVersion and optional trace to X-Amzn-Bedrock-GuardrailIdentifier, X-Amzn-Bedrock-GuardrailVersion and X-Amzn-Bedrock-Trace, then deletes the key from ExtraParams so it is not also emitted into the body. A half-formed config with only one of identifier or version is left untouched rather than sent, since both are required upstream, and the base header map is cloned rather than mutated (#7095)
• feat: use_openai_endpoints on Bedrock keys and aliases routes chat completions and responses through Bedrock's OpenAI-compatible /openai/v1 endpoints instead of Converse, for models that support them, mirroring the existing use_anthropic_endpoints pattern. It is opt-in by design: Converse carries Bedrock Guardrails, performanceConfig and requestMetadata that the OpenAI-compatible surface silently ignores, so diverting automatically could stop a guardrail from being enforced with no visible error. ResolveUseOpenAIEndpoints gives the alias value precedence over the key, and the narrow runtimeServesResponses check is replaced by a general runtimeServesOpenAIAPI that takes a BedrockAPI discriminator; chat completions, streaming and non-streaming, gained the runtime path that previously existed only for Responses (#7073)
• fix: Bedrock runtime models that serve the Responses API are routed to it through a dedicated surface resolver rather than falling back to Converse (#7071)
• fix: Bedrock Converse document blocks always ship their content base64-encoded through source.bytes, never through source.text, which Converse rejects unless citations are explicitly enabled. Plain text formats (text/plain, text/markdown, text/csv, text/html) previously took the text branch and failed with DocumentSource must set one of the following keys: bytes, s3Location; the branch is removed for data URLs, percent-encoded payloads and the non-data-URL file_data path alike (#7072)
• fix: the Bedrock Mantle base path is resolved from the model capabilities datasheet instead of hard-coded string matching in two packages. Mantle serves each model on exactly one of v1 or openai/v1 and returns a 400 on the other, and the previous matching covered only generations up to GPT-5 and Gemma 4, so GPT-6 and any future closed-generation model silently fell through to the wrong path. ResolveBedrockMantleBasePath applies family-name detection as a fallback and defers to the new BedrockMantleBasePath field on ModelCapabilities, so a new generation needs a datasheet row rather than a code change (#7077)
• fix: image_generation_call items where OpenAI emits action as a bare JSON string, such as "generate", decode correctly. ResponsesToolMessageActionStruct.UnmarshalJSON immediately peeked at a .type field, which cannot be read from a JSON string, so the whole decode failed with failed to peek at type field; that silently dropped the response.output_item.done and response.completed events carrying the image, leaving the stream without a terminal event and surfacing as a bogus provider closed the stream truncation error. The action is now tried as a bare string before falling back to the object type-peek, and ResponsesImageGenerationCall keeps the background, output_format, quality, revised_prompt and size settings OpenAI echoes back on completed items (#7060)

Installation

go get github.com/maximhq/bifrost/core@v1.9.0

This release was automatically created from version file: core/version