rmusser01/tldw_server v0.1.47

on GitHub

Promotes dev through PR #3443 and includes the changes already on main since
v0.1.46. See the full change inventory.
The release also removes the inactive workflow-run checkout path from backend
CI and fixes whitespace flagged by pre-commit.

Before upgrading: Use Python 3.12 or later. Back up persistent data and stop
API and worker writers before starting the upgrade. Production installs should
check authentication secrets, PostgreSQL settings and outbound allowlists.
Review the WebSocket credential changes and the keyword/tag migration notes
below. Clients using deprecated query credentials should move to a supported
header or the bearer subprotocol.

Added

• Agent Tasks can execute and track work. The encrypted message store introduced in 0.1.46 now feeds the executor. Tool calls carry a structured result envelope for inspection (#3042, #3231).
• Semantic search for Notes. Notes can be indexed by meaning for retrieval when embeddings are configured (#2862).
• Per-user limits for shared model use. Administrators can configure concurrency and usage quotas; usage recording now covers more LLM entry points. Personal provider keys are exempt from shared-provider quotas only when the request actually uses the caller's key (#3144, #3222, #3230, #3234, #3271, #3371, #3391, #3427).
• audio.cpp TTS private-network opt-in: providers.audio_cpp.extra_params.allow_private_network (default false, operator config only) lets a remote base_url (allow_remote_base_url) on a private or LAN address work without relaxing the global egress settings. It scopes requests to exactly the configured origin, public hosts included (any port; no EGRESS_ALLOWLIST entry needed under the strict profile); cloud metadata and link-local addresses stay refused (TASK-13950.11).

Changed

• Chat is easier to use on desktop and mobile. The WebUI and extension improve the composer, message views, conversation header, and history sidebar. Streaming updates and local persistence do less work; mobile composer controls use tighter defaults (#3251, #3255, #3309–#3313, #3316, #3372, #3400).
• Shared model credentials are no longer an implicit fallback on own-key paths. This covers Character Chat, evaluations, document and RAG operations, and Study assistants. Configure a personal provider key where these routes require one (#3423, #3426, #3427, #3439, #3444).
• Usage and pricing reporting. Accounting now captures more streaming completions, includes cache tokens where supported, and improves pricing tier resolution and Gemini pricing coverage (#3286, #3306, #3346, #3350, #3413, #3422).
• Runtime dependencies. Python 3.12+ remains required. This release includes the Uvicorn 0.54.0 and tinycss2 1.5.1 updates already on main (#3327, #2977).
• PostgreSQL connection pooling: PostgreSQL deployments, including the production Docker image, now always use the bounded psycopg_pool pool (pg_pool_size defaults to 20; a checkout waits 30 s, then raises PoolTimeout). The unbounded fallback pool used when psycopg-pool was missing is removed, and psycopg-pool is now a core dependency. A connection whose tenant scope cannot be applied is discarded and the request fails, instead of proceeding with another account's session settings (TASK-13950.3).
• Production detection is unified; upgrade warning. ENVIRONMENT, APP_ENV, DEPLOYMENT_ENV, FASTAPI_ENV, TLDW_ENV or ENV set to prod, production or live (any case) now marks the runtime as production everywhere tldw_production=true did, and the reverse. An install that sets, for example, ENVIRONMENT=production or APP_ENV=production without tldw_production will now refuse to start on a weak or default secret: a SINGLE_USER_API_KEY that is a placeholder, shorter than 24 characters or not in the server-generated tldw_<kid>.<secret> format (unless TLDW_ALLOW_LEGACY_SINGLE_USER_KEY=true), the template JWT_SECRET_KEY, or SQLite under AUTH_MODE=multi_user. Set strong secrets before upgrading (python -m tldw_Server_API.app.core.AuthNZ.initialize). The same marker also turns on the other production-only guards: required connector-secret encryption, stricter RAG guardrails, no raw-SQL RAG fallback, fatal evaluations-migration failures, and Resource Governor enforcement for embeddings: with RG_ENABLED=false, any production marker now makes importing the embeddings module fail (TASK-13950.6).
• Test seams are refused in production. API server startup now fails in a production-like runtime when any test seam is set: TEST_MODE, TESTING, TLDW_TEST_MODE, TLDW_ALLOW_TEST_FALLBACK_KEYS, SINGLE_USER_TEST_API_KEY, E2E_TEST_BASE_URL, MINIMAL_TEST_APP, JWT_SECRET_TEST_KEY, ENABLE_ADMIN_E2E_TEST_MODE, EVALUATIONS_TEST_DB_PATH, JOBS_TEST_NOW_EPOCH or any RG_TEST_*. PYTEST_CURRENT_TEST no longer unlocks the public test JWT secret or the test API key in production (TASK-13950.6).
• Media and Prompts keywords are case-insensitive and keep their display casing. "Foo" and "foo" are one keyword, stored with the first-seen casing (Media no longer lowercases keywords on write), and lookups also fold ß/SS, accented capitals and NFC forms. On PostgreSQL, Media DB startup merges existing case-variant keyword rows into the earliest one (links move to it; the others are soft-deleted with merge_parent_uuid set) and then enforces a unique LOWER(keyword) index. The PostgreSQL Media keyword vocabulary is shared, so the first user's casing becomes every user's display casing. Ingestion keeps processor-extracted keyword casing, and the Prompts keyword endpoints return stored casing instead of lowercase (TASK-13950.5).
• Collections (reading items) and Watchlists source tags are case-insensitive per user and keep their display casing. Tags are no longer lowercased on write: "Foo" and "foo" are one tag, stored with the first-seen casing, and tag lookups, filters, OPML export and content-alert source constraints also fold ß/SS, accented capitals and NFC forms. On first open after upgrade (SQLite and PostgreSQL), each user's case-variant duplicate tags are merged into their earliest row (tagged items and sources move to it, the duplicates are deleted), and a unique case-insensitive index per user is enforced. Tags are never merged across users. Existing tags keep their stored lowercase names (TASK-13950.5).
• Admin webhooks, github_create_issue and GitHub repository sync follow the general egress policy: ports other than 80, 443 and 8080 are refused unless WORKFLOWS_EGRESS_ALLOWED_PORTS lists them, so an admin-webhook receiver on a port such as :8443 or :5678 needs it set. When ENVIRONMENT, APP_ENV or ENV is prod or production, the strict profile denies every host not in EGRESS_ALLOWLIST or WORKFLOWS_EGRESS_ALLOWLIST, so production deployments without an allowlist must add the receiver hosts and api.github.com (TASK-13950.11).
• audio.cpp TTS base_url credentials: a loopback base_url with userinfo (user:pass@) has been refused since the audio.cpp client moved to core.http_client, as a raw ValueError. It, and one with allow_private_network on, now fails with a clear TTSValidationError. A remote base_url without the opt-in still sends its userinfo as Basic auth, as before. A base_url with an invalid port is now a TTSValidationError too (TASK-13950.11).
• Deprecated: ACP and Prompt Studio WebSockets (/api/v1/acp/multiplex, /api/v1/acp/sessions/{id}/stream, /api/v1/acp/sessions/{id}/ssh, /api/v1/prompt-studio/ws, /api/v1/prompt-studio/ws/{project_id}): credentials in the ?token= / ?api_key= query and in the x-api-key, <key> subprotocol still work but log a deprecation warning; send Authorization: Bearer, X-API-KEY, or the bearer, <token or key> subprotocol instead. These routes now authenticate through the core WebSocket resolver, so revocation, API-key scope and single-user rules match HTTP, which narrows two things ACP did on its own: the API_KEY environment variable is no longer accepted as a second single-user key next to a different SINGLE_USER_API_KEY (it still applies when SINGLE_USER_API_KEY is unset), and a ?token=&api_key= pair is resolved by get_auth_principal, the shared HTTP resolver, as it resolves both headers: the JWT first, and the API key only when the JWT is refused with 401. It also widens one: SINGLE_USER_TEST_API_KEY is now accepted wherever HTTP accepts it, under pytest or with TEST_MODE, TLDW_TEST_MODE or TESTING set (ACP required an explicit pytest runtime). Only a Bearer-scheme Authorization header counts as a WebSocket credential, so a reverse proxy's Authorization: Basic on the upgrade no longer hides the query or bearer-subprotocol credential. An API key without the route's scope now closes with 4403 instead of 4401, and the ACP clients no longer retry 4403 (TASK-13950.7).
• Deprecated: the Watchlists run stream WebSocket (/api/v1/watchlists/runs/{run_id}/stream) still takes ?token= / ?api_key=, which the WebUI's watchlists-stream client sends, but logs a deprecation warning. It and the Sandbox run stream (/api/v1/sandbox/runs/{run_id}/stream) now authenticate through the core WebSocket resolver, so they accept the bearer, <token or key> subprotocol (and echo bearer) and match HTTP: a JWT for a missing or inactive user is refused (the old copies checked only its signature and revocation), a JWT is not accepted in single-user mode, SINGLE_USER_TEST_API_KEY is accepted under pytest or with TEST_MODE, TLDW_TEST_MODE or TESTING set (they required an explicit pytest runtime), a Watchlists ?token=&api_key= pair falls back to the key when the JWT is refused with 401, and an API key without the read scope closes with 4403 instead of 4401. On Watchlists a header credential (Authorization: Bearer, X-API-KEY or the bearer subprotocol) now takes precedence over ?token= / ?api_key=, and the query is not read; the old code took ?token= instead of the Authorization header and ?api_key= instead of X-API-KEY, then verified that token, from either place, ahead of any API key, so ?api_key= lost to an Authorization: Bearer JWT (TASK-13950.7).
• Deprecated: the Persona stream WebSocket (/api/v1/persona/stream) still takes ?token= / ?api_key=, which the WebUI's persona-stream client falls back to for a credential that is not subprotocol-safe, but logs a deprecation warning once per connection; the voice assistant WebSocket (/api/v1/voice/assistant) likewise still reads ?token=, with the warning, when its auth frame carries no token (the voice SDKs send the token in both). Both now verify credentials through the core WebSocket resolver, so revocation, API-key scope and single-user rules match HTTP. For the voice assistant that means a JWT for a missing or inactive user is refused (it checked only the signature and revocation), a JWT is not accepted in single-user mode, and SINGLE_USER_TEST_API_KEY is accepted wherever HTTP accepts it, under pytest or with TEST_MODE, TLDW_TEST_MODE or TESTING set: the shared gate replaces voice's own pytest-only one. Its auth frame token is verified alone, so a handshake X-API-KEY cannot stand in for a refused frame token. On Persona a ?token=&api_key= pair, or a header token beside X-API-KEY, falls back to the key only when the JWT is refused with 401 (it fell back on any refusal), and a header credential (Authorization: Bearer, X-API-KEY or the bearer subprotocol) now takes precedence over the query, which is then not read; the old code read ?token= ahead of the bearer subprotocol, and fell back from a refused header token to ?api_key=. An API key without the read scope still closes with 4401 on both. The voice assistant's auth frame protocol and close codes are unchanged (TASK-13950.7).
• The audio streaming WebSockets (/api/v1/audio/stream/transcribe, /stream/tts, /stream/tts/realtime, /chat/stream) and realtime speech (/api/v1/audio/realtime, /v1/realtime) now authenticate through the core WebSocket resolver; the {"type": "auth", "token": ...} first frame and the single-user cookie, whose clients may send config first, work as before; with the cookie, an auth frame without a token now keeps the cookie session (it closed 4401). A handshake credential (Authorization: Bearer, X-API-KEY, or the bearer, <token or key> subprotocol, now accepted and echoed) is verified before the socket is accepted, so a refused one now fails the WebSocket handshake before accept (uvicorn answers HTTP 403 with no close code; the Starlette test client reports close 4401) instead of closing 4401 after accept. Without one the socket is accepted only to read the auth frame: a first frame that is not an auth frame now closes 4400 instead of 4401 (no shipped client checks the code), a first frame over 16,384 characters is refused 4400 without being parsed, and nothing is read after it. Verification matches HTTP: an API key is accepted in the auth frame and as a bearer in multi-user mode (only a JWT was), a JWT for an inactive user and a service token are refused, SINGLE_USER_TEST_API_KEY follows the shared test gate, a single-user key from outside SINGLE_USER_ALLOWED_IPS closes 4401 instead of 1008, an unavailable credential store closes 1011, and the caller's organizations and teams come from HTTP's membership check rather than unchecked JWT claims, so team-scoped BYOK credentials now apply on /chat/stream and the TTS streams as they do over HTTP. A scoped JWT's or multi-user key's allowed endpoints and paths (4403) and call quota (4003, or 1008 with AUDIO_WS_QUOTA_CLOSE_1008) are unchanged. Deprecated: ?token=, still read only with AUDIO_WS_ALLOW_QUERY_TOKEN_AUTH set, logs a deprecation warning and is verified like Authorization: Bearer, so it also takes a JWT in multi-user mode (it took only an API key); ?api_key= is not read, with or without the setting (TASK-13950.7).
• The voice assistant WebSocket (/api/v1/voice/assistant) reads its auth frame through the same core first-frame resolver. Its frames and close codes are unchanged, except that a first message that is not JSON now gets auth_error "Expected AUTH message" and 4400 (it closed 1011), and an auth frame without a token also accepts a credential on the upgrade request before the deprecated ?token=; ?api_key= is not read, as before (TASK-13950.7).

Fixed

• Audio Studio and workspace links: Audio preview and download validate media ticket destinations. Workspace diagnostics links only to the supported runtime configuration page.

• Recoverable chat history and cancellation. Stop works before the first token, and permanent history erasure coordinates recoverable copies so deleted data cannot reappear through legacy history (#3368, #3431).

• Authentication and administration. Sign-in links are consumed atomically, verification-email diagnostics omit secrets, and admin alert history reports the correct results (#3428–#3430).

• PostgreSQL and background jobs. ChaChaNotes operations keep their intended transaction and tenant scope across requests, WebSockets and background work; job completion verifies current ownership before persisting results (#3082, #3092, #3293, #3295, #3297, #3312, #3323).

• Audio processing. Audio jobs renew leases and preserve retry state, while model adapters coordinate loading and cancellation to keep in-use engines available (#3257, #3259, #3260, #3262, #3274, #3280, #3281, #3291, #3303).

• RAG and evaluations. A streamed RAG answer is emitted once, and the benchmark judge receives the inputs it needs to score the answer (#3445, #3447).

• Workflow and media recovery. Workflow cancellation stops sibling steps and cleans up detached work; media and knowledge flows improve error handling, state recovery, and source navigation (#3194, #3196, #3204, #3290, #3298, #3308).

• The TTS realtime, voice chat and TTS streaming WebSockets re-raise the server's own cancellation when it lands in their cleanup; they raised a new CancelledError that escaped the host's cancel scope, which the test client reported as concurrent.futures.CancelledError at teardown (TASK-13950.7).

• Workflows podcast_rss_publish SSRF: the remote seed-feed fetch now goes through core.http_client, so the egress policy (including private and link-local addresses) is enforced on the URL and every redirect hop, and the body is capped at 20 MiB. A refused URL returns remote_seed_blocked_egress, and a non-XML seed body returns invalid_feed_xml instead of raising. The 15 s remote-seed timeout now bounds the whole download, not each socket operation (TASK-13950.11).

• ACP MCP streamable-HTTP transport SSRF: requests now go through core.http_client, so the egress policy is checked on the address the connection uses (a DNS answer that changes to loopback after validate_mcp_http_url is refused) and env proxies are no longer read. Redirects are still not followed. With allow_private_network (or ACP_ALLOW_PRIVATE_MCP_HTTP) the configured private origin is admitted on any port, but cloud metadata and link-local addresses are refused. Without it, the default egress policy now applies, including its port list (80, 443, 8080; WORKFLOWS_EGRESS_ALLOWED_PORTS) and, in production, the strict profile's host allowlist (EGRESS_ALLOWLIST) (TASK-13950.11).

• ACP MCP-over-SSE transport SSRF: the SSE stream and the JSON-RPC POSTs now go through core.http_client (astream_sse and afetch), so the egress policy is checked on the address each connection uses and on every SSE redirect hop, and env proxies are no longer read. The SSE stream follows same-origin redirects only, so configured headers never reach another origin; POSTs still do not follow redirects. The private-network opt-in behaves as for the streamable-HTTP transport and applies only to the configured SSE origin, never to a server-supplied POST URL on another origin. astream_sse gains the configured_endpoint scope that afetch already takes, and an allow_cross_origin_redirects switch (TASK-13950.11).

• MCP web.fetch SSRF: the default fetch client now streams through core.http_client.astream_bytes, so the egress policy is enforced on the address the connection uses (a DNS answer that changes to a private address after the outbound-policy check is refused as outbound_policy_denied) and env proxies are no longer read. The outbound policy and robots check on every redirect hop, the byte cap and the timeout are unchanged (TASK-13950.11).

• Admin webhook delivery SSRF: AdminWebhooksService.deliver (incident notify-webhooks) now sends through core.http_client.afetch, so every attempt checks the egress policy (loopback, private and link-local addresses, operator allow/deny lists) and pins the resolved IP, and env proxies are ignored. A refused URL is logged as a failed delivery with no retries. Signing, headers, backoff, timeouts and the no-redirect rule are unchanged (TASK-13950.11).

• Workflows github_create_issue egress: the issue POST now goes through core.http_client.apost, so the egress policy (including the operator allow/deny lists and a private or link-local DNS answer for api.github.com) is checked before the token is sent, the resolved IP is pinned, and env proxies are ignored. A refused request returns github_api_blocked_egress. Valid requests and responses are unchanged (TASK-13950.11).

• Ingestion Sources GitHub sync SSRF: remote git_repository sources fetch the GitHub API through core.http_client.stream_response instead of urlopen. Every hop, redirects included, must stay on https://api.github.com and pass the egress policy, and its resolved IP is pinned, so a redirect can no longer carry the access token to another host or a private address. Env proxies are ignored, and the blob size cap still stops the read while streaming. A refused or failed request now fails as a ValueError, which the sync records as a failed item or a failed job as before. A control character in a stored repo_url now fails that one sync job instead of stopping the ingestion worker for every user (TASK-13950.11).

• Standalone HTML generation egress: provider completions now go through core.http_client (astream_bytes) instead of a bare httpx.AsyncClient, so a provider host that resolves to a private address is refused before any request and the loopback llama.cpp/Ollama adapters are allowed only on their own configured origin; astream_bytes gained configured_endpoint= (TASK-13950.11).

• audio.cpp TTS client SSRF: requests now go through core.http_client: one attempt, no redirects, and the egress policy checked on the address the connection uses. A loopback base_url (the default and the managed sidecar) is admitted for its exact origin. With allow_remote_base_url a remote server now gets the default egress policy, so a private, link-local or metadata address is refused (TTSConfigurationError, EGRESS_DENIED) unless the operator's egress settings allow it, only ports 80, 443 and 8080 are allowed unless WORKFLOWS_EGRESS_ALLOWED_PORTS lists others, and under the strict profile (ENVIRONMENT=prod) the host must be in EGRESS_ALLOWLIST or WORKFLOWS_EGRESS_ALLOWLIST (TASK-13950.11).

• Audio Studio ACE-Step egress: every ACE-Step request now goes through core.http_client instead of a raw httpx.AsyncClient, scoped to the exact origin AUDIO_STUDIO_EXTERNAL_ENDPOINT_ALLOWLIST admitted. A local ACE-Step server on a private address and any port still works, but an allowlisted host whose DNS answer is a cloud-metadata or link-local address is refused before anything is sent (ace_step_endpoint_refused_by_egress_policy, not retried), the resolved address is pinned, and env proxies are no longer read unless HTTP_TRUST_ENV is set. Redirect handling, request bodies, the bearer header and timeouts are unchanged (TASK-13950.11).

• Qwen3-ASR vLLM upload egress: the unplanned vLLM branch ([STT-Settings].qwen3_asr_vllm_base_url) now uploads through core.http_client like the planned branch, scoped to the configured server's exact origin. A local vLLM server on a private address and any port still works; a cloud-metadata or link-local DNS answer is refused before the audio is sent, the resolved address is pinned, a URL with userinfo is refused, and env proxies are no longer read unless HTTP_TRUST_ENV is set. The request, its 300 s timeout and the no-redirect rule are unchanged (TASK-13950.11).

• audio.cpp STT certificate pins: with the aiohttp transport, the per-transcription session now comes from the new core.http_client.create_aiohttp_session, so HTTP_CERT_PINS pins for the audio.cpp host are enforced and the default User-Agent and TLS floor match the shared session. Requests were already egress-checked through afetch (TASK-13950.11).

• Managed OCR readiness probe egress: wait_for_managed_http_ready (llama.cpp and ChatLLM OCR) now probes through core.http_client instead of http.client, scoped to the exact configured origin (*_OCR_HOST/*_OCR_PORT, the OCR profile or CHATLLM_OCR_HEALTHCHECK_URL). A local server on any private address and port is still probed, redirects are still not followed, and a host whose DNS answer is a cloud-metadata or link-local address is refused at once instead of being polled. The probe paths, interval and per-probe timeout are unchanged (TASK-13950.11).

• http_client pinned connections fall back through the vetted addresses: a connection pinned to an egress-approved address now tries the next approved address from the same DNS answer when the connect is refused or times out, instead of failing. A configured localhost endpoint that resolves to ::1 first therefore reaches a server listening only on 127.0.0.1 again, on the httpx and aiohttp paths, streams, downloads and the certificate-pin check. Nothing is re-resolved, Host and SNI are unchanged, an address the policy did not approve is never tried, and at most 4 addresses are tried. Every try but the last caps its connect phase at HTTP_CONNECT_TIMEOUT (default 5 s) and the last uses the request's own connect timeout, so a host whose addresses all fail costs at most 3 × HTTP_CONNECT_TIMEOUT plus that timeout. The certificate-pin preflight shares one HTTP_CONNECT_TIMEOUT budget across addresses, so it blocks no longer than before. afetch, apost and fetch (httpx) with no timeout now use the client's configured timeout instead of running with none; pass timeout explicitly to override it (TASK-13950.11).

• The Persona stream WebSocket accepts the single-user API key (X-API-KEY, Authorization: Bearer, the bearer subprotocol or the deprecated ?api_key=); it checked the key with the multi-user API-key manager and refused it on every channel (TASK-13950.7).

• WebSockets that echo the bearer subprotocol (Persona, Workflows, Sandbox, Watchlists, ACP, Prompt Studio) echo the marker as the client offered it, so a client that sends Bearer rather than bearer completes a browser handshake (TASK-13950.7).

Removed

• ACP and Prompt Studio WebSockets no longer accept Authorization: X-API-KEY <key> or the api-key, <key> subprotocol; no client sent either (TASK-13950.7).
• The Sandbox run stream WebSocket no longer reads a credential from ?auth_token= or, with signed URLs off, ?token=; no client sent either. A signed log_stream_url's token and exp are unchanged (TASK-13950.7).
• The Persona stream WebSocket no longer accepts MCP-issued JWTs (from /api/v1/mcp/auth/token), which it tried when AuthNZ refused a token and which carry no AuthNZ revocation check; no client sent one. Use an AuthNZ access token or an API key (TASK-13950.7).