plastic-labs/honcho v3.6.0

on GitHub

Added

• Multi-tenancy: one deployment can serve many tenants, behind MULTI_TENANT (default off). Every tenant-scoped table now carries a tenant_id that leads its primary key, unique constraints and foreign keys. With the flag on, Postgres row-level security scopes every read and write to the tenant named by the caller's JWT, which must carry a tenant claim (tn); an admin key is admin only within its own tenant. Each tenant's vectors live under their own namespace, keyed by the tenant's vector_correlation_id, and a tenant without one is refused rather than given an empty namespace. With the flag off nothing changes: Honcho runs single-tenant on plain Postgres, and the optional TENANT_ID names the instance's tenant (default default). It is fixed for the life of the database, and boot is refused if the rows carry another tenant (#1264, #1546)
• Tenant registry at /v3/tenants, for a control plane that provisions tenants: create (idempotent; the same id with different fields is a 409), get, list, delete, and PATCH for derivation_paused and a set-once vector_correlation_id. It authenticates with TENANT_API_SECRET, not a JWT, and is disabled unless both that secret and MULTI_TENANT are set (#1264)
• The deriver claims work round-robin across tenants, so one tenant's backlog cannot starve another's, and skips the billable work of a tenant with derivation_paused set. Deletion, webhooks and other maintenance keep running, and paused work stays queued and is reported on its own per-tenant gauges. DERIVER_TENANT_FAIRNESS_METRICS (default off) adds per-tenant claim counts, claim wait and in-flight units. Under MULTI_TENANT, CloudEvents carry a tenantid attribute, the attribution counters a tenant_id label, and Sentry a tenant_id tag (#1264)
• Startup refuses to boot under MULTI_TENANT when isolation cannot hold: a transaction-mode pooler (DB_POOLER_MODE), RLS not enforced, no RLS-bypassing service role (DB_SERVICE_CONNECTION_URI), auth disabled on an API instance, or a tenant role that can read a partition directly. Every deployment, single-tenant included, also refuses to boot when the deriver's queue_item_batches table or its triggers are missing (#1264, #1552)
• Representation History: each dream is recorded as a run, readable through POST /dreams/list (newest first, filtered by observed_id and optionally observer_id) and GET /dreams/{run_id}. A run lists the conclusions the dream created, each with the conclusions it was derived from; the conclusions it removed, as they read when removed, including ones a near-duplicate from the same dream replaced (replaced_by); and the peer card before and after. Each change is recorded in the transaction that makes it, so failed and interrupted runs show what they committed. Created conclusions and their sources are resolved when the run is read: current, removed (with the text it had and removed_in_run), or unavailable once deleted. Only removed conclusions' text is copied; deleting their session deletes the copies, and deleting a conclusion through the API deletes its copy, including one a dream already removed; deleting a workspace deletes its history. Changes are kept for DREAM_HISTORY_RETENTION_DAYS (default 30); after that a run keeps only the created conclusions that still exist and gets pruned_at, and a run with nothing left is deleted. Recording can be turned off with DREAM_HISTORY_ENABLED=false. Workspace and admin keys only. DDL-only migration a7e3c1d9f2b4 adds dream_runs and dream_run_changes (#1542)
• DB_OWNER_CONNECTION_URI (optional) is the connection migrations run on; unset, they use DB_CONNECTION_URI as before. When it is set, the migrations grant the app role (and DB_SERVICE_CONNECTION_URI's role, if set) access to the tables and sequences the owner created, so the app can run as a role that owns nothing. Under MULTI_TENANT the owner also owns the message-search functions, so it must be able to bypass RLS; the startup validator refuses to boot otherwise (#1264)
• INSTALL_SURPRISAL Docker build argument (default false) installs the scikit-learn dependencies for surprisal-based dream sampling, independently of INSTALL_LANCEDB. Both example Compose services pass it through. Any value other than true or false fails the build (#1522)

Changed

• Upgrading runs 14 migrations. e5fe7f8bcf62 adds tenant_id to every tenant-scoped table and rebuilds their primary keys, unique constraints and foreign keys with tenant_id first. Adding the column is catalog-only, but the key rebuilds block writes to each table while they run, and take longer the bigger the table is. Each statement runs under the migrations' 5-minute statement_timeout, so plan a window for a large database. An upgraded database keeps its tables unpartitioned. The new and replaced indexes (conclusion lists, peer context, session context, session peers, the reconciler claim, message text search) build without a long write lock; to build them ahead of the upgrade, run the steps in migrations.utils.create_index_online by hand under the same names, and the revisions find them done. Releases from 3.2.2 on run against the new schema, so they stay safe rollback targets (#1264)
• The deriver caps its concurrent work units at DERIVER_WORKERS_PER_POOL_CONNECTION (default 4.0) times its DB pool (DB_POOL_SIZE + DB_MAX_OVERFLOW), so DERIVER_WORKERS can no longer exhaust the pool under load. With the default pool that is 120 units, so it only lowers a deriver that runs a small pool with many workers; the deriver logs the cap at boot when it binds and exports it as deriver_effective_worker_cap. Deployments on DB_POOL_CLASS=null are not capped. DERIVER_WORKERS now accepts up to 512, from 100 (#1264)
• The deriver claims from queue_item_batches, a one-row-per-work-unit aggregate that triggers on queue keep current, instead of aggregating the queue on every poll. The migration backfills it from unprocessed items, so an instance upgraded mid-backlog claims exactly what was pending (#1264)
• Message search indexes lead with tenant_id and workspace_name, so a search narrows to the workspace before reading the heap. They need btree_gin, which the migrations create; where the migration role can't create it, the old indexes stay and search works as before (#1264)
• The dream-due scan reads per-collection counters of explicit conclusions, which the document writers keep current, instead of grouping every explicit conclusion on each pass (#1264)
• Offset-paginated list totals are counted up to PAGINATION_COUNT_LIMIT (default 100,000) and reported as that limit plus one when there are more, so a list call on a large workspace no longer counts every row on every page. Pages still reach the end of the list, and cursor pagination is unchanged (#1500)
• Under DERIVER_SCHEDULER=api, one API replica measures the deriver backlog, enqueues due dreams and cleans up stale claims for the fleet, under a lease in Redis, and the others serve its snapshot. Before, every replica ran the same cross-tenant counts. Without Redis, or on any Redis error, each replica measures for itself as before (#1557)
• The reconciler's pending-document gauge counts up to 100,000 instead of running an uncapped count on every pass (#1552)
• Message and conclusion reads (lists, lookups, search, session context, representation, clone) also skip rows under a tombstoned workspace, peer or session, matching parents within the tenant. Representation History does the same: a run about a tombstoned peer or in a tombstoned workspace is not listed and its GET is a 404, and conclusions and removed copies from a tombstoned session resolve as unavailable. Migration 4f9c2a7e8d31 adds two partial indexes on tombstoned peers and sessions, built online. Nothing sets deleted_at yet, so behavior is unchanged (#1474, #1548, #1559)

Fixed

• A Redis error no longer fails a request that reads a cached workspace, peer, session or collection: the read falls back to the database and logs a warning. CACHE_MAX_CONNECTIONS (default 50) sets the Redis pool size per process, or per node with CACHE_CLUSTER; before, the cache library's default of 10 applied. When a Redis Cluster client cannot initialize (for example, no startup node answers), the error is now raised as a ConnectionError: startup retries and then falls back to the in-memory cache, and a request reads the database. Before, the client was left with no nodes, and its next command raised IndexError (#1550)
• GET /workspaces/{id}/queue/status counts with one row per session instead of one row per queue item, so a large backlog no longer makes it slow. The response is unchanged (#1550)
• Each process runs at most a quarter of its DB pool (DB_POOL_SIZE + DB_MAX_OVERFLOW) of message enqueue writes at once, so a burst of enqueues cannot take every connection from request handlers (#1550)
• A tool loop on the Anthropic transport no longer fails with a 400 (thinking.signature: Input should be a valid string) after a proxy serves one of its rounds from a non-Anthropic fallback. That round's thinking block has no signature, and Anthropic rejects any request that replays one. When a request holds an unsigned thinking block, the Anthropic backend removes every thinking block from it and keeps the text and tool calls; budget-thinking models (Claude 4.5 and earlier) run that call with thinking off, and adaptive models keep their thinking settings. Requests whose thinking blocks are all signed are unchanged (#1529)
• A tool loop whose fallback model on another provider answered a round no longer fails when the next round goes back to the primary (Anthropic: Unexpected role "tool"; OpenAI: rejected Anthropic content blocks). History is kept as typed messages and rendered for each attempt's provider (#1501)
• Message and conclusion creation no longer returns 422 when content quotes a tokenizer special token such as <|endoftext|>. The string is tokenized as ordinary text everywhere content is counted or embedded, including the token estimators, which used to fall back silently to a length guess (#1517)
• A queue item with a malformed work-unit key no longer stops the deriver. Before, it leaked the worker's slot, so a one-worker deriver stopped claiming for good. Its pending rows are now marked processed with a sanitized error, and the worker's slot is released even if cleanup fails (#1507)
• A structured-output attempt that fails is recorded in LLM telemetry and Langfuse with the model's raw output, and an attempt whose response the deriver rejects (an empty, truncated, filtered or refused representation) is recorded as an error instead of a success. Langfuse generations carry outcome and error_class metadata so failed ones can be filtered (#1498)