yvgude/lean-ctx v3.8.16

on GitHub

Added

• Agent navigation upgrades for coding agents (#607–#611). A cohesive set of
  cross-turn navigation primitives that cut wasted re-discovery:
  • Stable symbol handles (#607). Symbols carry a resolvable, cross-turn
    handle (path#name@Lline); ctx_search action=symbol handle=… returns the
    exact body deterministically, so an agent can re-open a symbol next turn
    without re-searching.
  • Search hits tagged with their enclosing symbol (#608). Every
    ctx_search match now reports the function/class it lives in (plus that
    symbol's handle), turning a bare line hit into navigable context.
  • Agent-loop taxonomy + navigation-paradox guidance (#609). The canonical
    rules (now v3) name the act→observe→navigate loop and warn against the
    "more reads ≠ more understanding" paradox, surfaced as a compact per-turn
    one-liner.
  • signatures coverage for 5 more languages (#610). OCaml, Haskell,
    Julia, Solidity and Nix gain tree-sitter signature extraction (−68 %…−90 %
    tokens vs. full source) via statically linked grammars (no dynamic WASM).
  • Off-vs-on answer-quality testbench (#611). lean-ctx eval testbench
    measures answer quality, tokens, turns and walltime across pinned real
    repos, with a deterministic recorded subset that gates CI and a
    FINDINGS.md + regressions report.
• Codex ChatGPT-subscription proxy — durable, opt-in model-turn compression
  (#603/#616/#621). New [proxy] codex_chatgpt_proxy (env
  LEAN_CTX_CODEX_CHATGPT_PROXY, default false). A ChatGPT-subscription login is
  flat-rate, so the safe default leaves Codex talking directly to chatgpt.com —
  history visible, codex cloud/remote intact, no #597. When you opt in, lean-ctx
  setup pins the generated leanctx-chatgpt provider (model_provider +
  chatgpt_base_url + a [model_providers.leanctx-chatgpt] block) so model turns
  route through the proxy's /backend-api/codex/responses rail and get compressed;
  every other /backend-api/* call (auth, cloud/remote, MCP) is forwarded
  credential-preserving. Pinning a provider scopes Codex history to it (#597), so
  routing is opt-in — you accept that trade only when you ask for it. On that
  rail the proxy strips Codex's X-OpenAI-Internal-Codex-Responses-Lite marker so
  chatgpt.com serves the full Responses stream every model needs (gpt-5.5 was
  rejected in lite mode); single- and multi-turn previous_response_id
  continuation verified (#623). lean-ctx doctor is opt-in-aware — the sanctioned
  rail reads healthy, a half-written pair or an openai_base_url/backend-api
  override stays flagged as a stale artifact to heal. Turn it on/off durably with
  lean-ctx proxy codex-chatgpt on|off|status: it writes the opt-in straight
  to [proxy] codex_chatgpt_proxy — the single source of truth the env-less managed
  proxy, editor integrations and every later setup pass read (none inherit the
  shell env, #449/#590) — then re-applies Codex's provider config immediately, with
  clear feedback (and a heads-up when the proxy isn't running yet). Toggling back
  off strips the entries and restores native history + cloud/remote. Exporting
  LEAN_CTX_CODEX_CHATGPT_PROXY still works and is bridged to config on
  proxy enable/restart. This closes the trap where a shell env opt-in never
  reached the process that actually rewrote the Codex config. Builds on
  @ousatov-ua's PRs #616/#621, gated behind the opt-in so non-routing users keep
  full native history by default.
• Managed Connectors — hosted continuous source sync (#281). The team server
  runs a scheduled, in-process sync of configured GitLab/GitHub sources into a
  workspace's BM25/graph/knowledge stores, so every seat's ctx_semantic_search /
  ctx_knowledge surfaces the source's issues/PRs/pipelines without per-call
  credentials or manual ctx_provider runs. Per-connector credentials live only
  in the private team.json (encrypted at rest by the control plane) and are
  never returned; GET /v1/connectors exposes a secret-free roster plus
  per-connector health, audit-scope gated. Ingestion honours the hosted storage
  quota as a non-destructive backstop (pauses, never deletes — #282), and
  connector activity is rolled into the /v1/usage snapshot (#283). The
  managed_connectors count remains entitlement-gated by the control plane at
  provisioning.
• Context Time Machine — git-anchored, signed snapshots of the layer state
  (epic #1022). The state of the context layer (what the model saw, why, and at
  what token ROI) becomes a navigable, reproducible, shareable artifact — the
  temporal axis through everything lean-ctx does.
  • CONTEXT_SNAPSHOT_V1 (#1023). A distilled, typed, content-addressed
    (BLAKE3) and ed25519-signable projection of the live stores — git anchor,
    Context IR lineage, ledger Φ-scores, ROI, and the session slice — never raw
    transcripts. Deterministic per the output-determinism contract
    (contract).
  • Headless engine (#1024). lean-ctx snapshot create [--sign] | list | show | verify builds a snapshot from the live stores, anchors it to the current
    commit, and stores it on a crash-safe, append-only timeline (index.jsonl);
    verify proves both integrity (body hashes to its id) and the signature.
  • Replay in the dashboard (#1025). A new Time Machine tab scrubs the
    timeline and shows, per frame, the git anchor, ROI, lineage, ledger Φ, and the
    session behind it, over a JSON control-plane API.
  • Restore / resume (#1026). lean-ctx snapshot restore <id> [--git] merges
    the snapshot's session slice (task, decisions, files) into the live session so
    the next agent resumes where it left off, and with --git checks out the
    commit anchor — guarded so it never clobbers a dirty tree. Bare-CLI sessions
    now stamp their project root (as the MCP daemon already did), so a CLI-only
    session … ; snapshot create flow captures the session slice.
  • Share / import (#1027). lean-ctx snapshot publish <id> [--out <path>]
    writes a signed, portable *.ctxsnapshot.json; lean-ctx snapshot import <file> proves its integrity and signature and appends it to the local
    timeline (idempotent; tampered or wrongly-signed files are refused) — so a
    teammate can show, verify, and restore exactly the state you shared.
• Lossless memory & one consolidation engine (#995). Project memory is now
  fully recoverable and managed by a single capacity manager. Builds on and
  supersedes the original capacity-reclaim proposal by @ousatov-ua (PR #588).
  • Nothing is hard-dropped. Every store — facts, history, procedures and
    patterns — evicts through one archive-backed path
    (core/memory_capacity::reclaim_store): the lowest-value tail is written to
    memory/archive/<store>/ before removal, so it stays restorable. This
    replaces the previous per-store hard drops (history drain, procedure/pattern
    truncate) that lost data on overflow.
  • One capacity formula with hysteresis. A reclaim triggers only when a
    store reaches its cap and then settles it at a working-room target (75% by
    default), instead of churning on every write near the cap. On by default;
    set [memory.lifecycle] reclaim_enabled = false to trim only the overflow.
    The target fraction is [memory.lifecycle] reclaim_headroom_pct
    (env LEAN_CTX_LIFECYCLE_RECLAIM_*).
  • One consolidation engine. The CLI/MCP consolidate, the scheduled
    post-dispatch pass, startup auto-consolidate and the cognition loop now share
    a single canonical engine and session-import core (ConsolidateOptions), so
    promotion budgets, fact keys and confidences are identical everywhere. Fixes a
    long-standing cwd bug (#2362) where background consolidation imported the
    wrong project's session.
  • Recover on demand. lean-ctx knowledge restore [--store …] [--query …] [--limit N] and ctx_knowledge action=restore bring archived items back into
    the live stores (idempotent; a live fact's key is never shadowed by an older
    archived value). The recall-miss rehydrate now reaches every retained archive
    (previously 16 kept but only 4 reachable).
  • Preview before you commit. lean-ctx knowledge consolidate --dry-run
    (and ctx_knowledge action=consolidate dry_run=true) reports exactly what a
    run would import and archive, writing nothing. The consolidation report now
    breaks down archived counts per store and points at the restore command.
• Learn-loop enrichment (#980). The gotcha learn-loop now reaches further with
  three additions. (1) Multi-target write-back: lean-ctx learn --apply writes
  the distilled learnings to AGENTS.md (created if absent) and CLAUDE.local.md
  (updated only when the project already keeps one), each via an atomic
  tmp+rename so a crash can never truncate your memory file. (2) Zero-config
  transcript scan: lean-ctx learn --mine with no argument auto-discovers the
  agent-transcripts directory (~/.claude/projects, then
  ~/.cursor/agent-transcripts) instead of erroring. (3) Loop-weighting
  bridge: the cognition loop now promotes proven gotchas — high confidence,
  seen across ≥3 sessions, and shown to have prevented real errors — into durable
  project knowledge, so recall surfaces them like any other fact (idempotent,
  capped per pass).
• Never-compress path globs (#1150). New proxy.compress_protect takes a list
  of file-path globs (*.snap, **/golden/**, tests/fixtures/*) whose reads
  are always returned verbatim (full), bypassing every lossy mode (auto,
  aggressive, signatures, density, …) — for files where exact bytes matter
  more than token savings: golden snapshots, byte-asserted fixtures,
  security-sensitive configs. Each glob is matched against both the path and its
  file name, so *.snap works anywhere while **/golden/** targets a directory;
  explicit raw reads and lines: slices are left as requested. Empty by default
  (the lossless crushers and beneficial gate already keep compression safe), so it
  is a precise escape hatch, not a tax on the default fast path.
• Premium defaults: safe cache telemetry now ships on (#986). Everything that
  is pure measurement or a strict safety improvement is enabled by default, so
  every install and every update delivers the best lean-ctx without flipping a
  flag (config is written minimally, so code defaults reach existing users on
  upgrade). The cache-economics telemetry below (proxy.cache_policy) and the
  cache-aligner volatile-field telemetry (proxy.cache_aligner, #940) are now
  on by default — both are measurement-only / strictly cache-safe. Features
  that change provider-visible content or carry a real cost risk stay opt-in by
  design (a premium proxy never silently risks your bill): the cold-prefix repack
  (cold_prefix_repack, ~12× re-bill on a wrong cold guess), the active
  cache-aligner relocate (cache_align_relocate), breakpoint injection
  (cache_breakpoint), and output shaping (effort, verbosity_steer). All
  remain togglable via config or LEAN_CTX_PROXY_*=on|off.
• Cache-economics: prompt-cache miss attribution + net-cost repack gate
  (#986). proxy.cache_policy (now on by default) answers the one cache
  question the proxy could not yet measure — why a turn misses the provider
  prompt-cache.
  proxy/cache_attribution classifies every anchored turn by comparing the
  cacheable prefix hash and idle time against the conversation's previous turn:
  cold start, warm reuse (stable prefix within TTL — should hit),
  TTL lapse (stable prefix, expired by time), or prefix change (the
  prefix mutated, so the provider re-writes regardless of timing). The four
  outcomes surface as cumulative gauges under /status cache_attribution,
  turning "I keep missing cache" into an actionable diagnosis (extend the TTL /
  repack vs stop mutating the prefix). It is strictly measurement-only — the
  request body is never touched. The same flag also adds proxy/cache_policy, a
  priced (model_pricing) net-cost gate folded into the cold-prefix repack
  (#480): a repack is skipped when the cacheable prefix is below the provider's
  ~1024-token minimum, so re-seeding it could never produce a cache the provider
  keeps. The gate is an extra AND-condition, so it can only make repacking more
  conservative — it can never bust a cache the default kept. On by default; the
  attribution never mutates the wire bytes, so the request the provider sees is
  byte-identical whether the policy is on or off. Opt out with
  proxy.cache_policy = false or LEAN_CTX_PROXY_CACHE_POLICY=off.
• YAML crusher — kubectl -o yaml, manifests, CI configs (#985). A new
  core/yaml_crush maps a YAML document onto the JSON value model (yaml_serde)
  and compacts it through the shared json_crush core: the verbose YAML
  formatting is dropped and redundant items/list arrays are factored into
  _defaults, all behind a _lc_yaml_crush envelope that round-trips exactly to
  the parsed value. Wired into the aggressive read path (compressor, ctx_read)
  for .yaml/.yml and into shell-output compression (kubectl/helm -o yaml),
  with the same lossless-then-lossy ladder as the tabular crusher — the lossy
  stage drops high-entropy columns behind a CCR handle (yaml_ tee prefix).
  Fires only above the 25 % reduction gate (JSON quoting offsets YAML formatting
  for flat string maps, so those are correctly left alone) and never inflates.
  Deterministic (#498); a Condition::YamlCrush arm measures the win.
• Columnar crusher for CSV/TSV — reads and shell output (#982). A new
  core/tabular_crush rewrites row-oriented delimited data into a columnar JSON
  shape: constant columns are hoisted once into _const, varying columns stay
  positional in _rows, so a table whose values repeat down a column stops
  paying for that column on every line. It is wired into the aggressive read path
  (compressor, ctx_read) for .csv/.tsv and into shell-output compression
  (shell/compress/engine), and only fires when it clears a 25 % size gate
  (columnar JSON quoting has overhead, so the bar is tuned below the JSON
  crusher's 50 %). The lossless mode round-trips exactly; an opt-in lossy mode
  drops high-entropy columns into _dropped with full CCR recovery (tbl_
  tee-store prefix, <lc_expand:…> handle). Deterministic (#498) and never
  inflates. A Condition::TabularCrush arm in the A/B harness measures the win.
• CCR robustness regression suite (#983). Nine focused tests
  (proxy/ccr_robustness_tests) lock down the content-addressed-recovery path
  against the failure classes seen in comparable context layers: a lossy rewrite
  that emits a handle it cannot back, retrieval after the tee file is gone past
  its TTL, an in-band splice on a streaming-shaped request, the tee store
  colliding with read-stub bookkeeping, path-traversal / non-tee / bad-hex
  handles, and the cold stub index resurrecting content across a restart. A
  change-aware preflight gate runs them on fast whenever the recovery surface
  changes and on every full run, so the guarantees stay green without slowing
  unrelated work.
• See compression before it ships — compress diff + ctx_compare (#984). A
  read-only core/compress_preview renders original-vs-emitted side by side with
  byte and token accounting, the saved-token delta and ratio, and a unified diff,
  reusing the real read/shell pipelines (no separate code path to drift). Exposed
  as the CLI lean-ctx compress diff <file|-> [--shell "cmd"] [--json] and the
  read-only MCP tool ctx_compare (Debug category). Deterministic and
  self-describing, so an agent can decide whether a rewrite is worth it.
• ctx_outline levels up — directory outline, deterministic JSON, name filter,
  verifiable AST backend (gitlab #981). A public review (the ast-grep author,
  comparing his new ast-grep outline) called our outline "fishy" — fair only as
  a first impression: ctx_outline has always been real tree-sitter with
  declarative per-language queries (core/signatures_ts, ~22 languages, real
  multi-line spans), but the prominent file in the repo is the 790-line regex
  fallback (core/signatures.rs) and the tool overclaimed "via tree-sitter"
  without disclosing it. This release closes the gap and turns it into an advantage:
  • Directory outline. ctx_outline <dir> now emits a deterministic, sorted,
    gitignore/vendor-aware per-file table of contents (matching ast-grep outline src); a directory used to be rejected. Bounded (≤ 600 files, ≤ 1.5 MB/file).
  • Stable JSON output. format=json produces byte-stable JSON (#498) — fixed
    field order, sorted files, no timestamps — for a file or a directory, each file
    labelled with the extraction backend (tree-sitter | regex) so the
    syntax-aware claim is verifiable, not asserted. (ast-grep lists stable JSON
    as an open TODO.)
  • Name filter. match=<substr> keeps only symbols whose name contains the
    case-insensitive substring, composing with kind= — across file / dir / JSON.
  • Navigable by default. The text outline now always carries @Lstart-end
    line spans (located renderers), so it actually serves "navigate before a full
    read", as advertised.
  • Honesty + correctness. core/signatures.rs gained a module-doc pointing at
    the tree-sitter primary path; Rust impl blocks render as impl …, not
    class … (both backends); the tool description now states "tree-sitter primary,
    regex fallback"; dead handle_via_read removed. New
    extract_signatures_with_backend powers the per-file backend label.
• Active cache-aligner relocate — the opt-in tail-relocate the #940 detector
  was the precursor to (#974). With [proxy] cache_align_relocate (env
  LEAN_CTX_PROXY_CACHE_ALIGN_RELOCATE) enabled, the proxy rewrites an
  unanchored Anthropic system prompt into a stable block — every volatile
  value (ISO dates/datetimes, UUIDs, git SHAs) replaced by a constant [ctx#N]
  placeholder — carrying the cache_control breakpoint, plus an uncached
  trailing block that re-states the relocated values. The large, stable prefix
  then stays byte-identical turn-to-turn and finally caches at the provider, while
  only the small volatile tail is reprocessed. Anthropic-only, Treatment-arm, and
  gated on a client that anchored nothing of its own and on Anthropic's minimum
  cacheable size; deterministic (#498) and idempotent (a second pass sees only
  placeholders). Composes with the #939 breakpoint injection to exactly one
  anchor on the stable block. New /status cache_safety gauges
  (volatile_relocate_requests, volatile_fields_relocated) quantify the win.
  Default off — the request is byte-identical until you opt in.

Fixed

• Cline/Roo rules are MCP-first — dropped the stale lean-ctx -c prefix guidance
  (GH #603). install_cline_rules hardcoded a .clinerules body telling the agent
  to prefix every shell command with lean-ctx -c, even though Cline/Roo get the
  lean-ctx MCP server installed and the shell hook already wraps real terminal
  commands — so the manual prefix re-wrapped an already-wrapped command, tripped the
  re-entry passthrough and returned uncompressed output. The body now derives from
  core::rules_canonical (the single source of truth) like every other dedicated
  rule file: MCP-first (ctx_*), no lean-ctx -c, wrapped in the canonical markers
  so uninstall can strip it (the old freeform header was not removable). Reported
  by @ousatov-ua.
• lean-ctx no longer touches Codex under a ChatGPT subscription login (GH #597).
  GH #568 pinned model_provider = "leanctx-chatgpt" (plus a
  [model_providers.leanctx-chatgpt] block) and openai_base_url/chatgpt_base_url
  overrides in ~/.codex/config.toml to route ChatGPT turns through the proxy. That
  was the wrong trade for a subscription: a ChatGPT plan is flat-rate, so
  compression saves no money — while the pin hid every prior conversation (Codex
  scopes history by provider id by design, openai/codex#15494/#19318) from
  /resume, fork and the Desktop picker, the backend-api base-URL overrides
  funnelled Codex's cloud/remote + login traffic through a proxy built only for
  model turns (breaking codex cloud/remote), and they made Codex depend on a live
  local proxy. lean-ctx now writes nothing for ChatGPT auth — Codex talks
  directly to chatgpt.com, so history, codex cloud/remote and login all stay
  native (no data loss; rollouts + SQLite were always intact). API-key Codex is
  unchanged: it keeps the per-token /v1 proxy rail, where compression actually
  cuts cost. Upgrading auto-heals: the next lean-ctx proxy enable/setup strips
  the stale leanctx-chatgpt provider and the backend-api base-URL overrides,
  and lean-ctx doctor flags any lingering ChatGPT-proxy entries.
• Shell hook no longer blocks Claude Code's Bash tool (GH #595). Claude Code
  wraps every Bash call in its own scaffolding
  (shopt -u extglob … && eval '<cmd>' < /dev/null && pwd -P >| /tmp/claude-XXXX-cwd)
  before the lean-ctx shell hook forwards the whole line to lean-ctx -c. The
  allowlist then hard-blocked the eval at command position (exit 126) on every
  command — the wrapper shape is identical each time, so the Bash tool became
  unusable.
  • Look through the wrapper: the new shell::agent_wrapper detects the host
    scaffold, extracts the real inner command and runs that through the normal
    allowlist + compression pipeline, so the inner git/cargo/… command is gated
    and compressed as usual instead of dying on the eval.
  • cwd tracking preserved: the trailing pwd -P >| …-cwd snapshot is rebuilt
    onto the unwrapped command, so the host keeps tracking the working directory.
  • Security unchanged: detection requires both an eval '<cmd>' and a host
    cwd-snapshot redirect, so a bare eval the model itself chose still hits the
    allowlist's hard block (regression-tested end to end).
• Installer no longer fails on symlinked ~/.claude / ~/.codex (GH #596).
  Dotfiles users symlink their agent config (~/.claude.json,
  ~/.codex/config.toml, …) into a managed repo. The [Critical] symlink hijack protection previously added to config_io::write_atomic then hard-blocked
  every write through such a symlink (refusing to write through symlink), so
  setup/init could no longer register the MCP server or write agent config.
  • Write through the symlink: the new resolve_write_target follows a
    user-managed symlink to its real file and writes there atomically, leaving the
    symlink intact and the dotfile updated — the legitimate dotfiles pattern.
  • Hijack protection kept: following is allowed only when the resolved target
    stays within $HOME; a symlink whose target escapes $HOME is still refused,
    so a planted symlink can never redirect a config write to a system path.
  • Opt-in escape hatch for out-of-$HOME dotfiles: power users who keep their
    dotfiles repo outside $HOME (e.g. /opt/dotfiles) can now allow specific
    trusted roots via the new allow_symlink_roots config key (or the
    LEAN_CTX_ALLOW_SYMLINK_ROOTS env var). It is empty by default (strict
    $HOME-only stays the default) and security-sensitive — like extra_roots, an
    untrusted project-local config can never add a root. The refusal message now
    spells out all three ways forward (CLAUDE_CONFIG_DIR/CODEX_HOME, move under
    $HOME, or allow-list the root) instead of a bare "escapes $HOME".
  • Robust directory setup: a new ensure_dir tolerates a symlinked agent
    directory (and creates a dangling in-$HOME target), and the Claude/Codex
    setup steps now surface a clear error instead of silently swallowing a failed
    create_dir_all. The Claude skill also honors CLAUDE_CONFIG_DIR instead of a
    hardcoded ~/.claude.
  • Read-only / cross-FS fallback shared: config_io now reuses the edit
    tools' atomic-write mechanics (core::atomic_fs), gaining the
    read-only-directory in-place fallback (#459). Regression-tested end to end with
    symlinked claude/codex configs.
• CLI and MCP now always read the same config.toml (GH #594). When an older
  release had baked LEAN_CTX_DATA_DIR into an editor's MCP env, the server ran
  in single-dir mode and read config from the data dir (~/.local/share/lean-ctx)
  while the terminal CLI read it from the config dir (~/.config/lean-ctx), so
  settings silently diverged.
  • Resolver: a LEAN_CTX_DATA_DIR equal to the standard $XDG_DATA_HOME/ lean-ctx is now a data-only pin and no longer collapses config/state/cache
    onto the data dir. Custom/legacy single-dir paths still collapse (unchanged
    back-compat). Deterministic, no filesystem access (#498).
  • Self-healing writers: setup / update strip a stale LEAN_CTX_DATA_DIR
    from the lean-ctx MCP entry across all formats (JSON, Codex TOML, Hermes YAML).
  • Lossless migration: a config.toml stranded in the data dir is relocated
    to the config dir — adopted when canonical is empty, otherwise the
    CLI-authored config wins and the stray copy is archived to
    config.toml.superseded (never deleted).
  • doctor: flags config location — stray config.toml in the data dir, and
    doctor --fix unifies it.

Changed

• Frictionless updates. lean-ctx update — the prebuilt-binary self-updater
  everyone should use — now bounds its DNS/connect/response phases with timeouts,
  so a dead network or unresponsive mirror can no longer make it appear "stuck".
  dev-install prints an up-front notice that it builds from source (a contributor
  workflow that can take minutes) and points end-users at lean-ctx update. When
  the running binary is behind the latest release, the MCP session and
  lean-ctx doctor now surface a one-line "run lean-ctx update" nudge —
  notify-only, never an auto-install (on by default; opt out with
  update_check_disabled / LEAN_CTX_NO_UPDATE_CHECK).

Upgrade

lean-ctx update                 # recommended (auto-downloads + refreshes shell hooks)
cargo install lean-ctx          # or
npm update -g lean-ctx-bin      # or
brew upgrade lean-ctx

Note: After upgrading via cargo/npm/brew, run lean-ctx setup to refresh shell aliases. lean-ctx update does this automatically.

Full Changelog: v3.8.16...v3.8.16