arabold/docs-mcp-server v3.3.0

on GitHub

Breaking changes

• Scraping an existing library version now requires explicit replacement: use CLI --replace or MCP/API replace: true, use refresh to update in place, or use CLI --no-clean to append. Upgrade remote workers before or alongside their coordinators. (#517, #515, #168, #436)
• For authenticated HTTP deployments, set server.publicUrl, connect through /mcp, and use issuer-signed JWT access tokens with audience <public URL>/mcp or auth.audience; migrate away from the removed OAuth proxy and authenticated SSE endpoints. Review previously ignored authentication, read-only and protocol environment/config settings because they now take effect, and register foreign browser origins and otherwise-unrecognized dotted proxy or worker hostnames through server.allowedOrigins or server.publicUrl. (#518, #493, #494)
• MCP servers now send built-in tool guidance when no custom instructions are configured, with guidance tailored to read-only mode. Set server.instructions or server.instructionsFile if you need to replace it. (#535)
• Docker containers now serve HTTP by default when launched without a subcommand, including verbose launches. Select stdio explicitly in Docker-based MCP client configurations; runtime overrides and explicit subcommands remain supported. (#544)

Features

• Give each MCP instance a configurable name, title and assistant instructions, using configuration, environment variables or CLI flags. The server also reports its actual package version. (#512, #511)
• New MCP tools list_pages and read_page let assistants browse indexed URLs and read complete pages, continuing through long pages chunk by chunk. Page lookup supports encoded URLs and path suffixes, and reports ambiguous matches instead of choosing one silently. (#510)
• Scraping keeps available crawl slots busy while slow pages retry, without exceeding maxPages or changing breadth-first link admission. A crawl waits for all running pages to finish storing before it completes. (#520, #490)
• Embeddings use more compact binary storage, and SQLite trims its write-ahead log to 64 MB after checkpoints. Existing embeddings are converted during the database migration. (#523, #509)
• Browser fallback and HTML rendering share one Chromium process per web scrape, reducing duplicate browser processes. Stuck browsers are forcibly stopped when graceful shutdown times out. (#540)
• Remote CLI scrapes and refreshes show live progress for their own job, and MCP over stdio receives worker job updates too. Version removal cancels and waits for unfinished work before deleting data, while coordinators report worker disconnections and reconnect their live updates. (#545, #532, #530)
• Search limits full-text matching to the requested version earlier, avoiding long stalls on large stores. The dashboard activity chart loads more efficiently, counts chunks alongside their page’s original indexing date, and refreshes when jobs or library versions change. (#554, #550, #548)
• Full-text search avoids broad matches from common English words when other searchable words are present. Exact and quoted phrases retain those words, and queries containing only common words continue to work. (#556, #550)
• Docker builds exclude nested dependency directories, environment files, agent worktrees and the downloaded benchmark cache, keeping unnecessary files out of the build context. (#576)

Bug fixes

• Store compaction reclaims deleted full-text content as well as database free space, allowing a store with all pages removed to return to its empty size. (#522, #476, #509)
• Scraping relaunches Chromium if it exits mid-crawl and retries a page interrupted by the crash once. Concurrent pages share a single browser launch, and cleanup no longer closes a replacement browser another page is using. (#538, #531)
• Browser scraping works when Playwright has only the full Chromium build installed, including installations made with --no-shell. (#539)
• Browser fallback leaves Sec-Fetch-* page headers to Chromium, preventing intermittent failures that stop scripts, styles and data from loading. (#541)
• Failed crawls stop and wait for running pages before browser cleanup, preventing leaked Chromium processes. Browser fallback also handles downloadable resources such as YAML and retries pages interrupted when a shared browser is replaced. (#543)
• Remote CLI scrape, refresh and remove commands finish when the worker’s job ends instead of waiting indefinitely for an event. Worker failures retain their error message, and missing jobs or repeated lookup failures produce an error. (#537, #530)
• Unresponsive browser renders have an overall deadline instead of leaving a scrape stuck forever. When rendering times out, scraping keeps the fetched HTML and replaces the browser if it cannot be cleaned up safely. (#525)
• Local CLI scrapes and refreshes no longer mark another process’s jobs as failed. Read-only servers also leave unfinished jobs alone, allowing them to serve searches beside an indexing worker. (#546)
• CLI command failures show a concise error instead of full help text and a stack trace; --verbose retains diagnostic stacks, including configuration errors. Commands that report their own failure preserve their nonzero exit status. (#555)
• Telemetry sends URL hostnames and protocols rather than full URLs, and query lengths rather than search text. Error reports redact URLs, paths and tokens, sanitize stack traces, and use the installation ID from the selected store. (#558)
• Scraping preserves iframe content in the correct order when multiple iframes finish rendering at different times. (#561)
• Hybrid search ranks a chunk only in the search methods that actually found it, preventing missing full-text or vector matches from receiving an extra score. (#566)
• Local directory scraping uses a consistent filename order across platforms, avoiding Windows-specific differences in indexing order. (#560)
• Search groups nearby matching chunks into coherent passages, preserves structural context in code results, and breaks score ties by page URL and chunk position rather than indexing order. (#564)
• Scraping no longer indexes script source or a document title as page content when the body has no visible text. Titles remain available as page metadata. (#571)
• Search results include the actual content URL, so web interface links reach the served document when its indexed identity differs, such as a Markdown page stored without its .md extension. (#570)
• Browser scraping checks redirect hops against the access policy across pages, subresources and cross-process iframes, preventing redirects from reaching blocked private or off-allowlist addresses. (#568)
• Fastly client challenge pages produce an explicit error instead of being indexed as documentation, including when returned with HTTP 200 or through the browser’s request fallback. (#569)
• Servers sharing a store leave actively heartbeating jobs to the process running them and recover or fail only stale jobs. Conditional claims prevent two servers from handling the same interrupted version, and open views receive status changes. (#559)
• HTML rendering checks every subresource redirect against the access policy and keeps authorization headers off cross-origin redirects. Frames and framesets use their rendered content instead of an incorrectly shared raw-response cache entry. (#572)
• Browser fallback filters connection-level page headers that Chromium rejects, preventing intermittent navigation failures that otherwise skip JavaScript rendering. (#577)

Internal changes

• Test suites use isolated temporary stores and configuration, block unintended network access, and replace weak or timing-dependent checks with assertions against real stores, pipelines and browsers. Embedding retry tests run without a real API key, and CLI tests rebuild stale output before running. (#524, #536, #562, #563, #567)
• Documentation is audited against current behavior, including configuration, CLI flags, MCP tools, Docker usage, embedding providers and telemetry. Architecture and contributor guidance describe the React web interface and current development conventions. (#557)
• Release automation generates pull-request-based notes through Copilot CLI with no tools available, validates their structure, and falls back to commit-based notes if generation fails. (#573)