github HKUDS/LightRAG v1.5.5
on GitHub

4 hours ago

πŸŽ‰ AddSmart Heading Recognition Feature For Word Document

Starting from v1.5.5, LightRAG's built-in DOCX parser introduces the Smart Heading feature -- enabling better section-based chunking even for poorly formatted DOCX documents.

Many DOCX documents have visually clear section structures but lack properly configured Word outline levels, or retain incorrect heading styles from editing, copying, and concatenation processes. In the past, these issues could cause section titles to be treated as body text, or regular text to be misidentified as headings, resulting in chunking boundaries that deviate from the document's true semantic sections. For P chunking strategies that rely on heading structures, this creates fragmented context or mixed content from different sections, affecting subsequent indexing and retrieval.

Smart Heading targets such poorly formatted but semantically clear documents, restoring their true heading structures as much as possible:

  1. Recover Missing Section Titles: Identify content that visually appears as titles but lacks properly configured outline levels, reducing loss of section structures.
  2. Reduce False Positives of Body Text as Headings: Restore regular body text, complete clauses, captions, dates, and version notes that incorrectly carry heading styles back to body text, avoiding meaningless chunking boundaries.
  3. Improve Inconsistent Heading Hierarchies: Reduce issues such as inconsistent sibling heading levels, inverted parent-child hierarchies, and skipped levels, making section structures more coherent.
  4. Distinguish Multiple Contents in Merged Documents: Identify main titles and document boundaries of each piece of content, preventing section level interference between different documents.
  5. Reduce Noise from Long Tables of Contents: Compress excessively long table of contents, minimizing the interference of duplicate information on indexing and retrieval.

By providing more reliable section structures for subsequent chunking, Smart Heading helps maintain semantic associations between titles and their corresponding body text, making chunking results align better with the document's original content organization.

Smart Heading is an optional capability of the Native DOCX parser and requires activation to take effect. For enabling and configuration instructions, please refer to: docs/FileProcessingPipeline / δΈ­ζ–‡η‰ˆ.

What's New

What's Changed

  • fix(webui): restore edges after a drag that ends without camera animation by @danielaskdd in #3325
  • fix(api): warn when the server is network-exposed without authentication (GHSA-mmg5-8x8q-v934) by @danielaskdd in #3328
  • fix(external): prevent task ID URL path injection by @VectorPeak in #3324
  • πŸ”’ fix(api): gate /health configuration disclosure behind authentication (#3294) by @danielaskdd in #3329
  • πŸ”’ fix(docker): patch dependency CVEs, run as non-root, pin bookworm base (#3296) by @danielaskdd in #3330
  • Fix docs links and environment typos by @Wenjunyun123 in #3333
  • chore(deps): bump react-router-dom from 7.17.0 to 7.18.0 in /lightrag_webui in the react group by @dependabot[bot] in #3334
  • chore(deps): bump the ui-components group in /lightrag_webui with 12 updates by @dependabot[bot] in #3335
  • chore(deps): bump the frontend-minor-patch group across 1 directory with 2 updates by @dependabot[bot] in #3337
  • docs(webui): document Bun 'bin metadata file' error on WSL by @danielaskdd in #3339
  • chore(deps): bump actions/cache from 5 to 6 in the github-actions group by @dependabot[bot] in #3343
  • docs: add LLM timeout troubleshooting to Key Configuration Guide by @danielaskdd in #3345
  • docs(api-server): clarify X-API-Key and Authorization are mutually exclusive per request, fixes #2632 by @hata33 in #3358
  • chore(deps-dev): bump the ui-components group in /lightrag_webui with 2 updates by @dependabot[bot] in #3348
  • chore(deps): bump mermaid from 11.15.0 to 11.16.0 in /lightrag_webui in the content-rendering group by @dependabot[bot] in #3350
  • chore(deps-dev): bump the build-tools group across 1 directory with 5 updates by @dependabot[bot] in #3361
  • chore(deps): bump the frontend-minor-patch group across 1 directory with 3 updates by @dependabot[bot] in #3351
  • fix(kg): close Milvus and Qdrant clients in finalize() to release connections by @hata33 in #3360
  • chore(deps): add httpx2 test dependency for starlette testclient by @danielaskdd in #3366
  • docs(rerank): document qwen3-rerank via cohere binding (flat format) by @danielaskdd in #3368
  • test: isolate Bedrock health auth env by @danielaskdd in #3369
  • fix(llm): close AsyncOpenAI client in nvidia_openai_embed by @hata33 in #3371
  • πŸ“ docs(contributing): require English for Git commit messages by @danielaskdd in #3390
  • chore(deps): bump react-router-dom from 7.18.0 to 7.18.1 in /lightrag_webui in the react group by @dependabot[bot] in #3383
  • chore(deps): bump the ui-components group in /lightrag_webui with 11 updates by @dependabot[bot] in #3384
  • chore(deps-dev): bump the build-tools group in /lightrag_webui with 3 updates by @dependabot[bot] in #3385
  • fix: preserve cached xlsx formula values by @eteriall in #3375
  • fix: handle docling JSON result envelopes by @ekkoitac in #3344
  • fix(kg): release Redis client and pool reference in finalize() by @hata33 in #3370
  • fix(chunker): reject chunk_overlap >= chunk_size in token windowing by @santhreal in #3396
  • fix(concurrency): propagate enqueuer's contextvars into priority queue workers by @freya0926 in #3386
  • fixed an issue where js protoype names would lead to the graph not being rendered by @Xaverrrrr in #3357
  • fix(opensearch): handle DocStatus bulk item failures by @VectorPeak in #3354
  • Fix description accumulation on reprocess in merge_nodes_and_edges by @santhreal in #3395
  • fix(core): guard entity_vdb.upsert on edge-merge new-endpoint path (#3367) by @danielaskdd in #3398
  • fix(core): stop reprocess accumulation of edge weights (#3367) by @danielaskdd in #3399
  • Fix ainsert_custom_chunks silently skipping knowledge-graph build by @santhreal in #3394
  • fix(core): make ainsert_custom_chunks respect the pipeline busy contract (#3352) by @danielaskdd in #3401
  • fix: expose thinking token budget in role config by @Sanjays2402 in #3406
  • chore(deps): bump the ui-components group in /lightrag_webui with 10 updates by @dependabot[bot] in #3410
  • chore(deps-dev): bump the build-tools group in /lightrag_webui with 3 updates by @dependabot[bot] in #3411
  • chore(deps): bump the frontend-minor-patch group in /lightrag_webui with 3 updates by @dependabot[bot] in #3412
  • fix(share-storage): Improve shared-storage state atomicity and organization by @danielaskdd in #3417
  • refact(core): recoverable document mutations β€” write-ahead anchors, journaled custom chunks, scan rollback (#3400, all 5 phases) by @danielaskdd in #3416
  • fix(qdrant): recover vectors when payload id is missing by @VectorPeak in #3362
  • fix(rerank): raise ValueError for max_tokens < 1 by @santhreal in #3419
  • fix(utils): return [] for non-positive split_text_by_token_limit budget by @santhreal in #3420
  • fix(utils): return 0.0 for cosine_similarity on zero vectors by @santhreal in #3421
  • fix(api): Stop Leaking Raw Exception Text Through Raw Exception Messages in API Error Responses (CWE-209) by @danielaskdd in #3422
  • fix(auth): constant-time plaintext password comparison (GHSA-c759-cx9p-mrwq) by @danielaskdd in #3423
  • feat(api): rate-limit /login to stop brute-force attacks (GHSA-frch-4w6v-q5xx) by @danielaskdd in #3424
  • fix(parser): block IPv6-transition SSRF bypass of native-md image guard by @danielaskdd in #3426
  • fix(postgres): harden _dollar_quote against seam-collision breakout by @danielaskdd in #3427
  • perf(kg): snapshot JSON storage with DictProxy.copy() instead of dict(proxy) by @danielaskdd in #3433
  • fix(utils): reject non-finite values in parse_optional_float by @santhreal in #3430
  • fix(webui): sanitize untrusted chat Markdown to block stored XSS (GHSA-xpjq-3w4w-w5wr) by @danielaskdd in #3437
  • fix(webui): show instantaneous lock counts as m_lock/a_lock in status card by @danielaskdd in #3440
  • test(pipeline): mark pipeline tests offline for CI by @ysys143 in #3441
  • refactor(pipeline): operation-scoped lock-free logger for hot pipeline-status logging by @danielaskdd in #3432
  • fix(llm): do not cache token-limit-truncated LLM responses by @DivyaNarahari97 in #3446
  • test(pipeline): cover audit_kg_integrity's orphan detection path by @ysys143 in #3443
  • fix(lightrag): fall back when COSINE_THRESHOLD env is empty by @santhreal in #3447
  • fix(llm): extend truncation marker to gemini/ollama/bedrock and the multimodal analysis cache by @danielaskdd in #3448
  • fix(multimodal): keep EXTRACT prompt within budget and reject starved content by @santhreal in #3428
  • fix(utils): keep spaced <|> out of tuple delimiter repair by @santhreal in #3438
  • fix(utils): close compute_args_hash cache-key collision via length-prefixed encoding by @hata33 in #3435
  • fix(json): unify LLM JSON object recovery into tolerant_load_json_dict by @danielaskdd in #3449
  • ci: install offline-test deps via uv sync --frozen (lockfile) by @danielaskdd in #3450
  • fix(parser): fall back when CHUNK_R_SEPARATORS env is empty by @santhreal in #3452
  • fix(lightrag): fall back when env-backed int defaults are empty by @santhreal in #3451
  • test: stabilize pytest isolation and spaCy model gating by @danielaskdd in #3453
  • fix(llm): honor LLM_TIMEOUT in Bedrock driver and server wiring by @danielaskdd in #3454
  • πŸ› fix(parser): keep drawing path local-only, carry external image refs in src by @danielaskdd in #3455
  • refact(api): expose supported file types + parser capability matrix to WebUI by @danielaskdd in #3457
  • fix(base): fall back when QueryParam env-backed ints are empty by @santhreal in #3461
  • fix(utils): treat empty JSON store files as missing by @santhreal in #3462
  • fix(parser): fall back when CHUNK_V_BUFFER_SIZE env is empty by @santhreal in #3463
  • fix(api): reject query text that strips below min_length by @santhreal in #3468
  • fix(api): reject whitespace-only insert text bodies by @santhreal in #3469
  • βœ… test: heal module identity and derive the Bedrock args rig from the real parser (LR2 P0) by @danielaskdd in #3482
  • πŸ› fix(api): stop serving when the gunicorn master is gone by @danielaskdd in #3494
  • build(deps): bump actions/setup-python from 6 to 7 in the github-actions group by @dependabot[bot] in #3495
  • build(deps): bump the ui-components group in /lightrag_webui with 13 updates by @dependabot[bot] in #3474
  • build(deps): bump react-i18next from 17.0.9 to 17.0.11 in /lightrag_webui in the frontend-minor-patch group across 1 directory by @dependabot[bot] in #3476
  • build(deps-dev): bump @types/node from 25.9.5 to 26.0.1 in /lightrag_webui by @dependabot[bot] in #3477
  • build(deps-dev): bump the build-tools group across 1 directory with 6 updates by @dependabot[bot] in #3500
  • fix(pipeline): fall back when VLM image budget envs are empty by @santhreal in #3464
  • fix(api): require string content on query conversation_history messages by @santhreal in #3491
  • fix(utils): fall back when float env values are non-finite by @santhreal in #3492
  • fix(operate): keep default weight when relationship description is numeric by @santhreal in #3490
  • πŸ› fix(api): stop the ollama bracket prompt from corrupting the query by @gaurav0107 in #3498
  • fix(api): reject whitespace-only graph entity names by @santhreal in #3471
  • fix(utils): return no chunks for non-positive weighted polling max by @santhreal in #3493
  • fix(api): reject non-finite semantic breakpoint amounts by @santhreal in #3470
  • Fix multimodal analysis cache migration by @danielaskdd in #3504
  • fix(api): migrate to standalone uvicorn worker by @danielaskdd in #3506
  • fix(api): stream directory scan entries by @danielaskdd in #3507
  • refact(chunker): scan all blocks for references by default by @danielaskdd in #3508
  • fix(webui): reliable follow-scroll recovery and jump-to-bottom button by @danielaskdd in #3509
  • fix(operate): partition the query answer cache by conversation_history by @danielaskdd in #3510
  • fix(parser): fall back when CHUNK_* size/overlap envs are empty by @santhreal in #3460
  • fix(operate): iterate tasks list, not done set, when collecting chunk_results by @marsch in #3512
  • fix(retrieval): improve diagnostics for missing chunk vectors by @chakshu-dhannawat in #3479
  • docs: import AGENTS.md in Claude Code by @jasonewillis in #3517
  • fix(parser): configure optional Docling suffixes by @bm1016bm-svg in #3501
  • fix(rerank): ignore out-of-bounds document indices in chunk score aggregation by @santhreal in #3514
  • fix(pipeline): fall back to sidecar LaTeX when equation field is omitted or aliased by @Solaris-star in #3515
  • feat(parser): configure optional MinerU suffixes by @danielaskdd in #3520
  • fix(parser): stop reading a space-padded signature(落款) as a centered heading by @danielaskdd in #3521
  • fix(parser): stop a merge from stranding an outline heading; render CJK counting numFmt by @danielaskdd in #3526
  • fix(rerank): safely handle None index or score in aggregate_chunk_scores by @santhreal in #3518
  • build(deps): bump the react group in /lightrag_webui with 2 updates by @dependabot[bot] in #3522
  • build(deps): bump the ui-components group in /lightrag_webui with 13 updates by @dependabot[bot] in #3523
  • build(deps-dev): bump @types/node from 26.0.1 to 26.1.1 in /lightrag_webui in the frontend-minor-patch group by @dependabot[bot] in #3525
  • build(deps): bump katex from 0.17.0 to 0.18.1 in /lightrag_webui in the content-rendering group across 1 directory by @dependabot[bot] in #3524
  • feat(parser): record numbering counts past a converter's domain by @danielaskdd in #3528
  • fix(parser): stop a heavy member from swamping an outline merge owner by @danielaskdd in #3527
  • fix(rerank): guard malformed provider results by @danielaskdd in #3529
  • fix(api): remove user-supplied sentence_split_regex (ReDoS, GHSA-32jh-39m7-8x84) by @danielaskdd in #3530
  • fix(api): bound the token-renewal cache and stop pre-authorization writes (GHSA-3wg5-5w54-3rfm) by @danielaskdd in #3531
  • fix(docx): tolerate editorial drift in title-block locate-back by @danielaskdd in #3532
  • fix(pipeline): keep the in-batch feeder alive while auto-rescan is dirty by @danielaskdd in #3533
  • fix(api): match WHITELIST_PATHS against the route path, not the mounted path by @danielaskdd in #3534

New Contributors

Full Changelog: v1.5.4...v1.5.5

Check out latest releases or
releases around HKUDS/LightRAG v1.5.5

Don't miss a new LightRAG release

NewReleases is sending notifications on new releases.

Get notifications