krrome/docling-hierarchical-pdf v0.2.0

Speed-up and bug-fixes

on GitHub

This release reworks how ResultPostprocessor.process() moves the content of a document below its headings. The
detected headings and their levels are unchanged; what changes is the document tree built around them. The result
keeps lists, captions and the reading order intact, never loses content, and is much faster on long documents.

The tree looks different from 0.1.x in the cases below. If you work with the tree directly (JSON or HTML export,
chunkers, your own code walking doc.body), please check them against your documents.

Fixes

• Lists stay lists. Items used to be taken out of their lists and placed directly below the heading, leaving
  the lists empty. In markdown, each item then became a list of its own, separated by blank lines.
• Nested lists keep their nesting and their order. Sub-lists were flattened, and later items of a list could
  end up before the sub-items of earlier ones.
• Pictures keep their captions. Captions used to be taken out of their pictures.
• The reading order is kept. Content laid out in columns or tables could be reordered, e.g. an assignment of
  amounts to offices was output as all offices followed by all amounts.
• Top-level headings found inside lists become top-level sections. Numbered headings are often recognized as
  list items. A top-level one stayed inside its list, nested below the previous section, e.g. "2. Usage" ended up
  inside "1. Introduction". It is now a section of its own, like the other top-level headings.
• Lists split by a heading keep their items in lists. Items before the heading stay in the list; items after
  it go into a list continuing it below the heading.
• No content is lost when the bookmarks disagree with the reading order. When a PDF's table of contents put a
  heading below one that comes later in the document, whole sections could disappear from the result.
• No hidden changes when exporting. Because list items ended up outside of lists, docling repaired the document
  on export (e.g. export_to_markdown()): it moved those items into new lists and renumbered their refs, which
  was also slow on long documents. Results of 0.2.0 need no repair.

Performance

The processing time now grows linearly with the size of the document, instead of quadratically: 80,000 items take
about 1 second; 0.1.x took about 30 seconds for 4,000 items already, and four times as long for each doubling. The additional memory needed is small, around
80 bytes per item.

Changes you may notice

• The markdown of affected documents changes: lists are rendered as lists, sections that were nested in a list
  get blank lines between their paragraphs, and reordered passages are back in their original order.
• Lists split by a heading produce additional list groups (new #/groups/N entries) below that heading.
• A list that only held a heading is left behind empty, as before. Empty groups don't appear in any export.

Compatibility

No API changes. Tested with docling 2.53.0 to 2.139.0 and docling-core 2.48.0 to 2.101.1, and compared with 0.1.9
on real-world PDFs (with and without bookmarks, with lists, tables and figures): same headings and hierarchy, with
the differences listed above.