xberg-io/xberg v1.2.9 on GitHub

Added

(ocr): the near-empty PDF fallback, embedded-image recognition and the per-page selection gate each have a setting of their own. All three behaviours were keyed on one condition -- whether ExtractionConfig.ocr was present -- so a caller could not have one without the others. On a 731-page mixed native and scanned document that cost 102.8 s with an OCR block against 57.1 s without, while dropping the block lost the near-empty fallback (a scanned page carrying a text stamp fell from 996 to 6 characters) and picture text in DOCX files (250 to 36). ocr_near_empty_fallback, ocr_scanned_page_quality_gate and ocr_embedded_images are each Option<bool> whose None reproduces exactly the previous derived behaviour, so no existing caller changes; setting one to Some(true) without an ocr block reads OcrConfig::default() thresholds and still requires a registered automatic backend. A third copy of the embedded-image predicate was found and folded into the same helper -- extraction/image_ocr.rs returned early on config.ocr.is_none() independently of the two known sites, so the new setting would otherwise have been accepted and then silently ignored. ExtractionConfig uses deny_unknown_fields: a config file written for an older version still parses, but one that sets any of the three is rejected outright by a pre-1.2.9 binary. (GH#1752)

Fixed

(pdf): a two-column split no longer lands inside a figure caption instead of the gutter beside it. On a page whose left column is a figure and its 7.2 pt caption, the XY-cut's chosen valley ran 254 pt -- from the left column's edge all the way to the right column's first glyphs -- because a figure and a small caption fall below the density threshold nearly everywhere. Splitting that run at its arithmetic midpoint put the boundary at x 180.6, inside the caption, so every caption line the producer wrote as several spans (around a superscript, an italic, or simply word by word) had its right-hand spans assigned to the body column, where top-to-bottom sorting interleaved them with the section's own lines. The split is now placed at the deepest point of the run -- the centre of its widest lowest-density stretch -- which on the reporting page is x 301.6, left of the right column's first glyph and right of every caption span. The reporter could not supply a reproducing PDF but attached the projection their page produced, and it is committed as a regression fixture: every figure they quote is reproduced, including the 12 pt empty core that sits under min_valley_width, so the width test keeps applying to the run as a whole and not to its core. Three narrowings were needed to make this safe, each measured across 230 documents against binaries differing only in this code. The relocation applies only when the run is too wide to be a gutter (a real gutter is 15-80 pt on a ~500 pt region; this one is 48.6% of it), only when the midpoint actually lands on content, and only to a candidate that no span straddles -- the projection omits spans under two non-whitespace characters, spans wider than 55% of the region, and everything past each span's estimated text core, so a zero in the profile does not mean no glyphs are there. Without those, the change rewrote 23 of 230 documents and pulled words apart at single-character spans. With them, 9 of 230 change: by absolute dictionary-valid word count 4 improve and 2 regress, the net is +65 words (+8 excluding the single largest gain), and 53 non-word tokens are removed against 24 created. One document regresses by 18 words. (GH#1763)
(pdf): a page whose content stream decodes to more than 100 MB no longer loses its entire text layer. Two decompression caps disagreed. FlateDecoder enforces its own 256 MB ceiling, but the cap applied when a caller passes no ParserOptions -- which every page-content decode does -- was a separate hardcoded 100 MB. A stream the flate stage had just produced and accepted was therefore rejected one line later, and per-page text extraction turns that error into an empty page behind a single warning. The carrier was page 574 of a statistics textbook: a 24 MB content stream decoding to 194 MB at a ratio of 8:1, nothing like a bomb and well inside the flate stage's own ceiling. The pipeline cap is now the same number the flate stage enforces, so XBERG_NATIVE_PDF_MAX_DECOMPRESS_MB governs the crate's decompression ceiling rather than governing one stage while a hidden second limit overrode it downward. Bomb protection is unchanged: the 100:1 ratio check is what separates a bomb from a merely large stream, it is evaluated per stage against the original input so chained filters cannot multiply past it, and peak allocation per stream does not move -- the flate stage already materialised 256 MB before this change and the old cap merely discarded it afterwards. Measured across 229 documents and 12,526 pages against pdf_oxide 0.3.78: one page changed, from 0 to 2,518 non-whitespace characters, matching pdf_oxide exactly, with no page regressing. The two page-level guards that can discard a whole page without extracting anything now log which page they dropped and why. ParserOptions::strict, ::lenient and ::very_lenient derive their cap from the same source rather than carrying their own hardcoded copy, so a caller that supplies options is governed by the environment override too; very_lenient keeps a proportionally higher ceiling, matching the doubled compression ratio it already allows. (GH#1754)
(pdf): a deeply nested page tree returns an error instead of aborting the process. Reading the page tree recursed once per level across five separate walkers, only one of which had a depth cap -- and that one silently returned a page count of zero past it. A synthetic PDF whose page tree is a chain about 2,500 levels deep overflowed a 2 MiB thread stack in a release build, and about 370 levels in a debug build. A stack overflow aborts the process rather than raising a catchable panic, so one uploaded PDF killed the consuming process and every job it had in flight. All five walkers now share a MAX_PAGE_TREE_DEPTH of 256 -- more pages than any real document can address at a fan-out of two or more, and an order of magnitude below where the stack dies -- and return a recursion-limit error past it. Every caller already handled that error and degraded gracefully; nothing had ever constructed it. One of the five disagreed about what to do next: the page-reference walker propagated the error and so returned no pages at all, where the other four skipped the offending branch and carried on, meaning a single bad branch in an otherwise sound tree lost the whole document. All five now degrade. (GH#1755)
(pdf): a stray MoveTo no longer stretches a path's bounding box to the whole page. A MoveTo that no segment follows starts a subpath that paints nothing, but it was still unioned into the path's extent. The carrier is a footer band drawn as 0 42.63 595.28 -28.35 re 0 842 m f*: two operations, so the page-frame filter added for GH#1656 does not recognise it as a rectangle, and the lone MoveTo inflated a 28 pt band to 595 x 799 pt. extract_tables_native's strict configuration then fabricated a 15-row table spanning the page, whose first row welded a section heading to the prose beside it -- and because that pass hands its claimed pages to the bordered pass as skip_pages, the page's real table was never extracted at all. Rectangles with a negative stored width or height are now normalised as well: PathExtractor::rectangle stores the CTM-transformed delta, so a flipping CTM yields a negative extent that the previous unordered min/max never corrected. The two byte-identical copies of the bounding-box routine are now one function. (GH#1759)
(pdf): a table's sparse rows no longer outvote the page gutter. Excluding a table's full rows from the two-column gutter vote (GH#1742) left its sparse lines voting -- a wrapped column header's second line, a standard-deviation line, a row whose long label closes the gap to its first value, a group row with empty cells -- and with the full rows gone those became the majority. On the carrier the median moved from 383.3 to 449.1, inside the right column, and the rescue that would have recovered it missed the real gutter corridor by 1.5 pt against a 148.8 pt cap. Lines that follow the column grid the excluded rows establish are now excluded too, but only when the gap they vote for lies between two of that grid's own columns: a table row's rightmost gap is the page gutter, and on the GH#1742 reproducer it is the only evidence 16 of its rows carry. (GH#1756)
(pdf): a wrapped numbered heading keeps its continuation line. Three separate conditions each cut a heading in two and gave its tail to the paragraph beneath. A heading stopping short on a ragged right edge was refused by a fixed two-font-size tolerance even where the next word demonstrably could not have fitted; the width of that word is now estimated from the continuation line's own mean character width, with the fixed tolerance kept as a floor so the estimate can only ever admit more. A continuation opening with a capital was refused outright; it is now accepted when it keeps the heading's own style while the line beneath does not and the line before fills the column. And a three-line heading measured its middle line against its own last line rather than against the column, because the search for the line beneath skipped only lines sharing a baseline; it now skips lines that continue the same heading run, floored at the heading's own right edge so a stub caption cannot stand in for the column. (GH#1758)
(pdf): a heading run bordering a two-column gutter no longer welds in the other column's opening line. find_heading_runs's same-line fold runs before any column split exists, and it admitted an other-column span onto the run whenever that span merely shared a row with the run's last line -- on the carrier, a bold heading opening the right column 0.25 pt below the wrapped left heading's first line, 78.9 pt across the gutter. The fold now takes the page's own detected column gutter, already computed for the two-column repair, and treats a span lying entirely on the far side of it as belonging to a different column: neither folded into the run nor allowed to close it. The gutter has to reach every call site that partitions with a heading run -- postprocess_spans, partition_region and apply all thread it through, and threading it into one alone left the others reproducing the defect. A width-based stand-in was measured and rejected: on a page with no detected gutter the gaps such a threshold would have to reject are the same size as the gaps inside a legitimate heading line (median 82 pt, half at or above this page's own 79 pt gutter, on one corpus document), so no cutoff separates the two populations. A page with no detected gutter keeps the unconditional fold exactly as before. One document of 230 changed. (GH#1757)
(pdf): a regular body pitch no longer cuts a region where its rounding happens to land. The heuristic table pass clusters words into regions wherever two consecutive row centres lie more than 1.8 times the median word height apart, working entirely in whole units. segments_to_words rounds each word's top to a whole unit, so a body pitch of 10.45 pt is observed as an alternating 10 and 11 while the threshold for a median height of 6 truncates to 10 -- every 11 cut and every 10 did not, placing region boundaries wherever the rounding fell rather than where the page's layout changed. A band of exactly three rows could come out alone, and being two text columns wide it reached post-processing, where every prose test needs at least three rows with both cells filled and a three-row band has one. On a two-column research page that turned the last lines of one column's paragraph and the other column's References heading into a 3 x 2 table, and took the heading out of the element stream entirely. Row centres and the threshold are now carried as real numbers with a whole unit of slack for the rounding, and below the row count at which the prose tests can fire a region must show positive table evidence -- numeric value cells or a row of short column labels -- rather than merely failing to be rejected. Across 40 corpus documents four changed, each losing a false table: a shell listing, an apt sources snippet, a two-item bullet list and a prose paragraph. (GH#1760)
(pdf): a four-column table no longer pulls the two-column split into itself. Two gaps combined. The rule that stops a table's rows voting for the page gutter keys on four or more internal gaps, and a four-column table has three, so its rows still voted and the split landed between two of its columns. The rescue could not recover it either: the search for an empty corridor is made page-wide, while the split it rescues is applied per band, so a full-width table elsewhere on the page closed the gutter for every band below it. The corridor question is now asked of the band the split will be applied to, including the boundary lines that delimit it -- a boundary line belongs to the band above and the band below alike, and it is the caption or introductory line that legitimately closes a table's own cell gaps. (GH#1762)
(pdf): Merged cells and zebra-striped rows stay in their ruled table. The GH#1601 check that keeps a phantom cell out of the empty gap between two grids required a drawn rule on both sides of every cell, so a cell with an edge that has no line of its own in that row was dropped and its text spilled into paragraphs after the table: horizontally merged cells and full-width section rows, whose inner column rules stop at the row, and the unshaded rows of an alternately shaded table, whose outer edges exist only as the shaded neighbours' fill sides. A cell now also forms when its band between its own top and bottom rules belongs to a grid: a drawn rule crosses the band strictly inside those rules' ends, or drawn rules close both ends. The GH#1601 gap has neither, and a rule at one end only (an enclosing table's column rule) or beyond them (a page frame) still does not count. (GH#1765)
(image): the decode budget no longer compares a pixel count against a byte budget. ImageDecodeBudget::validate rejected an image when its pixel count exceeded security_limits.max_content_size, a number of bytes. The clause decided nothing, because every caller also passes a byte count at least as large, but it made the rejection message self-contradictory -- a 500-byte image reported as exceeding a 1,000-byte budget -- and it would have become wrong the moment either side changed. The pixel comparison is gone; the zero-dimension guard and the byte comparison are unchanged. (GH#1761)
(ocr): disable_ocr now also stops OCR of images embedded in a container document. runs_ocr_on_embedded_images consulted the ocr block and the per-image run_ocr_on_images switch but never effective_disable_ocr(), so a config that set disable_ocr: true still sent DOCX, PPTX, ODT and HTML pictures to recognition -- against that field's documented meaning, "OCR is skipped for all document types". Because needs_image_data ORs that predicate in and needs_image_processing ORs the result in turn, such a config also paid for full image decode it had asked not to do. The predicate now gates on effective_disable_ocr(), matching how extractors/image.rs and the extraction engine already treat the flag. force_ocr and an explicit ScannedPages strategy keep being rejected outright at validation, because those are explicit requests rather than automatic triggers.
(pdf): a form's label-to-value gap is no longer mistaken for a column gutter. Of the three detectors behind the page gutter used by the heading-run pre-pass, only density_central_gutter had no structural gate: on a twenty-row label:value form it returned a gutter at x 244.06 where both siblings correctly declined. A spuriously detected corridor only reaches spans whose tops agree within a point, which is precisely the shape of a table or form row, so two cells sharing a baseline could silently drop out of the heading run they belonged to. It now applies the same region classification that rejects Table and Form pages in the sibling detector. The deliberately narrower corridor this detector accepts, against the wider gutter the multi-column predicate requires, is documented as intended rather than left an unexplained disagreement.
(ocr): a PDF whose font puts a single-byte code page's letters at Latin-1 positions now routes to OCR instead of returning mojibake. A simple TrueType font placing Cyrillic glyphs at cp1251 code points, with no /Encoding and no /ToUnicode, extracted as Âåñåííÿÿ êàìïàíèÿ…, and the default auto strategy kept it: ocr_used false, implausible_text_pages empty, no warning. The language/dictionary plausibility check added in GH#1696 could not catch this by construction — it flags text only when language detection is both unreliable and low-confidence, and on this input whatlang is confident and reliable (Welsh, 4 of 5 chunks reliable, mean confidence 0.83). It is reporting the truth about character shape and nothing about content, so no adjustment of the confidence bar reaches this page without dragging in real prose. The added signal is orthogonal: it asks which codepoints appear rather than whether their pattern is recognised. A single-byte code page's alphabet read as Latin-1 puts nearly every letter in U+00C0-U+00FF, a block holding no unaccented ASCII letters, so no Latin-script orthography — all of which need them — can approach the 0.90 threshold. Measured: the reported text scores 1.00, while the most diacritic-dense real samples available score 0.18 (Icelandic), 0.12 (Portuguese), 0.10 (French) and 0.00 (Welsh, the language assigned to the mojibake itself); across 229 corpus documents the maximum is 0.0065 and the median 0. That corpus is predominantly English and so cannot speak for the population actually at risk, which is why the language samples are pinned as tests rather than the corpus figure alone. The existing three-prose-chunk floor still runs first, so a short page cannot reach this clause. (GH#1767)