xberg-io/xberg v1.3.9

on GitHub

Added

• (config): strict_parse controls what happens when a text-like document cannot be parsed in its own format.
  By default (false), a YAML, JSON, TOML or XML file that fails to parse, an XML or SVG file that trips
  security_limits.max_entity_length, and a CSV or TSV file that trips security_limits.max_table_cells are extracted
  as plain text instead of failing, and the reason is recorded in processing_warnings (source parse-fallback).
  Plain-text recovery is truncated to security_limits.max_content_size with a warning. Set strict_parse = true
  (CLI --strict-parse true, environment XBERG_STRICT_PARSE) to keep the error. The limits are not loosened, and
  limits aimed at hostile input (archive size, nesting, XML depth) still fail the extraction.

Changed

• Updated dependencies, including tree-sitter-language-pack 1.22.1 (25 refreshed grammars), zip 9.0.2, biblib 0.10,
  html-escape 0.3 and unhwp 0.19. ort stays at 2.0.0-rc.13 because rc.14 requires an Environment for every
  session builder.

Fixed

• (extraction): a Helm template or other unparseable YAML, JSON, TOML or XML file no longer fails the extraction.
  The content is returned as plain text with a warning; strict_parse = true restores the error. (GH#2100)
• (extraction): an oversized XML entity or table no longer discards a text file. A CSV over
  security_limits.max_table_cells or an XML or SVG file over security_limits.max_entity_length is extracted as plain
  text with a warning naming the limit that tripped; both limits remain adjustable through security_limits.
  (GH#2101)
• (release): the Helm chart is published even when the CLI image build fails. The chart publish was gated on the
  aggregate result of the image manifest job, so a CLI image leg timing out skipped the chart for 1.3.6 to 1.3.8 while
  the container images shipped. It is now gated on the xberg and xberg:<version>-core manifests resolving, the only
  images the chart references. (GH#2106)
• (pdf): image bounding boxes are correct for images drawn inside a Form XObject with a /Matrix. The form
  /Matrix was applied after the page transform instead of before it, so the form's page translation was scaled by
  its own matrix and ExtractedImage.bounding_box landed in the wrong place (a regression since 1.2.7). (GH#2103)
• (pdf): a narrow margin-note column is read before the main text again. A column of notes set on its own leading
  beside the main text (no note line shares a baseline with a body line) gave the two-column repair too few
  gutter votes once a table's grid rows stopped voting, so the page was read line by line across both columns and
  the margin label was welded to the main heading. A page whose only whitespace corridor is a gutter narrower than
  the native XY-cut's 15 pt valley is now split at that corridor. (GH#2102)
• (rendering): a block with many inline annotations renders in linear time. Building the inline nodes of one
  paragraph rescanned every later annotation for each annotation, so Markdown or chunked extraction of a block dense
  with emphasis, code spans and links grew quadratically (1 MiB: 29 s, 4 MiB: over 400 s). The scan now stops at the
  end of the annotated span. The output does not change. (GH#2099)
• (pdf): two-column pages whose column baselines differ by up to 1pt are read in column order again. The XY-cut
  line grouping measured gaps between spans in y order, so a right-column line set slightly higher sorted first and
  merged with the left column's line into one line, which the column split then sent to one side. Spans are now
  ordered by x before gaps are measured, and a gap is measured from the line's rightmost edge so far. A column cut
  that would separate short label/value rows whose baselines differ slightly is refused, so key-value rows and table
  rows read row by row. (GH#2097)
• (pptx, ocr): plain output of a PPTX file has the OCR text of a picture on a slide. The text is in content and
  in the content of the slide one time. Before, OCR read the picture and the text was in no part of the plain output.
  The text is also there when images.inject_placeholders is false in Markdown, Djot and HTML output. (GH#2086)
• (ocr): text that has the form of a Markdown image reference does not repeat the OCR text of a picture. When a
  document had a paragraph such as ![label](label.png) as text, and a picture whose OCR text is shown without a
  placeholder, the text of the picture was in content two times. A PDF file with the default settings is such a
  document. The text is now there one time. (GH#2093)
• (ppt, ocr): the content of a slide in pages has the OCR text of a picture in plain output. OCR read the picture
  on a PPT slide and the text was in content of plain output, but not in the content of the slide. Page content that
  is built from the elements of a page now has the text one time, after the text of the picture, when content is the
  plain rendering: in plain output, and in the output of a custom format name that has no renderer. Page content is
  built that way when the extractor stores no pages: a PPT file is such a document, and a PDF file is one when page
  extraction is not set. The page content of Markdown, Djot, HTML, JSON and a custom renderer does not change.
  (GH#2088)
• (pdf, ocr): a failed OCR fallback for a whole PDF now reports its failed pages in ocr_page_failures. When the
  OCR run of a whole document failed and the native text was kept, the result had a warning and no record. Each page
  that has no native text now has a record, as it does when OCR runs on the scanned pages only. A page that has
  native text has no record. A page that failed in one OCR pipeline stage and got no text from a later stage had no
  record either, and now has one. (GH#2083)