Added
- (pdf/ocr):
OcrQualityThresholds::enable_plausibility_ocr_routingandOcrQualityThresholds::min_reliable_language_chunk_ratiocontrol language/dictionary-plausibility OCR routing, andPdfMetadata.implausible_text_pagesreports which pages a page's decoded text failed to read as any real language. (GH#1696) - (config):
ExtractionConfig::runs_ocr_on_embedded_imagesandExtractionConfig::wants_own_bytes_in_resultare exposed on every binding, beside the existingneeds_image_data. The first is the predicate the pipeline uses to decide whether a container's embedded images are OCR'd; the second is the pre-GH#1662 formula (extract_images, captioning or QR codes) that decides whether a standalone image's own bytes are echoed intoimages. (GH#1662)
Changed
- (ocr): the
ocr,ocr-wasm, andocr-pipelineCargo features now implylanguage-detection. The language/dictionary-plausibility OCR-routing signal (GH#1696) is gated the same as the rest of the OCR module and would otherwise be a silent hole under a build that enables onlyocr-pipeline(the VLM-only, Tesseract-free pipeline). whatlang (behindlanguage-detection) depends only onhashbrown, so this is wasm/Android-safe. (GH#1696)
Fixed
- (ocr): OCR no longer returns every embedded image's raw bytes. Image bytes read only to feed embedded-image OCR (GH#1662) are dropped once OCR has consumed them and any OCR text it produced has been rendered into
content;imagesnow carries data only when the caller asked for image extraction, captioning, QR-code detection,pdf_options.ocr_inline_images, orimages.include_page_rasters;counts.imagesstill reports how many embedded images the document had. (GH#1703) - (pdf): a
/ToUnicodemapping that resolves to the wrong-but-real letters is now routed to OCR too. GH#1667 fixed the case where a page has no usable mapping tier at all (MappingProvenance::Fallback); a page whose/ToUnicodeCMap DOES resolve, but consistently to the wrong letter (e.g. a ROT-shifted mapping), passed every character-shape check unchanged and OCR never fired, because the text is structurally indistinguishable from real prose. A language/dictionary-plausibility signal now flags such a page under the defaultAutostrategy, and the opt-inScannedPagesstrategy picks it up through the existingscanned_pagesunion. A page with no explicitocrconfig keeps its native text, but a warning now names the defect. Closes the remaining gap in GH#1667. (GH#1696) - (pdf): the mixed native-and-OCR path no longer drops every native table on a page OCR never touched. On a long document where only some pages needed OCR, the mixed path replaced the whole document's table list with just the OCR pages' tables whenever OCR found even one, silently dropping every native table on every other page. Only the tables of pages OCR itself produced a table for are now replaced; a page the mixed path never sent to OCR keeps its native tables. (GH#1670)
- (pdf):
include_document_structurealone now triggers the structured native extraction pass. Previously the structured pass ran only for a Markdown/Djot/HTML/DocTags output format, an explicit hierarchy config, inline-image OCR, or a content filter -- never forinclude_document_structureitself. A caller who set only that flag, with output left at itsPlaindefault, silently got a flat, paragraph-only document, so the structure tree it asked for came back holding nothing butparagraphnodes even though the extractor found headings, tables and page breaks. This configuration now renders its plain-text output from the structured document, so the renderedcontentchanges with it: table rows render one per line, and headings and page footers reflow. A caller that sets only this flag and depends on the previousPlaintext sees that text change. The table count is unaffected. (GH#1668) - (docx): a paragraph inherits the list numbering that its style carries. A list whose paragraphs hold no direct numbering and take it from the paragraph style instead, which is the shape Word writes for its built-in list styles, came out as plain paragraphs with no list markers. The parser now resolves numbering through the style's
basedOnchain. Direct paragraph numbering still takes precedence over the style. (GH#1663) - (pdf): the fabricated-glyph-mapping provenance lookup is repaired, and a page it flags is sent to OCR. The extractor read each text run's mapping provenance after it had renamed the run's font to the resolved
/BaseFont, but the provenance table is keyed by the raw resource alias, so the lookup missed for every font whose base name differs from its alias and the signal was empty almost everywhere. The lookup now runs in the same pass and uses the alias, and a page whose text carries no usable mapping tier at all (MappingProvenance::Fallback) now forces the OCR fallback under the defaultAutostrategy. A font whose mapping resolves, even to the wrong characters, is a different failure mode this signal does not cover. (GH#1667) - (ocr): a low OCR confidence now lowers the reported quality score. The quality score measured only how clean the text looked, so a page that OCR itself had little confidence in still scored as clean whenever the characters happened to be well formed. The score is now capped by the word-count-weighted mean of per-page OCR confidence, once at least 20 words were recognized -- the same fold
ExtractionConfidence::ocr_aggregateuses, shared rather than recomputed, though the aggregate itself reports on any recognized word and so can differ below the 20-word floor. A native, non-OCR extraction is unaffected. (GH#1669, GH#1694) - (ocr): a cold start no longer builds one candle OCR engine per waiting caller. Each candle backend's engine pool looked for a cached engine, released the lock, built the engine, then stored it, so every request that arrived during the first build loaded its own copy of the model and a burst could exhaust the GPU memory. All four backends and the GLM-OCR layout pool now use the crate's engine cache, one per backend, which holds the write lock across the build: the first caller builds and the rest wait for that engine. Loads of different engines on the same backend (two TrOCR variants, say) therefore also serialise during a cold start. A failed load stores nothing, so the next caller retries. (GH#1683)
- (ocr): the aggregate OCR confidence is populated for page-level results, and both routes weight it the same way. A document whose OCR confidence arrived one score per page left the aggregate empty, so a caller that read it got nothing back. Both routes now fold the scores by recognized word count instead of by element, so a page of many short lines no longer counts for as much as a page of few long ones. This changes the number the embedded-image route already reported. (GH#1677)
- (ocr): the Tesseract language probe run at backend registration now finds the installed languages. The probe initialised Tesseract with an empty datapath, which makes the library look one
tessdatadirectory deeper than the resolver real OCR jobs use, so on a layout where jobs foundeng.traineddatadirectly the probe logged "couldn't load any languages" and silently fell back to a hardcoded language list. It now resolves the tessdata directory the same way a job does. (GH#1671) - (ocr): a page whose embedded-image OCR retry came back empty says so. When page-raster OCR failed and the embedded-image retry ran but returned nothing -- a candle VLM backend reports success with empty content -- the page warning named only the first failure, with no trace that a retry ran at all. The per-page warning and the wholesale "every page failed" error now report the retry outcome alongside the first failure. (GH#1673)
- (ocr): OCR throughput scales with the thread budget. The mixed native-and-OCR route rasterised each batch's pages one at a time regardless of the configured thread budget, so a 240-page document with 80 scanned pages took the same 9 s at 4, 8, 16 and 32 threads. Pages now render in parallel across the batch (sequentially on wasm32, which has no thread pool), in the same order as before; measured 8.8 s to 3.3 s at 4 threads and 9.0 s to 1.0 s at 32. (GH#1666)
- (ocr): a DOCX, PPT, PPTX or HTML embedded image is read before OCR runs on it. The predicate that decides whether a container reads an image out of its archive did not count embedded-image OCR, so with OCR configured and
run_ocr_on_imagesleft on, the image was attached with an empty buffer and OCR reported "Could not determine image format" for zero bytes. A standalone image extraction'simagesoutput is unchanged. (GH#1662) - (ocr): the mixed native-and-OCR route sizes its render batch against
security_limits.max_content_size. The batch was sized from the resolved thread budget alone, so a wide thread budget requested a batch whose estimated PNG-encode peak crossed the fixed content limit, and every page in the rejected batch was skipped rather than the extraction failing. The batch is now capped up front from one representative page's estimated encode cost at the effective render DPI; the real peak is still checked, and still rejected if genuinely too large, afterwards. (GH#1665) - (ocr): the candle OCR backends no longer reject every rendered PDF page. The PDF OCR route stamps
source_dpiandpage_rotation_degreesinto the sharedbackend_optionsfor every backend, and the candle backends (candle-paddleocr-vl,candle-glm-ocr,candle-deepseek-ocr) deserialise their options strictly, so every rendered page failed validation and came back empty. Exactly those two pipeline hint keys are stripped before the candle options are read; any other unknown key still fails validation. (GH#1672) - (ruby): a tagged enum's payload readers return the stored value. The generated
Datareaders calledsuperon a class that has no such method, so reading a payload field on a tagged-enum value raisedNoMethodError. Regenerated on alef 0.93.1, whose Magnus emitter reads the stored member. (alef 0.93.0) - (ocr): a candle VLM backend no longer emits implausible lines of noise.
candle-paddleocr-vland the other three candle backends would occasionally emit a CJK-script run in an otherwise English document, or bare LaTeX markup on a plain-text OCR task, most visibly on a non-text region such as a handwritten signature scribble. A line whose classified letters are mostly in a script the configured OCR language list does not cover, or whose content is mostly\command-shaped LaTeX tokens with no bare word on a plain-text task, is now dropped, with a warning naming how many lines were removed; a$$/``` fenced block and a genuine formula/table/chart task are exempt. (GH#1676) - (ocr):
candle-deepseek-ocrreads the whole page. Generation was capped at 128 new tokens (~400-650 characters, the 15-25% observed), the page was fed as a single 1024 px global view, and weights were loaded as F32 (16.5 GB on a 24 GB card). The cap is now a model configuration field defaulting to 4096 in line with the other full-page backend, pages larger than 640 px are also tiled into 640 px local crops as in the reference "Gundam" mode, the dtype follows the device (BF16 on CUDA, F16 on Metal, F32 on CPU), the weights auto-download from a checksum-pinneddeepseek-ai/DeepSeek-OCRrevision somodel_pathis optional, and a run that starts looping (the repeated-digit table on a signature block) is detected once the repeat fills a trailing detection window and truncated back to a single copy of the repeated unit, rather than burning the rest of the token budget on it. (GH#1674) - (ocr):
candle-glm-ocrno longer garbles hex identifiers such as UUIDs. The repetition penalty defaulted to 1.1 with no upstream basis (zai-org/GLM-OCR'sgeneration_config.jsonsets none) and was applied once per occurrence of a token in the whole decode history, so a token repeated k times was suppressed by1.1^k-- exponential enough that hex digits and-inside a long identifier lost probability mass until argmax drifted onto a re-emitted group. The default is now 1.0 (a no-op, matching upstream), and each token in the trailing 64-token window is now penalised exactly once regardless of how many times it recurs. (GH#1675) - (ocr):
candle-deepseek-ocrno longer runs its prefill without a causal mask.prepare_causal_attention_maskbuilt a 0/1 keep-mask that the attention step added straight onto the logits, so a future token was nudged down by one instead of blocked and the prompt-and-image prefill (and the KV cache it seeds for every later token) saw the whole sequence; the mask is now the additive 0/-inf form GLM-OCR and PaddleOCR-VL use. This is the one DeepSeek-only forward-pass defect the GH#1701 audit found; with it fixed a full Letter page decodes without repetition on Metal at both F16 and F32, but the reported CUDA F32 loop has not been re-run. (GH#1701) - (ocr):
candle-deepseek-ocrno longer fails on the first 640 px local crop. Resizing the SAM relative-position table to the crop size multiplied each row by a one-element tensor, which candle does not broadcast, so every page large enough to be tiled failed withshape mismatch in mulbefore a token was decoded; the resize now scales withaffineand samples at half-pixel positions like the referenceF.interpolate(mode="linear"). Verified on Metal at F16: a 2,018-character Letter page comes back whole. (GH#1674)