Metric Blindness in Document Information Extraction

Vision-language models now outperform traditional OCR on the character- and word-level metrics by which document extraction systems are ranked. A practitioner selecting a system in 2026 encounters a consistent claim of field-level accuracy between 97% and 99% — rarely accompanied by a method, and where a method exists, it usually measures something other than what the practitioner needs to know.

This paper argues that the ranking is not merely imprecise but structurally uninformative for high-stakes extraction, because the metrics fail selectively: they register the loud failures of generative readers — truncated or collapsed transcription — while remaining close to blind to the most consequential one, fluent substitution of named entities and numeric values on perfectly legible input. A tax identification number, a subtotal, or a jurisdiction code replaced with a well-formed alternative is a defect the measuring instrument is constructed not to see. This is a position and synthesis paper; it reports no new experiments — every empirical value below is attributed to the work that measured it.

Three lineages, three failure modes

The immediate evidence comes from a benchmark of seventeen traditional OCR and vision-language systems on 175 microfilm scans of archival documents [4]. Vision-language models swept median word error rate and took the best reported figure on every statistic, though the margin is narrower than the framing suggests: the strongest vision model improves mean character error rate (CER) over the best traditional system by 1.36 points, and several vision models score worse. A qualitative analysis of the same outputs then found systematic failure modes invisible to those metrics: orthographic normalisation, spurious content generation, and semantic substitutions that preserve fluency while altering meaning.

Three details from that study carry the argument further than the headline. The leading system records a mean CER of 11.17% against a median of 1.72% — a sixfold gap indicating that an aggregate score summarises a heavily skewed distribution in which occasional catastrophic outputs are averaged into apparent competence. The semantic substitutions occur predominantly on clearly legible regions, attributed to the decoder deviating from source content under learned entity priors rather than to image degradation — which is what allows the finding to travel beyond damaged microfilm. And the substitutions persist when models are explicitly instructed to mark unreadable text as illegible: prompted abstention does not suppress them.

The architectural history of document extraction is usually narrated as a progression in accuracy. It is more usefully read as a progression in what kind of error each system is capable of committing.

The lineages differ less in accuracy than in the class of error each can commit — and the guarantee in the first row was traded away, not superseded. The discriminative layout-aware lineage — LayoutLM and its successors [28, 27, 7, 26, 29] — assigns labels to tokens that exist on the page. A token classifier cannot emit a value that is absent from its input; its cost is equally structural, since supervision is required per document type and performance degrades on unseen layouts. The generative OCR-free lineage [13, 24] removed the recognition stage entirely, eliminating OCR error propagation and simultaneously opening the substitution surface, because a decoder that produces the answer can produce an answer with no support in the image. The current LLM-centric lineage [15, 25] — approximately 300 papers published between 2021 and mid-2025 fall under this heading [12] — inherits the generative failure surface in full.

Scale is not the remedy. Reframing business document extraction as a tool-use problem and pairing an ordinary text language model with retrieval-augmented structured generation achieved state-of-the-art results on both key-information extraction and line-item recognition, matching or exceeding large multimodal models lacking the retrieval scaffold [2]. An independent line of work converges from the opposite direction: segmentation quality rather than model scale is the binding bottleneck — supplied with ground-truth segments, a 7B model recovers most of the gap to far larger ones [1].

That same study complicates the lineage picture in a way worth stating against the paper’s own interest. On its cross-format transfer test, an unaugmented frontier language model scored 95.39 where a discriminative layout model collapsed to 33.43, and segmentation added under two further points [1]. Weak layout reading in generative models is a weakness relative to their own potential, not relative to the discriminative lineage, which degrades far more sharply off-distribution. The trade in the table is between a structural guarantee and format robustness — not between accuracy and inaccuracy. Generalisation nonetheless remains the binding constraint regardless of lineage, and the magnitude is striking: across fifteen state-of-the-art large multimodal models on a unified key-information extraction benchmark of 6,133 documents, per-scenario field-level F1 varied by up to 45.6 points within a single model, and merely loosening the task from a predefined schema to open-category extraction cost one leading model 20.2 points and another 26.7 [11]. This is consistent with the earlier finding that OCR capability within multimodal models is weaker and less uniform than headline benchmark scores suggest [14]. A single averaged benchmark figure conceals a spread of this size by construction.

Metric blindness

Consider the information a metric can carry. CER and WER are edit distances over character and token sequences; their magnitude is proportional to the number of symbols altered, not to the semantic weight of the symbols. Substituting one digit in a nine-digit identifier costs approximately 1/9 of that field’s contribution to the score, whatever the consequence of the substitution. The metric is behaving exactly as specified. The specification is the problem.

Two observations follow. First, entity substitution — singled out as the most consequential error class for generative readers, since it introduces large semantic distortion while contributing only marginally to aggregate metrics [4] — sits in the region where character-level metrics are weakest, so the metrics that most favour generative systems are the ones least able to detect their characteristic defect. Second, cross-field arithmetic inconsistency is registered by no metric in general use, despite being trivially checkable in any document that contains a total.

Two qualifications keep this precise. Aggregate metrics do not fail uniformly — they fail selectively. Generative readers exhibit both a loud failure mode that CER and WER register well and a quiet one they do not, and it is the selectivity, not a blanket insensitivity, that makes the aggregate misleading. And prevalence is not established: the qualitative taxonomy the argument relies on reports no failure-mode frequencies, so the claim available from the literature concerns the consequence of entity substitution rather than its rate.

The consequence is that improvements in reported accuracy and increases in deployed risk are not merely compatible; under a shift from discriminative to generative architectures the paper argues they are the expected joint outcome. This is a synthesis: no surveyed source measures a correlation between aggregate score and substitution rate, because none counts substitutions.

That aggregate character metrics understate downstream harm is itself established rather than novel: OCR errors affecting named entities degrade recognition and linking disproportionately to their character cost [6], the effect persists in retrieval [9], the closest prior position argues the same for business-document key-field and line-item extraction [18], and a concurrent large-scale study revisits whether OCR belongs in the pipeline at all under multimodal models [21]. Nor is this an outsider’s objection: the authors who set the state of the art on line-item recognition simultaneously proposed a replacement metric class, on the grounds that the existing measures were insufficiently aligned with practical extraction use cases [2]. When the group producing the leading numbers publishes a critique of the scoreboard in the same paper, the scoreboard should be treated as provisional.

The structured-output illusion

The remedy most readily available to practitioners is constrained decoding against a schema, and it is widely assumed to bound the failure. It fails to, for two independent reasons.

The first is that the guarantee is narrower than advertised even on its own terms. A rigorous evaluation of six constrained-decoding frameworks over 10,000 real-world JSON schemas measured empirical coverage — the fraction of schemas a framework declares it supports whose outputs actually validate — and found it far below the declared figure, falling to 0.03 for one framework on hard real-world schemas, with the best framework supporting roughly twice as many schemas as the worst [5]. Schema compliance is not a guarantee a practitioner receives automatically; it is a property that varies by an order of magnitude across implementations of the same nominal feature.

The second reason matters more, and the paper states it as its own analytical claim because no benchmark result establishes it: schema compliance is a statement about the shape of an output, whereas the failure under discussion is a statement about its content. The two are orthogonal. An object in which every field is present, correctly typed, and factually wrong satisfies every structural validator a practitioner would naturally write, and will do so silently. The constrained-decoding literature does not claim otherwise — indeed the same evaluation found that constraining generation slightly improved downstream task accuracy on reasoning benchmarks — but neither does it measure value correctness on documents, and the assumption that structural validation transfers to factual validation is supplied by the practitioner rather than by the evidence.

Arithmetic as discarded verification

Numerical reasoning over financial documents constitutes a distinct and more severe failure. It was established as its own task by expert-annotated benchmarks over earnings reports and hybrid tabular–textual content [3, 30], remains sufficiently difficult that specialised models continue to be developed rather than relying on general language-model capability [31], and direct evaluation confirms the deficit persists in current systems [23].

The design consequence is that derived quantities should be recomputed deterministically from extracted operands rather than generated. The stronger observation is that documents of this class carry an arithmetic dependency graph — subtotals summing to totals, line items reconciling against balances, prior-period values constraining current ones — which functions as an intrinsic checksum. Delegating computation to the extractor does not merely risk error; it discards a verification signal the document supplies at no cost. The absence of this check in Figure 1 is the most readily correctable gap identified in the paper.

The benchmark hole

The public evaluation surface covers receipts [8], noisy forms [10], document-image question answering [17], invoices with line items [22], and financial-report reasoning [3, 30]. Earlier work established that field extraction generalises across templates when a model learns field representations rather than absolute positions [16].

To the paper’s knowledge, no peer-reviewed benchmark evaluates extraction from tax returns or comparable regulatory filings. One distinction must be drawn carefully, since the paper cites the counterexample itself: the unified benchmark discussed above does include a tax-compliant document scenario [11], but those are transactional instruments averaging eight fields per document — tax invoices in the value-added sense — rather than filings. A personal or corporate return carries hundreds of interdependent fields, and nothing in the indexed literature addresses jurisdiction-specific form variation, multi-year carryforward consistency, or the arithmetic dependency graph described above. The gap is in filings, not in tax-adjacent documents generally.

Two consequences follow. Accuracy claims in this domain are not currently falsifiable, since no shared evaluation exists against which to falsify them. And the column corresponding to cross-field consistency is empty for every document class, not only for tax — the verification signal identified in the preceding section is unmeasured throughout the field.

Coverage in place of accuracy

The literature offers one mechanism that converts this problem into a managed quantity. Applying split conformal prediction post hoc to fine-tuned multimodal extractors on a receipt corpus yields entity-level prediction sets satisfying a user-specified error rate: reported marginal coverage was 98.3% at α = 0.02, with 70% of predictions returned as high-confidence singletons [20]. Reliability varied by field in an interpretable way — highly structured fields such as dates and prices produced small sets with near-perfect reliability, while rare or semantically ambiguous fields produced large sets and lower coverage — and the calibrated set sizes were used directly to automate confident extractions and route uncertain ones for human review.

The paper argues this should become the reporting convention rather than an optional addition. A coverage statement — this system processes x% of fields automatically at a guaranteed error rate α — is falsifiable, auditable, and carries the per-field structure that a single accuracy figure destroys. A point accuracy figure, by contrast, is a summary over an unstated distribution measured by an instrument shown above to be insensitive to the relevant error class. Comparable architectures are being assembled in adjacent regulated workflows [19], suggesting the convention is not specific to document extraction.

A benchmark proposal

Since the gap in Figure 2 is tractable, the paper specifies what the missing benchmark must contain. It requires: (i) tax or equivalent regulatory filings across at least two jurisdictions and two filing years, to expose the distribution shift that periodic form redesign induces; (ii) field-level ground truth with entity-type annotation, so that substitution in identifiers and amounts is scored separately from character-level noise; (iii) declared arithmetic dependencies between fields, making cross-field consistency directly scorable; (iv) a held-out calibration split, so that coverage rather than accuracy can be reported; and (v) a hallucination probe — documents containing fields that are absent, illegible, or ambiguous, where the correct output is abstention.

Criterion (v) is the one no surveyed benchmark scores, and it cannot be satisfied at the prompt. The nearest existing practice measures faithfulness post hoc, checking whether predicted values can be located in the document and classifying those that cannot as hallucinated [11]; that is an analysis correlated against F1, not a task in which abstention is the correct output and is rewarded as such. The distinction matters because semantic substitutions persist in models explicitly instructed to mark unreadable regions as illegible [4], so abstention has to be trained and scored rather than requested. A system that never abstains cannot be distinguished, under current scoring, from one that abstains correctly.

Limitations

This is a synthesis and position paper and reports no new experiments; every empirical value cited is attributed to the work that measured it. The four sources on which the argument most depends were read in full [4, 11, 5, 1]; the remainder of the survey was assembled from bibliographic metadata and abstracts retrieved through Crossref and OpenAlex, which is adequate for establishing the shape of a literature and the presence of a gap but insufficient to critique those methodologies individually.

The load-bearing evidence is also narrower than the argument. The study establishing that semantic substitution escapes character metrics evaluates 175 pages of bi-level microfilm of Spanish-language typewritten documents, reports no error bars, includes no frontier commercial systems, and measures full-page transcription rather than field extraction; it also reports no failure-mode frequencies, so no prevalence claim is available from it. Its finding that substitutions occur on legible regions under decoder priors, rather than under image degradation, is what licenses the extrapolation to clean business documents — and that extrapolation remains an argument rather than a measurement. No cited source tests numeric or arithmetic fields at all, so the benchmark-hole argument rests on the structure of such documents rather than on evidence about them. Figure 1 is an analytical framework derived from metric definitions, not a measurement, and reasonable disagreement about individual cells is possible. The claim that no tax-extraction benchmark exists is a negative result over indexed literature and cannot exclude proprietary or unindexed evaluations. Finally, the conformal prediction result the paper builds upon was demonstrated on receipts; whether its guarantees survive the distribution shift of a new filing year is precisely the open question the proposed benchmark is designed to answer.

Conclusion

The field ranks document extraction systems using instruments that are least sensitive exactly where generative architectures are most likely to fail, validates their outputs with structural checks that certify shape rather than content, and leaves unmeasured an arithmetic consistency signal that many target documents supply for free. The correction is available: report calibrated coverage rather than point accuracy, recompute derived values deterministically, score entity substitution as its own class, and require abstention to be evaluable. What is missing is a shared benchmark on which such reporting could be compared — and in the document class where the stakes are highest, that benchmark does not exist.

References

1. Aniket Bhattacharyya, Anurag Tripathi, Ujjal Das, et al. Information extraction from visually rich documents using LLM-based organization of documents into independent textual segments. arXiv preprint, 2025.
2. Franz Louis Cesista, Rui L. Aguiar, Jason Z. Kim, et al. Retrieval augmented structured generation: Business document information extraction as tool use. IEEE MIPR, 2024.
3. Zhiyu Chen, Wenhu Chen, Charese Smiley, et al. FinQA: A dataset of numerical reasoning over financial data. EMNLP, 2021.
4. Marina Gardella, Camilo Mariño, Diego Belzarena, et al. When low CER is not enough: An analysis of hallucinations in vision-language OCR systems on historical uruguayan documents. arXiv preprint, 2026.
5. Saibo Geng, Hudson Cooper, Michał Moskal, et al. JSONSchemaBench: A rigorous benchmark of structured outputs for language models. arXiv preprint, 2025.
6. Ahmed Hamdi, Elvys Linhares Pontes, Nicolas Sidère, Mickaël Coustaty, and Antoine Doucet. In-depth analysis of the impact of OCR errors on named entity recognition and linking. Natural Language Engineering, 29(2):425–448, 2023.
7. Teakgyu Hong, Donghyun Kim, Mingi Ji, et al. BROS: A pre-trained language model focusing on text and layout for better key information extraction from documents. AAAI, 2022.
8. Zheng Huang, Kai Chen, Jianhua He, et al. ICDAR2019 competition on scanned receipt OCR and information extraction. ICDAR, 2019.
9. Alexandre Jaud, Ahmed Hamdi, Antoine Doucet, Adam Jatowt, and Mickaël Coustaty. Beyond CER and WER: How does OCR really impact information retrieval? ACM/IEEE JCDL, 2025.
10. Guillaume Jaume, Hazım Kemal Ekenel, and Jean-Philippe Thiran. FUNSD: A dataset for form understanding in noisy scanned documents. ICDAR Workshops, 2019.
11. Yifan Ji, Zhipeng Xu, Zhenghao Liu, et al. UNIKIE-BENCH: Benchmarking large multimodal models for key information extraction in visual documents. arXiv preprint, 2026.
12. Wenjun Ke, Yifan Zheng, Youlan Li, et al. Large language models in document intelligence: A comprehensive survey, recent advances, challenges, and future trends. ACM Transactions on Information Systems, 2025.
13. Geewook Kim, Teakgyu Hong, Moonbin Yim, et al. OCR-free document understanding transformer. ECCV, 2022.
14. Yuliang Liu, Zhang Li, Mingxin Huang, et al. OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences, 2024.
15. Chuwei Luo, Yufan Shen, Zhaoqing Zhu, et al. LayoutLLM: Layout instruction tuning with large language models for document understanding. CVPR, 2024.
16. Bodhisattwa Prasad Majumder, Navneet Potti, Sandeep Tata, et al. Representation learning for information extraction from form-like documents. ACL, 2020.
17. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. WACV, 2021.
18. Ngoc Nhi Nguyen, Ahmed Hamdi, Antoine Doucet, Adam Jatowt, and Mickaël Coustaty. Rethinking OCR evaluation for information extraction in business documents. Lecture Notes in Computer Science, 2026.
19. Robert Richardson, Josh Meyers, Brian Hartman, et al. Agentic AI and retrieval-augmented models in straight-through underwriting. arXiv preprint, 2026.
20. Alexander Rombach and Nijat Mehdiyev. Beyond accuracy: Understanding model confidence in key information extraction with conformal prediction. IJDAR, 2026.
21. Jiyuan Shen, Peiyue Yuan, Atin Ghosh, et al. OCR or not? rethinking document information extraction in the MLLMs era with real-world large-scale datasets. EACL Industry Track, 2026.
22. Štěpán Šimsa, Milan Šulc, Michal Uřičář, et al. DocILE benchmark for document information localization and extraction. ICDAR, 2023.
23. Pragya Srivastava, Manuj Malik, Vivek Gupta, et al. Evaluating LLMs’ mathematical reasoning in financial document question answering. Findings of ACL, 2024.
24. Zineng Tang, Ziyi Yang, Guoxin Wang, et al. Unifying vision, text, and layout for universal document processing. CVPR, 2023.
25. Dongsheng Wang, Natraj Raman, Mathieu Sibue, et al. DocLLM: A layout-aware generative language model for multimodal document understanding. ACL, 2024.
26. Jiapeng Wang, Lianwen Jin, and Kai Ding. LiLT: A simple yet effective language-independent layout transformer for structured document understanding. ACL, 2022.
27. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, et al. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. ACL, 2021.
28. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. LayoutLM: Pre-training of text and layout for document image understanding. KDD, 2020.
29. Yiheng Xu, Tengchao Lv, Lei Cui, et al. XFUND: A benchmark dataset for multilingual visually rich form understanding. Findings of ACL, 2022.
30. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, et al. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. ACL, 2021.
31. Fengbin Zhu, Ziyang Liu, Fuli Feng, et al. TAT-LLM: A specialized language model for discrete reasoning over financial tabular and textual data. ICAIF, 2024.