The DOI matches its DataCite record; 54 of the 56 references it cites were found in a registry or an index, and 2 are not checked yet; none of the 35 sources its claims cite is checked for support yet.
Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents
Open accessAbstract
We explore how multimodal Large Language Models (mLLMs) can help researchers transcribe historical documents, extract relevant historical information, and construct datasets from historical sources. Specifically, we investigate the capabilities of mLLMs in performing (1) Optical Character Recognition (OCR), (2) OCR Post-Correction, and (3) Named Entity Recognition (NER) tasks on a set of city directories published in German between 1754 and 1870. First, we benchmark the off-the-shelf transcription accuracy of both mLLMs and conventional OCR models. We find that the best-performing mLLM model significantly outperforms conventional state-of-the-art OCR models and other frontier mLLMs. Second, we are the first to introduce multimodal post-correction of OCR output using mLLMs. We find that this novel approach leads to a drastic improvement in transcription accuracy and consistently produces highly accurate transcriptions (<1% CER), without any image pre-processing or model fine-tuning. Third, we demonstrate that mLLMs can efficiently recognize entities in transcriptions of historical documents and parse them into structured dataset formats. Our findings provide early evidence for the long-term potential of mLLMs to introduce a paradigm shift in the approaches to historical data collection and document transcription.
Checked 2026-11-13 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
This paper benchmarks multimodal LLMs (GPT-4o, Gemini 2.0 Flash) against conventional OCR engines for transcribing German city directories printed 1754–1870, and introduces image-aware multimodal OCR post-correction. Gemini 2.0 Flash produced the most accurate off-the-shelf transcriptions and, when used to post-correct a Transkribus model's output, achieved the lowest error rates of any tested approach.
Key findings
- Off-the-shelf OCR: Gemini 2.0 Flash achieved a normalized CER of 1.27% across the corpus, versus 3.67% for Transkribus Print M1, 4.16% for Text Titan I, and 6.31% for GPT-4o. Tesseract's "deu_frak" model was far worse (30.22% normalized CER, 199.91% normalized WER) and was omitted from the results table.
- Multimodal OCR post-correction (feeding the noisy Print M1 transcription plus the source PNG back into an mLLM) yielded the best results overall: 0.84% normalized CER with Gemini 2.0 Flash (as low as 0.08% for Trier-1853) and 1.00% with GPT-4o, with no image pre-processing or model fine-tuning.
- NER: with ground-truth text as input, Gemini 2.0 Flash reached 89.32% fuzzy match across the full sample (up to 99.49% for Aachen-1838); with noisy OCR text, 88.61%; extracting entities directly from the PNG, 80.86%. GPT-4o scored lower throughout (91.00%, 87.58%, and 58.38% fuzzy respectively).
- mLLMs struggled with directories organized by sub-headings (e.g., Leipzig-1753 at 73.75% fuzzy from GT text), and font type (Antiqua vs. Fraktur) appeared to have no significant impact on parsing accuracy.
Methods
- Corpus of 30 page-images drawn from 10 German-language city directories published 1754–1870 in Aachen, Dresden, Leipzig, Frankfurt, Lübeck, Riga, and Trier, predominantly Blackletter/Fraktur with one Antiqua directory; images sourced from archive websites and deliberately left un-preprocessed.
- Three experiment sets: (1) OCR benchmarking of Gemini 2.0 Flash and GPT-4o against Tesseract 5.5.0 "deu_frak" and Transkribus Print M1 and Text Titan I; (2) multimodal OCR post-correction using the best non-mLLM transcription plus the source image; (3) NER/entity parsing into JSON then CSV from ground-truth text, noisy OCR text, and the raw PNG.
- Accuracy measured via Levenshtein distance, CER and WER (reported both normalized and non-normalized); NER evaluated with strict string matching and fuzzy matching using Jaro-Winkler similarity with a 0.9 threshold. Prompts were developed iteratively with mLLM assistance; temperature set to 0.0.
Limitations
- Stated by the paper: the corpus is restricted to historical Latin-alphabet prints, so performance on non-Latin-alphabet sources is unknown; the authors could not provide an in-depth comparison of mLLMs against conventional fine-tuned NER approaches; a comprehensive analysis of mLLM-assisted prompt design is beyond the paper's scope; and the authors call for a dedicated historical OCR benchmark for mLLMs.
- Evident from the text: the corpus is small (30 pages from 10 directories), and NER evaluation excluded directories whose generated CSV did not match the ground-truth row count after one attempt — Gemini and GPT-4o GT/OCR text results are based on only eight directories, and GPT-4o's PNG-to-CSV results on only three. The authors also note they cannot disentangle whether PNG-to-CSV performance differences stem from OCR versus entity-recognition ability, given the models' black-box nature.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 20 Not checked · 68 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the best-performing mLLM model significantly outperforms conventional state-of-the-art OCR models and other frontier mLLMs.”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, while the structure of entries is usually internally consistent, many entries deviate from the structure and entries are mixed between those of individuals, businesses, and office holders.”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > 3 Source and Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, our corpus features many of the most common challenges faced by researchers working with historical prints.These include ink degradation, discolored backgrounds, human annotations, skewed texts, 3D distortions, multiple historical fonts on the same page, and excess visual information.”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > 3 Source and Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we demonstrate that mLLMs can efficiently recognize entities in transcriptions of historical documents and parse them into structured dataset formats.”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > Abstract
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“consistently produces highly accurate transcriptions (<1% CER), without any image pre-processing or model fine-tuning.”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
50 more claims
Create a free account to see all 90 claims and every source verdict.
- Probably not a substantial claim. 15
- It may state two separate claims as one. 24
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“| Full Sample | 3124 | 77.48% | 89.32% | 74.92% | 88.61% | 68.76% | 80.86% | 70.35% | 91.00% | 65.67% | 87.58% | 34.29% | 58.38% |”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > 7 Conclusion
Reported 77.48 · recomputed 80.055
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > 3 Source and Dataset
“The basis of our analysis is a corpus of thirty page-images, drawn from ten different city directories, published in German between 1754-1870”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > 7 Conclusion
“| Leipzig-1753 | 1.91% | 12.25% | 4.54% | 29.08% | 1.99% | 12.75% | 1.94% | 12.42% | 0.92% | 5.88% | 0.82% | 5.23% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > Abstract
“We find that this novel approach leads to a drastic improvement in transcription accuracy and consistently produces highly accurate transcriptions (<1% CER), without any image pre-processing or model fine-tuning.”
§ MULTIMODAL LLMS FOR OCR, OCR POST-CORRECTION, AND NAMED ENTITY RECOGNITION IN HISTORICAL DOCUMENTS > 7 Conclusion
“| Frankfurt-1778 | 2.86% | 18.39% | 5.22% | 33.56% | 17.92% | 115.17% | 14.16% | 91.03% | 1.82% | 11.72% | 2.43% | 15.63% |”
Citations
Cites
An Overview of the Tesseract OCR Engine
Smith, R. W. (2007) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2007.4376991 · Cited by 2,457
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · cited 1×
“The issue of prompting is particularly important given that, for any given task, the performance of an mLLM can be significantly influenced by even minor variations in how a prompt is written [76].”
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610
Nguyen (2022) · doi:10.1145/3453476 · Cited by 286
Fu, L., Kuang, Z., Song, J., Huang, M., Yang, B., Li, Y., Zhu, L., Luo, Q., Wang, X., Lu, H. F., et al. (2025) · neural information processing systems · doi:10.52202/085713-3253 · Cited by 206
LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis
Shen, Z., Zhang, R., Dell, M., Lee, B. C. G., Carlson, J., & Li, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86549-8_9 · Cited by 186
OpenAI, :, Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., et al. (2024) · arXiv · Cited by 180
OCR binarization and image pre-processing for searching historical documents
Gupta, M. R., Jacobson, N. P., & Garcia, E. K. (2006) · Pattern Recognition · doi:10.1016/j.patcog.2006.04.043 · Cited by 157
Assessing the Impact of OCR Quality on Downstream NLP Tasks
van Strien, D., Beelen, K., Ardanuy, M. C., Hosseini, K., McGillivray, B. C., & Colavizza, G. (2020) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0009169004840496 · Cited by 150
Research works printed but not linked: 30
- Eltjo Buringh and Jan Luiten Van Zanden. Charting the "rise of the west": Manuscripts and printed books in europe, a long-term perspective from the sixth through eighteenth centuries. The Journal of Economic History, 69 (2):409-445, 2009.
- Lynne Tatlock. Introduction: The book trade and "reading nation" in the long nineteenth century. In Lynne Tatlock, editor, Publishing Culture and the "Reading Nation": German Book History in the Long Nineteenth Century, Studies in German Literature, Linguistics, and Culture, pages 1-24. Camden House, Rochester, NY, 2010.
- Alaa Sulaiman, Khairuddin Omar, and Mohammad F Nasrudin. Degraded historical document binarization: A review on issues, challenges, techniques, and future directions. Journal of imaging, 5(4):48, 2019.
- Phillip Benjamin Ströbel. Flexible Techniques for Automatic Text Recognition of Historical Documents. PhD thesis, University of Zurich, Zurich, Switzerland, 2023.
- Uwe Springmann and Anke Lüdeling. Ocr of historical printings with an application to building diachronic corpora: A case study using the ridges herbal corpus. Digital Humanities Quarterly, 11(2), 2017.
- Shaobin Xu and David Smith. Retrieving and combining repeated passages to improve ocr. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1-4, 2017.
- Huaigu Cao, Stephen Rawls, and Prem Natarajan. 1990 us census form recognition using ctc network, wfst language model, and surname correction. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 977-982, 2017.
- Quan Duong, Mika Hämäläinen, and Simon Hengchen. An unsupervised method for ocr post-correction and spelling normalisation for finnish. In Simon Dobnik and Lilja Øvrelid, editors, Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 240-248, Reykjavik, Iceland (Online), 2021. Linköping University Electronic Press, Sweden.
- Oksana Dereza, Deirdre Ní Chonghaile, and Nicholas Wolf. "To Have the 'Million' Readers Yet": Building a Digitally Enhanced Edition of the Bilingual Irish-English Newspaper an Gaodhal (1881-1898). In Rachele Sprugnoli and Marco Passarotti, editors, Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 65-78, Torino, Italia, 2024. ELRA and ICCL.
- Mahdi Hajiali. OCR Post-processing Using Large Language Models. Phd thesis, University of Nevada, Las Vegas, August 2023. URL https://digitalscholarship.unlv.edu/thesesdissertations/4811.
- Nianheng Wu. Multimodal ocr post-correction on german historical documents. Master's thesis, University of Stuttgart, 2023.
- Oksana Dereza, Deirdre Ní Chonghaile, and Nicholas Wolf. "To Have the 'Million' Readers Yet": Building a Digitally Enhanced Edition of the Bilingual Irish-English Newspaper an Gaodhal (1881-1898). In Rachele Sprugnoli and Marco Passarotti, editors, Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 65-78, Torino, Italia, 2024. ELRA and ICCL.
- Andre Wolters and Andreas Van Cranenburgh. Historical dutch spelling normalization with pretrained language models. Computational Linguistics in the Netherlands Journal, 13:147-171, March 21 2024.
- Viktoria Löfgren and Dana Dannélls. Post-OCR correction of digitized Swedish newspapers with ByT5. In Yuri Bizzoni, Stefania Degaetano-Ortlieb, Anna Kazantseva, and Stan Szpakowicz, editors, Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 237-242, St. Julians, Malta, March 2024. Association for Computational Linguistics.
- Jonathan Bourne. CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models. arXiv, 2408.17428, 2025. Preprint. · arXiv
- Yung-Hsin Chen and Phillip B. Ströbel. Trocr meets language models: An end-to-end post-correction approach. In Harold Mouchère and Anna Zhu, editors, Document Analysis and Recognition -ICDAR 2024 Workshops, volume 14935 of Lecture Notes in Computer Science, pages 12-26. Springer Nature Switzerland, Cham, 2024.
- Maud Ehrmann et al. Named entity recognition and classification on historical documents: A survey. ACM Computing Surveys, 56(2):1-47, February 2024.
- Denis Coquenet, Clément Chatelain, and Thierry Paquet. Dan: a segmentation-free document attention network for handwritten document recognition. IEEE transactions on pattern analysis and machine intelligence, 45(7): 8227-8243, 2023.
- Dayvid Castro, Byron Leite Dantas Bezerra, and Cleber Zanchettin. An end-to-end approach for handwriting recognition: From handwritten text lines to complete pages. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 264-273, June 2024.
- Philip Kahle, Sebastian Colutto, Günter Hackl, and Günter Mühlberger. Transkribus-a service platform for transcription, recognition and retrieval of historical documents. In 2017 14th Iapr International Conference on Document Analysis and Recognition (Icdar), volume 4, pages 19-24. IEEE, 2017.
- Clemens Neudecker, Konstantin Baierer, Maria Federbusch, Matthias Boenig, Kay-Michael Würzner, Volker Hartmann, and Elisa Herrmann. Ocr-d: An end-to-end open source ocr framework for historical printed documents. In Proceedings of the 3rd international conference on digital access to textual cultural heritage, pages 53-58, 2019.
- Emanuela Boros et al. Post-correction of historical text transcripts with large language models: An exploratory study. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), The 8th Joint SIGHUM Workshop on Compu- tational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, page 141. Association for Computational Linguistics, 2024.
- Alan Thomas, Robert Gaizauskas, and Haiping Lu. Leveraging llms for post-ocr correction of historical news- papers. In Rachele Sprugnoli and Marco Passarotti, editors, Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 116-121, Torino, Italia, 2024. ELRA and ICCL.
- Gemini Team. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv.org/abs/ 2312.11805.
- Gundram Leifert, C.A. Romein, Achim Rabus, Phillip Benjamin Ströbel, and Tobias Hodel. Transkribus and beyond: Pioneering the future of transcription technology. In Transkribus User Conference 2024, February 2024.
- Vinh-Nam Huynh, Ahmed Hamdi, and Antoine Doucet. When to use ocr post-correction for named entity recognition? In Emi Ishita, Natalie Lee San Pang, and Lihong Zhou, editors, Digital Libraries at Times of Massive Societal Transition, pages 33-42. Springer International Publishing, Cham, 2020.
- Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707-710, 1966.
- William E. Winkler. String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage. In Proceedings of the Section on Survey Research Methods, American Statistical Association, pages 354-359, 1990.
- John Evershed and Kent Fitch. Correcting Noisy OCR: Context beats Confusion. In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage, pages 45-51, Madrid, 2014.
- Samuel J. Schröckh. Allgemeine kaufmännische Bibliothek. in der Schröck'schen Buchhandlung, Frankfurt und Leipzig, 1778. Accessed via Google Books. URL: https://www.google.co.uk/books/edition/ Allgemeine_kaufm%C3%A4nnische_Bibliothek/7HtDIgZMC7EC?hl=en&gbpv=0.
Cited by
Benchmarking large language models for handwritten text recognition
Crosilla, G., Klic, L., & Colavizza, G. (2025) · Journal of Documentation · doi:10.1108/jd-03-2025-0082 · Cited by 34
Singh, L. G., & Middleton, S. E. (2025) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-025-00543-9 · Cited by 14
Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and Qwen2.5-VL
Vesalainen, A., Eetu Mäkelä, A., Ruotsalainen, L., & Tolonen, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36042-7_15 · Cited by 6
A Language Model from 1913: Pretraining on Historical Text
Luo, X., Shinnick, Z., Griesshaber, N., Wang, Y., Yu, J., Shi, F., Torr, P., & (185912), Y. L. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.2474 · Cited by 6
Self-citationOmniOCR: Generalist OCR for Ethnic Minority Languages
Liu, B., Zhang, Z., Meng, B., Wang, H., Zhang, H., Wang, C., Ergu, D., & Cai, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3432-5_41 · Cited by 5
Metadata
- Authors
- Greif, G., Griesshaber, N., & Greif, R.
- Affiliations
- University of Oxford (GB)
- University of Mannheim (DE)
- ORCID
- Greif, Robin
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.00414
- OpenAlex
- W4417226674
- Semantic Scholar
- 92f67092f29582178f2fd4428364a9846d59d8ce
- Cited by
- 49
- References
- 53
- Citations by year
- 2026: 38 · 2025: 11
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- University of Oxford
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- multimodal large language models, optical character recognition, named entity recognition, historical documents, city directories, transcription of historical documents, historical data collection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-13, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-13 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
