Student or educator? Sign up with your school email and get 6 months of Pro free.
PreprintCited by 49Handwritten Text Recognition TechniquesTopic ModelingDigital Humanities and Scholarship

The DOI matches its DataCite record; 54 of the 56 references it cites were found in a registry or an index, and 2 are not checked yet; none of the 35 sources its claims cite is checked for support yet.

Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents

Open access
Greif, Gavin, Griesshaber, Niclas, Greif, Robin·2025·arXiv

Abstract

We explore how multimodal Large Language Models (mLLMs) can help researchers transcribe historical documents, extract relevant historical information, and construct datasets from historical sources. Specifically, we investigate the capabilities of mLLMs in performing (1) Optical Character Recognition (OCR), (2) OCR Post-Correction, and (3) Named Entity Recognition (NER) tasks on a set of city directories published in German between 1754 and 1870. First, we benchmark the off-the-shelf transcription accuracy of both mLLMs and conventional OCR models. We find that the best-performing mLLM model significantly outperforms conventional state-of-the-art OCR models and other frontier mLLMs. Second, we are the first to introduce multimodal post-correction of OCR output using mLLMs. We find that this novel approach leads to a drastic improvement in transcription accuracy and consistently produces highly accurate transcriptions (<1% CER), without any image pre-processing or model fine-tuning. Third, we demonstrate that mLLMs can efficiently recognize entities in transcriptions of historical documents and parse them into structured dataset formats. Our findings provide early evidence for the long-term potential of mLLMs to introduce a paradigm shift in the approaches to historical data collection and document transcription.

Checked 2026-11-13 · 2 references not checked yet

Overview

CoChat Summary

TL;DR

This paper benchmarks multimodal LLMs (GPT-4o, Gemini 2.0 Flash) against conventional OCR engines for transcribing German city directories printed 1754–1870, and introduces image-aware multimodal OCR post-correction. Gemini 2.0 Flash produced the most accurate off-the-shelf transcriptions and, when used to post-correct a Transkribus model's output, achieved the lowest error rates of any tested approach.

Key findings

  • Off-the-shelf OCR: Gemini 2.0 Flash achieved a normalized CER of 1.27% across the corpus, versus 3.67% for Transkribus Print M1, 4.16% for Text Titan I, and 6.31% for GPT-4o. Tesseract's "deu_frak" model was far worse (30.22% normalized CER, 199.91% normalized WER) and was omitted from the results table.
  • Multimodal OCR post-correction (feeding the noisy Print M1 transcription plus the source PNG back into an mLLM) yielded the best results overall: 0.84% normalized CER with Gemini 2.0 Flash (as low as 0.08% for Trier-1853) and 1.00% with GPT-4o, with no image pre-processing or model fine-tuning.
  • NER: with ground-truth text as input, Gemini 2.0 Flash reached 89.32% fuzzy match across the full sample (up to 99.49% for Aachen-1838); with noisy OCR text, 88.61%; extracting entities directly from the PNG, 80.86%. GPT-4o scored lower throughout (91.00%, 87.58%, and 58.38% fuzzy respectively).
  • mLLMs struggled with directories organized by sub-headings (e.g., Leipzig-1753 at 73.75% fuzzy from GT text), and font type (Antiqua vs. Fraktur) appeared to have no significant impact on parsing accuracy.

Methods

  • Corpus of 30 page-images drawn from 10 German-language city directories published 1754–1870 in Aachen, Dresden, Leipzig, Frankfurt, Lübeck, Riga, and Trier, predominantly Blackletter/Fraktur with one Antiqua directory; images sourced from archive websites and deliberately left un-preprocessed.
  • Three experiment sets: (1) OCR benchmarking of Gemini 2.0 Flash and GPT-4o against Tesseract 5.5.0 "deu_frak" and Transkribus Print M1 and Text Titan I; (2) multimodal OCR post-correction using the best non-mLLM transcription plus the source image; (3) NER/entity parsing into JSON then CSV from ground-truth text, noisy OCR text, and the raw PNG.
  • Accuracy measured via Levenshtein distance, CER and WER (reported both normalized and non-normalized); NER evaluated with strict string matching and fuzzy matching using Jaro-Winkler similarity with a 0.9 threshold. Prompts were developed iteratively with mLLM assistance; temperature set to 0.0.

Limitations

  • Stated by the paper: the corpus is restricted to historical Latin-alphabet prints, so performance on non-Latin-alphabet sources is unknown; the authors could not provide an in-depth comparison of mLLMs against conventional fine-tuned NER approaches; a comprehensive analysis of mLLM-assisted prompt design is beyond the paper's scope; and the authors call for a dedicated historical OCR benchmark for mLLMs.
  • Evident from the text: the corpus is small (30 pages from 10 directories), and NER evaluation excluded directories whose generated CSV did not match the ground-truth row count after one attempt — Gemini and GPT-4o GT/OCR text results are based on only eight directories, and GPT-4o's PNG-to-CSV results on only three. The authors also note they cannot disentangle whether PNG-to-CSV performance differences stem from OCR versus entity-recognition ability, given the models' black-box nature.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.00414

Cites

54 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 2 Not valid · 20 Not checked · 68 Own results and interpretations

50 more claims

Create a free account to see all 90 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 15
  • It may state two separate claims as one. 24

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 53 references

Research works printed but not linked: 30

  1. Eltjo Buringh and Jan Luiten Van Zanden. Charting the "rise of the west": Manuscripts and printed books in europe, a long-term perspective from the sixth through eighteenth centuries. The Journal of Economic History, 69 (2):409-445, 2009.
  2. Lynne Tatlock. Introduction: The book trade and "reading nation" in the long nineteenth century. In Lynne Tatlock, editor, Publishing Culture and the "Reading Nation": German Book History in the Long Nineteenth Century, Studies in German Literature, Linguistics, and Culture, pages 1-24. Camden House, Rochester, NY, 2010.
  3. Alaa Sulaiman, Khairuddin Omar, and Mohammad F Nasrudin. Degraded historical document binarization: A review on issues, challenges, techniques, and future directions. Journal of imaging, 5(4):48, 2019.
  4. Phillip Benjamin Ströbel. Flexible Techniques for Automatic Text Recognition of Historical Documents. PhD thesis, University of Zurich, Zurich, Switzerland, 2023.
  5. Uwe Springmann and Anke Lüdeling. Ocr of historical printings with an application to building diachronic corpora: A case study using the ridges herbal corpus. Digital Humanities Quarterly, 11(2), 2017.
  6. Shaobin Xu and David Smith. Retrieving and combining repeated passages to improve ocr. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1-4, 2017.
  7. Huaigu Cao, Stephen Rawls, and Prem Natarajan. 1990 us census form recognition using ctc network, wfst language model, and surname correction. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 977-982, 2017.
  8. Quan Duong, Mika Hämäläinen, and Simon Hengchen. An unsupervised method for ocr post-correction and spelling normalisation for finnish. In Simon Dobnik and Lilja Øvrelid, editors, Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 240-248, Reykjavik, Iceland (Online), 2021. Linköping University Electronic Press, Sweden.
  9. Oksana Dereza, Deirdre Ní Chonghaile, and Nicholas Wolf. "To Have the 'Million' Readers Yet": Building a Digitally Enhanced Edition of the Bilingual Irish-English Newspaper an Gaodhal (1881-1898). In Rachele Sprugnoli and Marco Passarotti, editors, Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 65-78, Torino, Italia, 2024. ELRA and ICCL.
  10. Mahdi Hajiali. OCR Post-processing Using Large Language Models. Phd thesis, University of Nevada, Las Vegas, August 2023. URL https://digitalscholarship.unlv.edu/thesesdissertations/4811.
  11. Nianheng Wu. Multimodal ocr post-correction on german historical documents. Master's thesis, University of Stuttgart, 2023.
  12. Oksana Dereza, Deirdre Ní Chonghaile, and Nicholas Wolf. "To Have the 'Million' Readers Yet": Building a Digitally Enhanced Edition of the Bilingual Irish-English Newspaper an Gaodhal (1881-1898). In Rachele Sprugnoli and Marco Passarotti, editors, Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 65-78, Torino, Italia, 2024. ELRA and ICCL.
  13. Andre Wolters and Andreas Van Cranenburgh. Historical dutch spelling normalization with pretrained language models. Computational Linguistics in the Netherlands Journal, 13:147-171, March 21 2024.
  14. Viktoria Löfgren and Dana Dannélls. Post-OCR correction of digitized Swedish newspapers with ByT5. In Yuri Bizzoni, Stefania Degaetano-Ortlieb, Anna Kazantseva, and Stan Szpakowicz, editors, Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 237-242, St. Julians, Malta, March 2024. Association for Computational Linguistics.
  15. Jonathan Bourne. CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models. arXiv, 2408.17428, 2025. Preprint. · arXiv
  16. Yung-Hsin Chen and Phillip B. Ströbel. Trocr meets language models: An end-to-end post-correction approach. In Harold Mouchère and Anna Zhu, editors, Document Analysis and Recognition -ICDAR 2024 Workshops, volume 14935 of Lecture Notes in Computer Science, pages 12-26. Springer Nature Switzerland, Cham, 2024.
  17. Maud Ehrmann et al. Named entity recognition and classification on historical documents: A survey. ACM Computing Surveys, 56(2):1-47, February 2024.
  18. Denis Coquenet, Clément Chatelain, and Thierry Paquet. Dan: a segmentation-free document attention network for handwritten document recognition. IEEE transactions on pattern analysis and machine intelligence, 45(7): 8227-8243, 2023.
  19. Dayvid Castro, Byron Leite Dantas Bezerra, and Cleber Zanchettin. An end-to-end approach for handwriting recognition: From handwritten text lines to complete pages. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 264-273, June 2024.
  20. Philip Kahle, Sebastian Colutto, Günter Hackl, and Günter Mühlberger. Transkribus-a service platform for transcription, recognition and retrieval of historical documents. In 2017 14th Iapr International Conference on Document Analysis and Recognition (Icdar), volume 4, pages 19-24. IEEE, 2017.
  21. Clemens Neudecker, Konstantin Baierer, Maria Federbusch, Matthias Boenig, Kay-Michael Würzner, Volker Hartmann, and Elisa Herrmann. Ocr-d: An end-to-end open source ocr framework for historical printed documents. In Proceedings of the 3rd international conference on digital access to textual cultural heritage, pages 53-58, 2019.
  22. Emanuela Boros et al. Post-correction of historical text transcripts with large language models: An exploratory study. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), The 8th Joint SIGHUM Workshop on Compu- tational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, page 141. Association for Computational Linguistics, 2024.
  23. Alan Thomas, Robert Gaizauskas, and Haiping Lu. Leveraging llms for post-ocr correction of historical news- papers. In Rachele Sprugnoli and Marco Passarotti, editors, Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 116-121, Torino, Italia, 2024. ELRA and ICCL.
  24. Gemini Team. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv.org/abs/ 2312.11805.
  25. Gundram Leifert, C.A. Romein, Achim Rabus, Phillip Benjamin Ströbel, and Tobias Hodel. Transkribus and beyond: Pioneering the future of transcription technology. In Transkribus User Conference 2024, February 2024.
  26. Vinh-Nam Huynh, Ahmed Hamdi, and Antoine Doucet. When to use ocr post-correction for named entity recognition? In Emi Ishita, Natalie Lee San Pang, and Lihong Zhou, editors, Digital Libraries at Times of Massive Societal Transition, pages 33-42. Springer International Publishing, Cham, 2020.
  27. Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707-710, 1966.
  28. William E. Winkler. String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage. In Proceedings of the Section on Survey Research Methods, American Statistical Association, pages 354-359, 1990.
  29. John Evershed and Kent Fitch. Correcting Noisy OCR: Context beats Confusion. In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage, pages 45-51, Madrid, 2014.
  30. Samuel J. Schröckh. Allgemeine kaufmännische Bibliothek. in der Schröck'schen Buchhandlung, Frankfurt und Leipzig, 1778. Accessed via Google Books. URL: https://www.google.co.uk/books/edition/ Allgemeine_kaufm%C3%A4nnische_Bibliothek/7HtDIgZMC7EC?hl=en&gbpv=0.

Cited by

Metadata

Authors
Greif, G., Griesshaber, N., & Greif, R.
Affiliations
  • University of Oxford (GB)
  • University of Mannheim (DE)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4417226674
Cited by
49
References
53
Citations by year
2026: 38 · 2025: 11
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
University of Oxford
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
Keywords
multimodal large language models, optical character recognition, named entity recognition, historical documents, city directories, transcription of historical documents, historical data collection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-13, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-13 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.