The DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 4 sources its claims cite is checked for support yet.
Benchmarking large language models for handwritten text recognition
Open accessAbstract
Purpose: The aim of this work is to provide an overview of the current capabilities of Multimodal Large Language Models (MLLMs) for Handwritten Text Recognition (HTR), assessing their potential when compared to traditional task-specific, supervised models. Design/methodology/approach: The approach is that of using a set of openly-available benchmarks to compare different LLMs with strong task-specific supervised baselines for the task of HTR. Findings: The results show that LLMs currently show a strong performance on English texts, yet they demonstrate a weaker performance on languages other than English, and do not possess a significant capability for self-correction. Moreover, their comparison with Transkribus’s models highlight the fact that proprietary LLM models are the best performing, in particular on modern handwriting, while for historical documents the overall performance comparison between LLMs and Transkribus is not consistent. Originality/value: The authors are not aware of a similar study relying on open benchmarks.
Checked 2026-11-13 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
This paper benchmarks eight multimodal LLMs against Transkribus models for handwritten text recognition using open datasets and zero-shot plus self-correction prompts. It finds LLMs are strong on English and modern handwriting, weaker on non-English and historical documents, and do not reliably self-correct.
Key findings
- LLMs perform well on English and modern handwriting: on IAM, GPT-4o-mini reaches 1.71% CER and 3.34% WER; on RIMES, GPT-4o reaches 1.69% CER (Table 5).
- Performance declines for non-English and historical material: for Italian, Claude Sonnet 3.5 is best but still poor, with 20.55% CER / 27.78% WER on LAM and 26.43% CER / 35.08% WER on Leopardi; for German/multilingual, The Text Titan achieves 40.63% CER / 64.28% WER on READ2016 and 14.40% CER / 29.40% WER on ICDAR2017.
- Self-correction is inconsistent: the best improvement was GPT-4o on ICDAR2017 with an 8% CER and 4.7% WER decrease, but results remained unusable; open-source models failed to improve and often increased error rates.
- A masked-word pre-training check found minimal correct guesses: Claude Sonnet 3.5 guessed 27/300, GPT-4o 17/300, and open-source models 0–2/300.
Methods
- Benchmarked eight MLLMs—three proprietary (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet) and five open-source (MiniCPM-V-2 6, Qwen2-VL-7B, Pixtral-12B, InternVL2-8B, Phi-3-mini-instruct-128k)—against Transkribus models (The Text Titan I, Italian Handwriting M1) on seven open datasets: IAM, RIMES, LAM, Leopardi, Bentham, READ2016, and ICDAR2017.
- Used zero-shot prompting with the same prompts and hyperparameters across models (temperature 0/greedy decoding; repetition penalty 1.2), then a three-step self-correction procedure; evaluated with CER, WER, and WER-BoW.
- Ran a name-cloze masked-word test on 100 pages / 300 masked words to assess whether HTR benchmark datasets may have been in LLM pre-training.
Limitations
- Paper-stated: dataset choice was limited to available HTR datasets in a few languages and does not fully capture multilingual contexts; the baseline prompt lacked contextual elements that could improve results; open-source models were limited to up to 12B parameters due to computational constraints.
- Evident from the text: the study is a snapshot of models available as of October 2024, uses only zero-shot and intrinsic self-correction without fine-tuning, and does not cover ideographic or non-Latin scripts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 0 Not valid · 4 Not checked · 45 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, it can be observed that post-corrections produced by open-source models reduced accuracy overall.”
§ Benchmarking large language models for handwritten text recognition > 5. Conclusion and future developments
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, their comparison with Transkribus's models highlight the fact that proprietary LLM models are the best performing, in particular on modern handwriting, while for historical documents the overall performance comparison between LLMs and Transkribus is not consistent.”
- Also stated in: § 4. Analysis of results and discussion · hedged
“However, it cannot be broadly stated that Transkribus models consistently outperform LLMs, as accuracy varies depending on the specific use case.”
§ Benchmarking large language models for handwritten text recognition > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4. Analysis of results and discussion · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While The Text Titan achieves the best outcome with 7.07% CER and 12.41% WER, LLMs are not far behind.”
§ Benchmarking large language models for handwritten text recognition > 4. Analysis of results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“open-source model performance decreases significantly for historical documents in other languages.”
§ Benchmarking large language models for handwritten text recognition > 5. Conclusion and future developments
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While the accuracy is similar between proprietary and open-source models on modern handwriting and English materials”
§ Benchmarking large language models for handwritten text recognition > 5. Conclusion and future developments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
21 more claims
Create a free account to see all 49 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 17
- The quote only partly states this claim. 1
- Probably not a substantial claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking large language models for handwritten text recognition > 4. Analysis of results and discussion
“On IAM, GPT-4o-mini attains 1.71% CER and 3.34% WER, while on RIMES, GPT-4o reaches 1.69% CER and 3.66% WER, outperforming Transkribus' supermodel.”
§ Benchmarking large language models for handwritten text recognition > 5. Conclusion and future developments
“| Gpt-4o-2024-08-06 | 1.69% | 3.76% | 3.30% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking large language models for handwritten text recognition > 4. Analysis of results and discussion
“In German and multilingual datasets, The Text Titan surpasses LLMs, achieving 40.63%CER and 64.28% WER on the READ2016 dataset and 14.40% CER and 29.40% WER on ICDAR2017.”
§ Benchmarking large language models for handwritten text recognition > 5. Conclusion and future developments
“| The Text Titan I | 14.40% | 29.47% | 24.69% |”
Citations
Cites
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,358
The IAM-database: an English sentence database for offline handwriting recognition
Marti, U.-V., & Bunke, H. (2002) · International Journal on Document Analysis and Recognition · doi:10.1007/s100320200071 · Cited by 1,589
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214
TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models
Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D. A. F., Zhang, C., Li, Z., & Wei, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26538 · Cited by 923
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712
Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition?
Puigcerver, J. (2017) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2017.20 · Cited by 365
A full English sentence database for off-line handwriting recognition
Marti, U.-V., & Horst Bunke, A. (1999) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.1999.791885 · Cited by 250
Handwriting Recognition with Large Multidimensional Long Short-Term Memory Recurrent Neural Networks
Voigtlaender, P., Doetsch, P., & Ney, H. (2016) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2016.0052 · Cited by 229
Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention
Bluche, T., Louradour, J., & Messina, R. (2017) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2017.174 · Cited by 205
Research works printed but not linked: 3
- Augustin, E., Carré, M., Grosicki, E., Brodin, J., Geoffrois, E. and Prêteux, F. (2006), “RIMES evaluation campaign for handwritten mail processing”, available at: https://www.semanticscholar.org/paper/RIMES-evaluation-campaign-for-handwritten-mail-Augustin-Carr%C3%A9/1a08e3055dd76c307f5f2993d54465dd407ad1ab (accessed 14 January 2025).
- Cascianelli, 2021, Computer Analysis of Images and Patterns. CAIP 2021. Lecture Notes in Computer Science, 340
- Ströbel, P.B. (2023), “Flexible techniques for automatic text recognition of historical documents”, PhD dissertation, University of Zurich, available at: https://core.ac.uk/outputs/574072779/?source=2 (accessed 15 January 2025).
Cited by
Belot, M., Tuberosa, J., Preuss, L., Svezhentseva, O., Claessen, M., Bölling, C., Schuster, F., & Léger, T. (2026) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.70235 · Cited by 4
Ardanuy, M. C., Berganzo‐Besga, I., Huesca, R. S., & Cuadrada, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04624-6_20 · Cited by 3
A Multi-Stage Pipeline for Accurate Handwritten Information Extraction from Financial Forms
Wang, G., Yu, J., Zhang, X., Deb, T., Liu, X., & He, P. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00634 · Cited by 1
METATR: A Multilingual, Evolving Benchmark for Automatic Text Recognition
Boillet, M., Tarride, S., & Kermorvant, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36039-7_2 · Cited by 1
Semi-automatic pipeline for constructing HTR corpora from Ukrainian-language historical documents
Івасечко, А. В., & Lipianina-Honcharenko, K. (2026) · doi:10.55056/ceur-ws.org/vol-4159/paper12.pdf · Cited by 1
Machidon, O. M., Rejec, J., Vreš, D., & Machidon, A. L. (2026) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-026-00612-7 · Cited by 1
An HTR-LLM Workflow for High-Accuracy Transcription and Analysis of Abbreviated Latin Court HandPreprint
Isom, J. D. (2025) · arXiv (Cornell University) · Cited by 1
AI-Driven Handwritten Text Recognition and Digital Conversion for a Smart Campus
Parsai, Y., Rai, V. K., Tripathi, S., Meena, K. B., & Tyagi, V. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-6831-3.ch002
Self-Governing Agentic AI-driven Framework for Multi-Phase Handwriting Identification Tasks
Menaka, R., Sowmiyaa, R., Nivethitha, M., Tharshan, S., & Mithunraj, P. (2025) · International Conference on Emerging Research in Electronics, Computer Science and Technology (ICERECT) · doi:10.1109/icerect65215.2025.11377691
Rybenská, K., & Sklenářová, S. (2026) · Pedagogika · doi:10.14712/23362189.2025.4918
ELLMW: an enhanced vision–language model for reliable text extraction from manually composed scripts
Dhivya, V., & Rajakumari, S. B. (2026) · International Journal of Reconfigurable and Embedded Systems (IJRES) · doi:10.11591/ijres.v15.i1.pp194-203
Znaczenie nowoczesnych metod cyfrowych dla badań nad misjami jezuickimi w Chinach
Śmigla, T. (2026) · Akademia Ignatianum w Krakowie: Czasopisma (Akademia Ignatianum w Krakowie) · doi:10.35765/rfi.2026.3201.17
Hybrid YOLOv8–TrOCR Method for Handwritten Text Extraction
TRAORE, F., Séré, A., & Bamogo, M. (2026) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa67686.2026.11468250
From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten FormsPreprint
Pather, N., Fouché, J., Mundia, S., Technau, K., Malaba, T. R., Welte, A., Mehta, U., & Bassett, B. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.16504
LLM benchmarking for HTR on french historical death records
Frossard, G., Theurel, F., AlKendi, W., & Gechter, F. (2026) · International Journal of Digital Humanities · doi:10.1007/s42803-026-00137-9
Jacobs, T. G., Beca, L. S. D. A. B., van der Linden, P. D., & van Nuland, M. (2026) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2026.100807
Déchiffrer l'illisible: Un modèle d'HTR pour l'écriture d'Henri Pirenne
Luján, C. C., de Valeriola, S., Wilkin, A., & Ceresato, F. (2026) · Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles)
Grabowski, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.11477
Towards an Interoperable Corpus of Austrian Historical Newspapers: The case of PressMint-AT
Wissik, T., Hassenbach, J. M., Pirker, H., Resch, C., & Resch, S. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4ionpc6w2rmt
APAU-Net: Adaptive prior-aware U-Net for handwritten text-line segmentation in historical documents
Beghoura, M. A., Attıa, A., Bouzıane, A., & Hassaballah, M. (2026) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-026-00599-1
From Text to Map: AI-Based Graphic Translation of Information
Biolo, F., Guzzetti, F., & Balestreri, I. C. R. (2026) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlix-b2-2026-1389-2026
Zhang, N., Coquenet, D., Soullard, Y., & Coüasnon, B. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7225575
Taşyürek, E. Y., Omurca, S. İ., Gerçeker, E., YILMAZ, Ç., İnceoğlu, H., Tufan, F. İ., Damar, T. P., & Çakıl, L. (2026) · 2026 International Conference Automatics, Robotics and Artificial Intelligence (ICARAI) · doi:10.1109/icarai70085.2026.11635744
Hallyburton, T., Berset, L., Fink, G. A., Fischer, A., & Scius-Bertrand, A. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-2600-9_10
Recent Advances in Information Extraction from Historical Archival Records
Matei, A., Hallyburton, T., Hennies, L., Raß, C., & Fink, G. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36039-7_6
Improving MLLM Historical Record Extraction with Test-Time Image Augmentation
Archibald, T., Morse, B. S., & Martinez, T. R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36023-6_11
When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset StudyPreprint
Farazi, M., Alam, M. F., Maaradji, A., Maamar, Z., Mubarak, H., & Zaghouani, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.22366
OCR-Based Field Extraction for Archaeological Pottery Metadata: The CENTURIA DatasetPreprint
Naghavi, G. V., Hagmann, D., Kampel, M., & Ballester, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30616
Ragavenderan, N., Jakob, J., & Manonmani, S. (2026) · IEEE Access · doi:10.1109/access.2026.3729718
Carrascosa, A., Ardanuy, M. C., Servat, L., Malaga, A., Huesca, R. S., Cuadrada, C., & Crosas, M. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846511
Respublika yigʻma elektron katalogi markazini rivojlantirishning strategik yoʻnalishlari
Xusanova, L. (2026) · Central Asian Journal of Academic Research · doi:10.70413/cajar-v4-i9-part-2-58008
Towards Robust Text Recognition: Benchmarking AI Models and Designing a High Accuracy Alternative
Shetty, S. K., & Ansari, A. (2026) · International Journal of Creative and Open Research in Engineering and Management · doi:10.55041/ijcope.v2i10.120
CENATAV-HTR: A Spanish Offline Handwriting Dataset and Benchmark Evaluation
González-Gazapo, R. A., Alonso, E. C., Martínez-Díaz, Y., Morales-González, A., & Méndez-Vázquez, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-43214-8_19
Kalmbach, T., & Wagner, A. (2026) · Research Square · doi:10.21203/rs.3.rs-11348739/v1
Metadata
- Authors
- Crosilla, G., Klic, L., & Colavizza, G.
- Affiliations
- University of Bologna (IT)
- Digital Humanities Lab, Villa I Tatti , ,
- University of Copenhagen (DK)
- Venue
- Journal of Documentation
- Publisher
- Emerald
- Published
- 2025-08-15
- Volume
- 81
- Issue
- 7
- Pages
- 334-354
- ISSN
- 0022-0418 (ISSN-L)
- 1758-7379
- OpenAlex
- W4414311642
- Cited by
- 34
- References
- 50
- Citations by year
- 2026: 30 · 2025: 4
- FWCI
- 8.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Documentation (publishedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- Harvard University Center for Italian Renaissance Studies
- Harvard University
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Vehicle License Plate Recognition
- Keywords
- multimodal large language models, handwriting recognition, benchmarking, Transkribus, self-correction, historical documents, multilingual performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-13, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-13 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
