The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index; none of the 6 sources its claims cite is checked for support yet.
Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books
Open accessAbstract
Large language models offer transformative potential for digitizing historical texts, but their application to humanities research raises critical questions about temporal bias and historical representation. We present the first systematic evaluation of multimodal LLMs for historical optical character recognition (OCR), testing 11 leading models on 1,030 pages of 18th-century Russian texts printed in Civil font. Using a contamination-free dataset from the National Library of Russia, we demonstrate that while LLMs substantially outperform traditional OCR systems (achieving 3.36% vs. 21.55–45.96% character error rates), they exhibit systematic temporal biases that fundamentally compromise historical authenticity. Our analysis reveals two distinct forms of distortion: a “modernization trap" where models automatically “correct" historical orthography to contemporary standards, and paradoxical “over-historicization" where models insert anachronistic medieval Slavonic characters into 18th-century texts. These errors reflect what we term the absence of “historical linguistic competence" — models treat historical language not as a continuum of specific periods but as an undifferentiated space labeled “old". Different model families exhibit distinct error signatures, exposing how architectural choices and training data composition shape temporal bias. These findings reveal that “epistemic anachronism" in AI systems goes beyond inherited editorial biases. While training data explains modernization, the concurrent archaization demonstrates a fundamental architectural limitation: without temporal metadata as a training signal, models cannot develop “historical linguistic competence" even when explicitly provided with dates. Our work shows how these systems create temporal chimeras that appear historical while actively corrupting the historical record.
Overview
CoChat Summary
TL;DR
This paper systematically evaluates 11 multimodal LLMs for historical OCR on 1,030 pages of 18th-century Russian Civil-font texts, finding that they outperform traditional OCR but systematically distort historical orthography. The authors identify a “modernization trap” and paradoxical “over-historization,” arguing that current models lack historical linguistic competence and create temporal chimeras.
Key findings
- Multimodal LLMs substantially outperformed traditional OCR systems: best model Gemini-2.5-Pro achieved 3.36% CER and 4.69% WER, versus 21.55–45.96% CER for Tesseract, Surya OCR, and Transkribus; a fine-tuned TrOCR model achieved 1.83% CER.
- Models systematically modernized historical orthography, e.g., replacing ѣ→е, і→и, and ъ omissions; for the letter ї, preservation rates ranged from 2.1% (Qwen-2.5-VL, o4-mini) to 30.3% (Gemini-2.5-Flash), with 60–84% of instances substituted to modern і.
- Models also inserted anachronistic pre-Petrine or medieval Slavonic characters into 18th-century texts (“over-historization”); Russian-language prompts reduced archaic insertions by 45% (0.23→0.13 per document, p = 0.200), while improvements in legitimate character preservation were highly significant (p < 0.001).
- Distinct model-family error signatures emerged: OpenAI models showed over-historicization and GPT-4.1 substituted Cyrillic ї with Latin i in 19.5% of cases; Gemini models had highest preservation (Gemini-2.5-Pro: 93.1% ъ, 91.7% ѣ) but still failed ї in 70% of cases; Claude models showed case inversion errors (ъ→Ъ: 16.5–18.4%).
Methods
- Dataset: 1,030 scanned pages from 428 unique 18th-century Russian books (1750–1800) printed in Civil font, sourced from the National Library of Russia’s limited-access collection and designed to prevent training-data contamination.
- Ground truth: automated layout analysis with fine-tuned YOLOv8, initial OCR with a TrOCR model fine-tuned on 15,914 lines (575 pages), and 100% manual correction following diplomatic transcription principles; a 100-page sample with 2,954 lines and 15,970 words was published.
- Evaluation: 11 multimodal LLMs tested in single-line, full-page, and sliding-window (3-line) modes; metrics included CER, WER, CI-CER, and HChE; controlled prompt engineering compared basic English, context-enhanced English with historical metadata, and context-enhanced Russian prompts; stability tested over seven consecutive days and sensitivity analysis across 17 document features.
Limitations
- Stated: scope is limited to 18th-century Russian Civil-font texts (1750–1800), so findings may not generalize to other languages, scripts, handwritten manuscripts, earlier print traditions, or other historical transitions.
- Stated: ground truth may contain residual annotation errors; the 1,030-page corpus is a fraction of 18th-century Russian print culture; API-only access prevented analysis of internal model architecture or training data; model outputs vary between runs; some models were excluded due to cost; and commercial models are continuously updated, challenging exact replication.
- Stated: causal connections between error patterns and training-data composition remain partially inferential, and model-family “signatures” are observable patterns rather than confirmed architectural explanations; no additional limitations beyond those stated are evident from the text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 1 Not valid · 6 Not checked · 134 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Paradoxically, OpenAI models also insert modern characters such as ё, which only appeared after 1797.”
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > 5.4 Model Family "Signatures" and Error Taxonomy
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Characters eliminated in the 1918 reform are consistently modernized (ѣ (yat) → е), і (decimal i) → и, (fita) → ф) or totally skipped.”
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > 2. Historical Character Replacement / Elimination.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Paradoxically, when multimodal LLMs are prompted to recognize 18th-century book pages as 'historical' Russian text, they frequently insert anachronistic characters drawn from much earlier periods -a phenomenon we term over-historization.”
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > 5.2 Over-historization: The Medieval Default
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“paradoxical “over-historicization” where models insert anachronistic medieval Slavonic characters into 18th-century texts.”
- Also stated in: § 5.2 Over-historization: The Medieval Default · plain
“when multimodal LLMs are prompted to recognize 18th-century book pages as 'historical' Russian text, they frequently insert anachronistic characters drawn from much earlier periods -a phenomenon we term over-historization.”
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5.2 Over-historization: The Medieval Default · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The model achieves moderate historical character preservation (ъ: 68-73%, ѣ: 82-86%)”
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > 5.4 Model Family "Signatures" and Error Taxonomy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
70 more claims
Create a free account to see all 141 claims and every source verdict.
- It may state two separate claims as one. 42
- Probably not a substantial claim. 31
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > 5.4 Model Family "Signatures" and Error Taxonomy
“Table 3: Historical error signatures in multimodal LLMs.Archaic insertions include pre-1708 characters ( , , ѵ) and Old Church Slavonic diacritics, eliminated in Peter's reform.”
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > 3 Peter the Great's Civil Script Reform and Its Digital Legacy
“They completed Peter's rationalization, eliminating four letters (ѣ, , і, ѵ) and terminal hard signs (ъ).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > 3. The Lost ї Phenomenon.
“with 60-84% of instances systematically substituted to modern і.”
§ Building Historical Corpora with Multimodal LLMs: Epistemic Gaps and Misreadings in 18th-Century Russian Books > Historical Form
“І, і decimal i И, и лінія → линия”
Citations
Cites
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
Time-aware language models as temporal knowledge bases
Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00459 · Cited by 212
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
Rao, A. S., Yerukola, A., Shah, V., Reinecke, K., & Sap, M. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.120 · Cited by 112
Digital History and the Politics of Digitization
Zaagsma, G. (2022) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqac050 · Cited by 50
Machine Bias. How Do Generative Language Models Answer Opinion Polls?
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2025) · Sociological Methods & Research · doi:10.1177/00491241251330582 · Cited by 48
Language Models Represent Space and Time
Gurnee, W., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02207 · Cited by 46
LLMs, Truth, and Democracy: An Overview of Risks
Coeckelbergh, M. (2025) · Science and Engineering Ethics · doi:10.1007/s11948-025-00529-0 · Cited by 46
Large Language Models based on historical text could offer informative tools for behavioral science
Varnum, M. E. W., Baumard, N., Atari, M., & Gray, K. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2407639121 · Cited by 39
Can Language Models Represent the Past without Anachronism?Preprint
Underwood, T., Nelson, L. K., & Wilkens, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.00030 · Cited by 35
Time is Encoded in the Weights of Finetuned Language Models
Nylund, K., Gururangan, S., & Smith, N. A. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.141 · Cited by 25
Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities
Levchenko, M. (2025) · doi:10.26615/978-954-452-106-6-007 · Cited by 20
Self-citationDissecting Bias in LLMs: A Mechanistic Interpretability PerspectivePreprint
Chandna, B., Bashir, Z., & Sen, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.05166 · Cited by 17
Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information
Park, Y., Yoon, C., Park, J., Jeong, M., & Kang, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.812 · Cited by 8
Context-Independent OCR with Multimodal LLMs: Effects of Image Resolution and Visual ComplexityPreprint
Inoue, K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.23667 · Cited by 7
Open Problems and a Hypothetical Path Forward in LLM Knowledge ParadigmsPreprint
Ye, X., Zhang, M., & Wu, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.06823 · Cited by 3
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
Rao, A., Yerukola, A., Shah, V., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.12464
Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities
Levchenko, M. (2025) · arXiv · doi:10.48550/arxiv.2510.06743
Self-citationTime is Encoded in the Weights of Finetuned Language Models
Nylund, K., Gururangan, S., & Smith, N. A. (2023) · arXiv · doi:10.48550/arxiv.2312.13401
Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information
Park, Y., Yoon, C., Park, J., Jeong, M., & Kang, J. (2025) · arXiv · doi:10.48550/arxiv.2502.14258
Research works printed but not linked: 1
- Theimer, Kate. "A Distinction Worth Exploring: "Archives" and "Digital Historical Rep- resentations"". In: Journal of Digital Humanities 3, no. 2 (2014). Summer. URL: https : //journalofdigitalhumanities.org/3-2/a-distinction-worth-exploring-archives-and- digital-historical-representations/.
Cited by
Carniolan Provincial Assembly: Corpus Improvements and Enhancements
Žagar, A. P., & de Maiti, K. P. (2026) · Digital Humanities in the Nordic and Baltic Countries Publications · doi:10.5617/dhnbpub.13202
Levchenko, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.27275
Self-citation
Metadata
- Authors
- Levchenko, M.
- Affiliations
- University of Bologna (IT)
- Corresponding author
- Levchenko, Maria
- ORCID
- Levchenko, Maria
- Venue
- Anthology of Computers and the Humanities
- Published
- 2025-11-21
- Pages
- 725-737
- ISSN
- 3070-8931
- OpenAlex
- W4416264675
- Semantic Scholar
- ae6ae6f59e76e5a7c24ef748f931a4ae9416cda0
- Cited by
- 2
- References
- 14
- Citations by year
- 2026: 2
- FWCI
- 62.97
- Type
- book chapter
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Topics
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multimodal large language models, temporal bias, historical orthography, anachronism, archaization, historical corpora, National Library of Russia, training data composition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-13, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-13 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
