Unlocking the archives: Using large language models to transcribe handwritten historical documents

TL;DR

This paper tests commercial multimodal LLMs for transcribing handwritten historical documents and introduces an open-source tool, Transcription Pearl. It reports that LLMs outperform specialized HTR software out of the box and, when used for heterogeneous correction, approach non-expert human accuracy at much lower cost and higher speed.

Key findings

• On out-of-the-box transcription, Claude Sonnet-3.5 achieved strict CER 7.3% and WER 15.9%, and modified CER 5.7% and WER 8.9%; Transkribus Titan achieved strict CER 8.0% and WER 19.7%, while Transkribus PyLaia achieved strict CER 10.3% and WER 27.0%.
• LLM correction improved heterogeneous outputs: Claude Sonnet-3.5 correcting Gemini transcriptions achieved strict CER 5.7% and WER 13.8%, and modified CER 4.1% and WER 7.0%; self-correction was not statistically significant, with gpt-4o-06-08-2024 declining by 1–2% on strict and modified metrics.
• LLMs correcting Transkribus outputs reduced strict WER by 59% to an average of 9% and strict CER by 61% to around 4%; the paper reports Claude Sonnet-3.5 reduced Titan character-level errors by 68% to 1.8% and PyLaia word-level errors by 74% to 3.5%.
• LLMs completed a batch of 50 transcriptions in about 30 seconds versus about 22 minutes for Transkribus, roughly 50 times faster; correction cost $0.005–$0.01 per page, and automated LLM transcriptions cost about 1,500 times less than human transcription and roughly 50 times less than comparable automated HTR services.

Methods

• The authors built Transcription Pearl, an open-source Python GUI (3,907 lines) that calls commercial multimodal LLMs via API; tests used OpenAI gpt-4o-06-08-2024, Anthropic Claude Sonnet 3.5, and Google Gemini 1.5 Pro 002 with zero-shot prompts, temperature 0.0, and Top-P 1.0.
• The testing corpus comprised 50 pages (9,911 words) of English documents from 1761–1827 in 33 hands: 13 black-and-white microfilm scans, 26 medium-to-high-quality scans/photographs, and 11 iPhone or handheld photographs; all were 96 dpi, and ground truth was created from a Transkribus transcription manually corrected and proofed.
• Each test ran the 50 documents through Transkribus PyLaia/Titan and the LLM functions ten times (500 pages per model per test), averaged results, reported high/low scores and standard deviations, and computed strict and modified CER/WER with Python libraries while recording latency and cost.

Limitations

• Stated: the dataset is relatively small and focused solely on English-language 18th/19th-century documents; results may differ for other periods and languages; the dataset is not public; open-source models like Llama were not tested due to computing resources; and the tests used a simple zero-shot baseline, so further prompting may improve results.
• Stated: Gemini recitation false positives were possible, with one document flagged roughly 1 in 30 times; training-data pre-exposure could artificially deflate error rates, so the authors excluded internet-available documents.
• Evident from the text: only three commercial LLMs were tested; human error rates are cited from other studies rather than measured on the same corpus; and the ground truth was produced with Transkribus plus manual correction, which may shape the measured error rates.