CoChat

Large language models struggle with ethnographic text annotation

Open access

The DOI matches its DataCite record; 59 of the 59 references it cites were found in a registry or an index; none of the 18 sources its claims cite is checked for support yet.

Abstract

Large language models (LLMs) have shown promise for automated text annotation, raising hopes that they might accelerate cross-cultural research by extracting structured data from ethnographic texts. We evaluated 7 state-of-the-art LLMs on their ability to annotate 121 ritual features across 567 ethnographic excerpts. Performance was limited, falling well below levels required for reliable automated annotation. Longer texts, features requiring ordinal distinctions, and ambiguous constructs proved particularly difficult. Human inter-coder reliability set an approximate ceiling on LLM accuracy: features that human coders found difficult to agree upon were also difficult for LLMs. Yet even on features where humans reliably agreed, models fell short of human performance. Our findings suggest that LLMs cannot yet substitute for human expertise in ethnographic annotation.

Overview

CoChat Summary

TL;DR

Evaluating 7 state-of-the-art LLMs on 121 ritual features across 567 ethnographic excerpts, the best configuration reached only F1=0.41 on the larger morphospace dataset and F1=0.54 on the smaller synchrony dataset. The authors conclude LLMs cannot yet substitute for human expertise in ethnographic annotation.

Key findings

  • Across all models and prompting conditions, morphospace annotation was poor: mean F1 ranged from 0.12 to 0.41, with the best configuration (DeepSeek V3.1 and GPT-OSS 120B under multi-task prompting plus ensemble sampling) at F1=0.41; the majority-class baseline mean F1 was 0.02.
  • On the 6-feature synchrony dataset, performance was marginally higher but still limited: best F1=0.54 (GPT-5 Nano and GPT-OSS 120B with multi-task prompting); synchronous dancing and singing were easiest (F1>0.78), while synchronous marching was hardest (F1=0.43).
  • Human inter-coder reliability predicted LLM performance: mean human Cohen’s kappa was 0.57 (raw agreement 89%), and human kappa correlated with LLM F1 at r=0.61; LLMs agreed with each other far less than humans did (mean LLM-LLM κ=0.23 vs. human-human κ=0.57).
  • Error drivers included text length (longer excerpts reduced F1, β=-0.015, p<.001), feature type (multiclass features had 90% lower odds of correct detection than binary features, OR=0.10, 95% CI 0.03–0.35), and longer feature descriptions (β=-0.024, p<.001).

Methods

  • Corpus: 567 ethnographic texts (median=653 words, range=43–8,924) from 73 unique cultures, derived from eHRAF; two datasets: a 115-feature morphospace dataset (100 binary, 15 multi-class) and a 6-feature binary synchrony dataset with two independent human annotations per ritual.
  • Models and prompting: 7 LLMs (Qwen 3 Instruct 4B, Llama 3.2 Instruct 3B, GPT-OSS 120B, DeepSeek V3.1 671B, GPT-5 Nano, Claude Sonnet 4.5, and web-enabled Perplexity Sonar) evaluated with zero-shot baseline, multi-task prompting, and multi-task prompting plus ensemble sampling (mode of 10 repetitions), except Perplexity Sonar.
  • Analysis: precision, recall, and F1 against human ground truth; Cohen’s kappa for human-human and LLM-LLM agreement; nested mixed-effects models at ritual and prediction levels with model, text, and task predictors; Kruskal-Wallis tests for regional variation; and correlation between ensemble certainty and accuracy.

Limitations

  • Paper-stated: the study used zero-shot and multi-task prompting without extensive prompt engineering or structured codebooks with detailed examples; no models were fine-tuned; evaluation relied on a single resolved ground truth rather than soft or probabilistic labels; and other prompting strategies may prove more effective.
  • Evident from the text: raw ethnographic texts cannot be redistributed due to eHRAF licensing, and model predictions, human annotations, and analysis code are available only upon reasonable request; causes of regional performance differences remain unclear; some models refused to answer questions about sexual intercourse due to model constraints; proprietary model parameter counts are undisclosed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2601.12099

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Prompting strategy and ensembling improve LLM annotation performance across 115 morphospace features. Mean F1 score for each model under three conditions: baseline (zero-shot prompting), MTP (multi-task prompting), and MTP+Ensemble (majority voting across MTP runs). Error bars indicate 95% confidence intervals. MTP consistently outperforms baseline prompting, with ensembling providing additional gains for most models. Due to cost restrictions, only open-source models included ensembling.
Figure 1 Prompting strategy and ensembling improve LLM annotation performance across 115 morphospace features. Mean F1 score for each model under three conditions: baseline (zero-shot prompting), MTP (multi-task prompting), and MTP+Ensemble (majority voting across MTP runs). Error bars indicate 95% confidence intervals. MTP consistently outperforms baseline prompting, with ensembling providing additional gains for most models. Due to cost restrictions, only open-source models included ensembling.CC-BY · From the PDF
Figure 2 LLM performance across morphospace categories and synchrony features. (a) Heatmap showing mean F1 score for each model-condition combination (columns) across feature categories (rows). Categories are ordered by average performance. Duration, comprising a single feature, was evaluated under baseline conditions only as multi-turn prompting requires multi-feature iteration. Performance is highest for well-defined categorical features and lowest for subjective, context-dependent attributes and multi-option features. (b) Feature-level F1 scores for synchrony feature annotations across models and conditions. Features requiring identification of unambiguous coordinated actions (e.g., dancing and singing) show higher performance than those involving nuanced judgements about group coordination (e.g., generic synchronous movement). This pattern suggests that feature complexity partly drives classification difficulty.
Figure 2 LLM performance across morphospace categories and synchrony features. (a) Heatmap showing mean F1 score for each model-condition combination (columns) across feature categories (rows). Categories are ordered by average performance. Duration, comprising a single feature, was evaluated under baseline conditions only as multi-turn prompting requires multi-feature iteration. Performance is highest for well-defined categorical features and lowest for subjective, context-dependent attributes and multi-option features. (b) Feature-level F1 scores for synchrony feature annotations across models and conditions. Features requiring identification of unambiguous coordinated actions (e.g., dancing and singing) show higher performance than those involving nuanced judgements about group coordination (e.g., generic synchronous movement). This pattern suggests that feature complexity partly drives classification difficulty.CC-BY · From the PDF
Figure 3 Human inter-coder reliability predicts LLM performance. (a) Cohen's κ measuring agreement between two independent human coders for each synchrony feature. (b) Mean F1 score across all LLM models for the same features; error bars show the range across models. Colour indicates difficulty category based on human agreement: features that humans find difficult to code reliably (low κ ) also prove challenging for LLMs, suggesting inherent ambiguity in the ethnographic texts and/or task rather than model-specific limitations.
Figure 3 Human inter-coder reliability predicts LLM performance. (a) Cohen's κ measuring agreement between two independent human coders for each synchrony feature. (b) Mean F1 score across all LLM models for the same features; error bars show the range across models. Colour indicates difficulty category based on human agreement: features that humans find difficult to code reliably (low κ ) also prove challenging for LLMs, suggesting inherent ambiguity in the ethnographic texts and/or task rather than model-specific limitations.CC-BY · From the PDF
Figure 4 Feature-level F1 performance on all 115 ritual features. RitualDuration was a single-feature category and thus only the baseline condition could be evaluated. Both Claude Sonnet 4.5 and Perplexity Sonar refused to answer questions pertaining to sexual intercourse due to model constraints.
Figure 4 Feature-level F1 performance on all 115 ritual features. RitualDuration was a single-feature category and thus only the baseline condition could be evaluated. Both Claude Sonnet 4.5 and Perplexity Sonar refused to answer questions pertaining to sexual intercourse due to model constraints.CC-BY · From the PDF
Figure 5 LLM-LLM agreement heatmap. Mean Cohen's κ is reported across all model pairs.
Figure 5 LLM-LLM agreement heatmap. Mean Cohen's κ is reported across all model pairs.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 7 Model certainty is a good predictor of annotation accuracy.
Figure 7 Model certainty is a good predictor of annotation accuracy.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 0 Not valid · 11 Not checked · 133 Own results and interpretations

59 more claims

Create a free account to see all 144 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 54
  • Probably not a substantial claim. 37
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 6 · the rest aren’t loaded

1 more finding

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 59 references

Research works printed but not linked: 6

  1. Alec Radford, Jeff Wu, R. Child, D. Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.
  2. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. Scaling Lan- guage Models: Methods, Analysis & Insights from Training Gopher, January 2022. arXiv:2112.11446 [cs]. · arXiv
  3. Grzegorz Bryda and António Pedro Costa. Qualita- tive Research in Digital Era: Innovations, Methodolo- gies and Collaborations. Social Sciences, 12(10):570, October 2023.
  4. Alfred Gell. Art and agency: an anthropological theory. Clarendon Press, Oxford, repr edition, 2013.
  5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah-[64] OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Ar- bus, Rahul K. Arora, Yu Bai, Bowen Baker, Haim- ing Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach John- son, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guil- laume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambat- tista Parascandolo, Jongsoo Park, Leher Pathak, Car- olina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Su- vansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wal- lace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whit- ney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss-120b & gpt-oss-20b Model Card, August 2025. arXiv:2508.10925 [cs]. · arXiv
  6. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingx- uan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xi- aojin Shen, Xiaokang Chen, Xiaokang Zhang, Xi- aosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yao- hui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yud- uan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yux- iang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. DeepSeek-V3 Technical Report, February 2025. arXiv:2412.19437 [cs]. · arXiv

Cited by

See all 3

Metadata

Authors
Goodall, L. S., Shilton, D., Mullins, D. A., & Whitehouse, H.
Affiliations
  • University of Oxford (GB)
  • Tel Aviv University (IL)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
OpenAlex
W7125097666
Cited by
3
References
56
Citations by year
2026: 3
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
Keywords
large language models, ethnographic texts, intercoder reliability, cross-cultural research, text length effects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large language models struggle with ethnographic text annotation. https://app.cochat.ai/papers/10.48550/arxiv.2601.12099
The paper
Goodall, L. S., Shilton, D., Mullins, D. A., & Whitehouse, H. (2026). Large language models struggle with ethnographic text annotation. arXiv. https://doi.org/10.48550/arxiv.2601.12099

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.