Large language models struggle with ethnographic text annotation
Open accessThe DOI matches its DataCite record; 59 of the 59 references it cites were found in a registry or an index; none of the 18 sources its claims cite is checked for support yet.
Abstract
Large language models (LLMs) have shown promise for automated text annotation, raising hopes that they might accelerate cross-cultural research by extracting structured data from ethnographic texts. We evaluated 7 state-of-the-art LLMs on their ability to annotate 121 ritual features across 567 ethnographic excerpts. Performance was limited, falling well below levels required for reliable automated annotation. Longer texts, features requiring ordinal distinctions, and ambiguous constructs proved particularly difficult. Human inter-coder reliability set an approximate ceiling on LLM accuracy: features that human coders found difficult to agree upon were also difficult for LLMs. Yet even on features where humans reliably agreed, models fell short of human performance. Our findings suggest that LLMs cannot yet substitute for human expertise in ethnographic annotation.
Overview
CoChat Summary
TL;DR
Evaluating 7 state-of-the-art LLMs on 121 ritual features across 567 ethnographic excerpts, the best configuration reached only F1=0.41 on the larger morphospace dataset and F1=0.54 on the smaller synchrony dataset. The authors conclude LLMs cannot yet substitute for human expertise in ethnographic annotation.
Key findings
- Across all models and prompting conditions, morphospace annotation was poor: mean F1 ranged from 0.12 to 0.41, with the best configuration (DeepSeek V3.1 and GPT-OSS 120B under multi-task prompting plus ensemble sampling) at F1=0.41; the majority-class baseline mean F1 was 0.02.
- On the 6-feature synchrony dataset, performance was marginally higher but still limited: best F1=0.54 (GPT-5 Nano and GPT-OSS 120B with multi-task prompting); synchronous dancing and singing were easiest (F1>0.78), while synchronous marching was hardest (F1=0.43).
- Human inter-coder reliability predicted LLM performance: mean human Cohen’s kappa was 0.57 (raw agreement 89%), and human kappa correlated with LLM F1 at r=0.61; LLMs agreed with each other far less than humans did (mean LLM-LLM κ=0.23 vs. human-human κ=0.57).
- Error drivers included text length (longer excerpts reduced F1, β=-0.015, p<.001), feature type (multiclass features had 90% lower odds of correct detection than binary features, OR=0.10, 95% CI 0.03–0.35), and longer feature descriptions (β=-0.024, p<.001).
Methods
- Corpus: 567 ethnographic texts (median=653 words, range=43–8,924) from 73 unique cultures, derived from eHRAF; two datasets: a 115-feature morphospace dataset (100 binary, 15 multi-class) and a 6-feature binary synchrony dataset with two independent human annotations per ritual.
- Models and prompting: 7 LLMs (Qwen 3 Instruct 4B, Llama 3.2 Instruct 3B, GPT-OSS 120B, DeepSeek V3.1 671B, GPT-5 Nano, Claude Sonnet 4.5, and web-enabled Perplexity Sonar) evaluated with zero-shot baseline, multi-task prompting, and multi-task prompting plus ensemble sampling (mode of 10 repetitions), except Perplexity Sonar.
- Analysis: precision, recall, and F1 against human ground truth; Cohen’s kappa for human-human and LLM-LLM agreement; nested mixed-effects models at ritual and prediction levels with model, text, and task predictors; Kruskal-Wallis tests for regional variation; and correlation between ensemble certainty and accuracy.
Limitations
- Paper-stated: the study used zero-shot and multi-task prompting without extensive prompt engineering or structured codebooks with detailed examples; no models were fine-tuned; evaluation relied on a single resolved ground truth rather than soft or probabilistic labels; and other prompting strategies may prove more effective.
- Evident from the text: raw ethnographic texts cannot be redistributed due to eHRAF licensing, and model predictions, human annotations, and analysis code are available only upon reasonable request; causes of regional performance differences remain unclear; some models refused to answer questions about sexual intercourse due to model constraints; proprietary model parameter counts are undisclosed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
0 Valid · 0 Not valid · 11 Not checked · 133 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All models across both datasets outperformed a majority-class baseline classifier, which always predicts the most frequent class for each feature.”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > LLMs show poor annotation performance for ethnographic texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among task factors, longer feature descriptions were associated with poorer performance (β=-0.024,p <.001).”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Feature complexity and text length drive errors
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, LLMs did not fully reach the human reliability ceiling for the easier features.”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Human agreement sets a ceiling on LLM performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For synchronous singing, where humans achieved near-perfect agreement (κ=0.92), the best-performing LLM (Claude Sonnet 4.5) reached F1=0.75.”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Human agreement sets a ceiling on LLM performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ensemble sampling (taking the mode of 10 repetitions) provided additional but diminishing returns, typically adding 0.01 to 0.02 points over multi-task prompting alone.”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > LLMs show poor annotation performance for ethnographic texts
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
59 more claims
Create a free account to see all 144 claims and every source verdict.
- It may state two separate claims as one. 54
- Probably not a substantial claim. 37
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > LLMs show poor annotation performance for ethnographic texts
“The mean baseline F1 across all features for the first, larger dataset was 0.02, compared to 0.47 for the best LLM configuration.”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Discussion
“Across 7 models, 3 prompting strategies, and 115 ritual features, the best-performing configuration achieved an F1 score of only 0.41.”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Feature complexity and text length drive errors
“with North America showing a similar but non-significant trend (β=-0.030,p=.069).”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Mixed-effects regression models
“North America and Oceania showed significantly lower F1 scores than the reference region (p <.02).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Human agreement sets a ceiling on LLM performance
“For synchronous dancing (κ=0.79),however, the best LLM matched human reliability (F1=0.79)”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Accuracy Increases with Model Certainty
“| Sync. Dancing | GPT-OSS 120B | Ensemble MTP .809 .905 .731 | | |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > LLMs show poor annotation performance for ethnographic texts
“multi-task prompting (annotate all features in a predefined category at once) improved F1 scores by 0.02 to 0.07 points for most models, with DeepSeek V3.1 showing the largest gain.”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Accuracy Increases with Model Certainty
“| Claude Sonnet 4.5 | MTP baseline | .394 .326 | .328 .249 | .658 .764 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Regional analysis of model performance
“Perplexity Sonar displayed a distinct pattern: Europe significantly outperformed all other regions”
§ LARGE LANGUAGE MODELS STRUGGLE WITH ETHNOGRAPHIC TEXT ANNOTATION > Accuracy Increases with Model Certainty
“| | Africa < Europe | <.001 | | | Asia < Europe | <.001 | | Perplexity Sonar | M. America & Carib < Europe N. America < Europe | .002 .019 | | | Oceania < Europe | <.0001 | | | S. America < Europe | <.0001 |”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
McCarthy, P. M., & Jarvis, S. (2010) · Behavior Research Methods · doi:10.3758/brm.42.2.381 · Cited by 905
Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2302.06476 · Cited by 869
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707
Computational Grounded Theory: A Methodological Framework
Nelson, L. K. (2020) · Sociological Methods & Research · doi:10.1177/0049124117729703 · Cited by 697
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
Research works printed but not linked: 6
- Alec Radford, Jeff Wu, R. Child, D. Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.
- Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. Scaling Lan- guage Models: Methods, Analysis & Insights from Training Gopher, January 2022. arXiv:2112.11446 [cs]. · arXiv
- Grzegorz Bryda and António Pedro Costa. Qualita- tive Research in Digital Era: Innovations, Methodolo- gies and Collaborations. Social Sciences, 12(10):570, October 2023.
- Alfred Gell. Art and agency: an anthropological theory. Clarendon Press, Oxford, repr edition, 2013.
- Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah-[64] OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Ar- bus, Rahul K. Arora, Yu Bai, Bowen Baker, Haim- ing Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach John- son, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guil- laume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambat- tista Parascandolo, Jongsoo Park, Leher Pathak, Car- olina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Su- vansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wal- lace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whit- ney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss-120b & gpt-oss-20b Model Card, August 2025. arXiv:2508.10925 [cs]. · arXiv
- DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingx- uan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xi- aojin Shen, Xiaokang Chen, Xiaokang Zhang, Xi- aosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yao- hui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yud- uan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yux- iang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. DeepSeek-V3 Technical Report, February 2025. arXiv:2412.19437 [cs]. · arXiv
Cited by
Observational Equivalence of LLM and Human Annotation *Preprint
Nakamura, K., Tan, J. L., & Yean, G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7389359
Can Large Language Models Replace Human Coders? Introducing ContentBenchPreprint
Haman, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.19467
Knowing Is Not Telling: Identity Ossification in LLM Representations of Trans-border CommunitiesPreprint
Sun, M., Chen, L., & Jiang, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.12610
Metadata
- Authors
- Goodall, L. S., Shilton, D., Mullins, D. A., & Whitehouse, H.
- Affiliations
- University of Oxford (GB)
- Tel Aviv University (IL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2601.12099
- OpenAlex
- W7125097666
- Cited by
- 3
- References
- 56
- Citations by year
- 2026: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- large language models, ethnographic texts, intercoder reliability, cross-cultural research, text length effects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large language models struggle with ethnographic text annotation. https://app.cochat.ai/papers/10.48550/arxiv.2601.12099
- The paper
- Goodall, L. S., Shilton, D., Mullins, D. A., & Whitehouse, H. (2026). Large language models struggle with ethnographic text annotation. arXiv. https://doi.org/10.48550/arxiv.2601.12099
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.