The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
The last decade witnessed a spectacular rise in the volume of available textual data. With this new abundance came the question of how to analyze it. In the social sciences, scholars mostly resorted to two well-established approaches, human annotation on sampled data on the one hand (either performed by the researcher, or outsourced to microworkers), and quantitative methods on the other. Each approach has its own merits - a potentially very fine-grained analysis for the former, a very scalable one for the latter - but the combination of these two properties has not yielded highly accurate results so far. Leveraging recent advances in sequential transfer learning, we demonstrate via an experiment that an expert can train a precise, efficient automatic classifier in a very limited amount of time. We also show that, under certain conditions, expert-trained models produce better annotations than humans themselves. We demonstrate these points using a classic research question in the sociology of journalism, the rise of a “horse race” coverage of politics. We conclude that recent advances in transfer learning help us augment ourselves when analyzing unstructured data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“In our case, tokens are subword units encoded with WordPiece (Wu et al. 2016).”
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297
How Censorship in China Allows Government Criticism but Silences Collective Expression
KING, G., PAN, J., & ROBERTS, M. E. (2013) · American Political Science Review · doi:10.1017/s0003055413000014 · Cited by 2,415 · cited 1×
“See among others (Barberá et al. 2021; King, Pan, and Roberts 2013).”
Research works printed but not linked: 41
- 10.1177/1464884911427799 · DOI · Paper page
- 10.1017/pan.2020.8 · DOI · Paper page
- 10.1177/2053951715612497 · DOI
- 10.1017/S0003055416000058 · DOI · Paper page
- Berelson Bernard, 1948, The Analysis of Communication Content
- Blei David M., 2003, Journal of Machine Learning Research, 3, 993
- 10.1086/268620 · doi:10.1086/268620
- Brown Tom, 2020, Advances in Neural Information Processing Systems, 33, 1877
- Casilli Antonio, 2019, En Attendant les Robots: Enquêtes sur le travail du clic
- Chang Jonathan, Boyd-Graber Jordan, Gerrish Sean, Wang Chong, Blei David M. 2009. “Reading Tea Leaves: how Humans Interpret Topic Model.” Pp. 288–96 in Proceedings of the 22nd International Conference on Neural Information Processing Systems. New York: Curran Associates Inc.
- 10.3917/rfs.593.0533 · DOI
- 10.21236/AD0289565 · DOI · Paper page
- Farnsworth Stephen J., 2011, The Nightly News Nightmare: Media Coverage of U.S. Presidential Elections. 1988–2008
- 10.1002/9781119306696 · DOI
- 10.1257/jel.20181020 · DOI · Paper page
- 10.1145/3117807 · DOI
- 10.1177/0049124117729705 · doi:10.1177/0049124117729705
- 10.1109/MSP.2012.2205597 · DOI · Paper page
- 10.31235/osf.io/ft84u · DOI
- Kaciaf Nicolas. 2005. Les métamorphoses des pages politiques dans la presse française (1945–2000). Doctoral thesis, Université Paris-1 Panthéon Sorbonne, Paris, France.
- 10.4000/books.pur.71409 · doi:10.4000/books.pur.71409
- 10.1007/978-94-017-1525-6 · DOI · Paper page
- 10.1145/1557019.1557077 · DOI · Paper page
- Littlewood Thomas, 1998, Calling Elections: The History of Horse-Race Journalism
- 10.18653/v1/2020.findings-emnlp.296 · DOI
- Martin Louis, Muller Benjamin, Suárez Pedro Javier Ortiz, Dupont Yoann, Romary Laurent, de la Clergerie Éric, Seddah Djamé, Sagot Benoît. 2020. “CamemBERT: a Tasty French Language Model.” Pp. 7203–19 in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics. · DOI · Paper page
- 10.1057/s41599-021-00815-9 · DOI
- 10.18653/v1/2021.naacl-main.179 · DOI
- Mohr John W., 2020, Measuring Culture
- Moretti Franco, 2013, Distant Reading
- 10.1177/0049124118769114 · DOI · Paper page
- Patterson Thomas, 1994, Out of Order: An Incisive and Boldly Original Critique of the News Media’s Domination of America’s Political Process
- 10.1002/sim.1253 · DOI · Paper page
- Ruder Sebastian. 2019. Neural Transfer Learning for Natural Language Processing. Doctoral thesis, National University of Ireland, Galway, Ireland.
- Schreibman Susan, 2004, A Companion to Digital Humanities
- Schumacher Tobias, Strohmaier Markus, Lemmerich Florian. 2021. “A Comparative Evaluation of Quantification Methods” Retrieved February 15, 2021 (https://arxiv.org/abs/2103.03223). · arXiv
- Stone Philip J., 1966, The General Inquirer: A Computer Approach to Content Analysis
- Tratz Stephen, Hovy Eduard. 2010. “A Taxonomy, Dataset, and Classifier for Automatic Noun Compound Interpretation.” Pp. 678–87 in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Uppsala, Sweden: Association for Computational Linguistics.
- Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Łukasz, Polosukhin Illia. 2017. “Attention is All you Need.” Pp. 6000–10 in Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, California: Curran Associates Inc.
- 10.1177/0038038519828906 · DOI · Paper page
- You Yang, Li Jing, Reddi Sashank, Hseu Jonathan, Kumar Sanjiv, Bhojanapalli Srinadh, Song Xiaodan, Demmel James, Keutzer Kurt, Hsieh Cho-Jui. 2020. “Large Batch Optimization for Deep Learning: Training BERT in 76 min.” Retrieved June 12, 2021 (https://arxiv.org/abs/1904.00962). · arXiv
Cited by
Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models
Than, N., Fan, L., Law, T., Nelson, L. K., & McCall, L. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339188 · Cited by 63
“Few-shot learning thus might work better for short-text (e.g., sentence-or paragraph-length) classi fi cation tasks (e.g., Chae and Davidson 2025; Do, Ollion, and Shen 2022; Rytting et al. 2023) than for ones such as ours.”
InfluentialThe dangers of using proprietary LLMs for research
Ollion, É., Shen, R., Macanovic, A., & Chatelain, A. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00783-6 · Cited by 62
Self-citationNavigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Lin, H., & Zhang, Y. (2026) · Social Science Computer Review · doi:10.1177/08944393251366243 · Cited by 33
Embedding Democratic Values into Social Media AIs via Societal Objective Functions
Jia, C., Lam, M. S., Mai, M. C., Hancock, J. T., & Bernstein, M. S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641002 · Cited by 31
Metadata
- Authors
- Do, S., Ollion, É., & Shen, R.
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2022-12-04
- Volume
- 53
- Issue
- 3
- Pages
- 1167-1200
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W4311510002
- Cited by
- 54
- References
- 13
- FWCI
- 24.54
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Funders
- Svenska Vetenskapradet (Mining for Meaning 2018-05170)
- Labex (ANR-11-LABX-0047)
- Agence Nationale de la Recherche (ANR-19-P3IA-0001)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy. https://app.cochat.ai/papers/10.1177/00491241221134526
- The paper
- Do, S., Ollion, É., & Shen, R. (2024). The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy. Sociological Methods & Research. https://doi.org/10.1177/00491241221134526
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.