CoChat

The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy

Open access
Do, Salomé, Ollion, Étienne, Shen, Rubing · 2024 · Sociological Methods & Research
Cited by 54

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Abstract

The last decade witnessed a spectacular rise in the volume of available textual data. With this new abundance came the question of how to analyze it. In the social sciences, scholars mostly resorted to two well-established approaches, human annotation on sampled data on the one hand (either performed by the researcher, or outsourced to microworkers), and quantitative methods on the other. Each approach has its own merits - a potentially very fine-grained analysis for the former, a very scalable one for the latter - but the combination of these two properties has not yielded highly accurate results so far. Leveraging recent advances in sequential transfer learning, we demonstrate via an experiment that an expert can train a precise, efficient automatic classifier in a very limited amount of time. We also show that, under certain conditions, expert-trained models produce better annotations than humans themselves. We demonstrate these points using a classic research question in the sociology of journalism, the rise of a “horse race” coverage of politics. We conclude that recent advances in transfer learning help us augment ourselves when analyzing unstructured data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/00491241221134526

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 41

  1. 10.1177/1464884911427799 · DOI · Paper page
  2. 10.1017/pan.2020.8 · DOI · Paper page
  3. 10.1177/2053951715612497 · DOI
  4. 10.1017/S0003055416000058 · DOI · Paper page
  5. Berelson Bernard, 1948, The Analysis of Communication Content
  6. Blei David M., 2003, Journal of Machine Learning Research, 3, 993
  7. 10.1086/268620 · doi:10.1086/268620
  8. Brown Tom, 2020, Advances in Neural Information Processing Systems, 33, 1877
  9. Casilli Antonio, 2019, En Attendant les Robots: Enquêtes sur le travail du clic
  10. Chang Jonathan, Boyd-Graber Jordan, Gerrish Sean, Wang Chong, Blei David M. 2009. “Reading Tea Leaves: how Humans Interpret Topic Model.” Pp. 288–96 in Proceedings of the 22nd International Conference on Neural Information Processing Systems. New York: Curran Associates Inc.
  11. 10.3917/rfs.593.0533 · DOI
  12. 10.21236/AD0289565 · DOI · Paper page
  13. Farnsworth Stephen J., 2011, The Nightly News Nightmare: Media Coverage of U.S. Presidential Elections. 1988–2008
  14. 10.1002/9781119306696 · DOI
  15. 10.1257/jel.20181020 · DOI · Paper page
  16. 10.1145/3117807 · DOI
  17. 10.1177/0049124117729705 · doi:10.1177/0049124117729705
  18. 10.1109/MSP.2012.2205597 · DOI · Paper page
  19. 10.31235/osf.io/ft84u · DOI
  20. Kaciaf Nicolas. 2005. Les métamorphoses des pages politiques dans la presse française (1945–2000). Doctoral thesis, Université Paris-1 Panthéon Sorbonne, Paris, France.
  21. 10.4000/books.pur.71409 · doi:10.4000/books.pur.71409
  22. 10.1007/978-94-017-1525-6 · DOI · Paper page
  23. 10.1145/1557019.1557077 · DOI · Paper page
  24. Littlewood Thomas, 1998, Calling Elections: The History of Horse-Race Journalism
  25. 10.18653/v1/2020.findings-emnlp.296 · DOI
  26. Martin Louis, Muller Benjamin, Suárez Pedro Javier Ortiz, Dupont Yoann, Romary Laurent, de la Clergerie Éric, Seddah Djamé, Sagot Benoît. 2020. “CamemBERT: a Tasty French Language Model.” Pp. 7203–19 in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics. · DOI · Paper page
  27. 10.1057/s41599-021-00815-9 · DOI
  28. 10.18653/v1/2021.naacl-main.179 · DOI
  29. Mohr John W., 2020, Measuring Culture
  30. Moretti Franco, 2013, Distant Reading
  31. 10.1177/0049124118769114 · DOI · Paper page
  32. Patterson Thomas, 1994, Out of Order: An Incisive and Boldly Original Critique of the News Media’s Domination of America’s Political Process
  33. 10.1002/sim.1253 · DOI · Paper page
  34. Ruder Sebastian. 2019. Neural Transfer Learning for Natural Language Processing. Doctoral thesis, National University of Ireland, Galway, Ireland.
  35. Schreibman Susan, 2004, A Companion to Digital Humanities
  36. Schumacher Tobias, Strohmaier Markus, Lemmerich Florian. 2021. “A Comparative Evaluation of Quantification Methods” Retrieved February 15, 2021 (https://arxiv.org/abs/2103.03223). · arXiv
  37. Stone Philip J., 1966, The General Inquirer: A Computer Approach to Content Analysis
  38. Tratz Stephen, Hovy Eduard. 2010. “A Taxonomy, Dataset, and Classifier for Automatic Noun Compound Interpretation.” Pp. 678–87 in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Uppsala, Sweden: Association for Computational Linguistics.
  39. Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Łukasz, Polosukhin Illia. 2017. “Attention is All you Need.” Pp. 6000–10 in Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, California: Curran Associates Inc.
  40. 10.1177/0038038519828906 · DOI · Paper page
  41. You Yang, Li Jing, Reddi Sashank, Hseu Jonathan, Kumar Sanjiv, Bhojanapalli Srinadh, Song Xiaodan, Demmel James, Keutzer Kurt, Hsieh Cho-Jui. 2020. “Large Batch Optimization for Deep Learning: Training BERT in 76 min.” Retrieved June 12, 2021 (https://arxiv.org/abs/1904.00962). · arXiv

Cited by

See all 54

Metadata

Authors
Do, S., Ollion, É., & Shen, R.
Venue
Sociological Methods & Research
Published
2022-12-04
Volume
53
Issue
3
Pages
1167-1200
ISSN
  • 0049-1241 (ISSN-L)
  • 1552-8294
OpenAlex
W4311510002
Cited by
54
References
13
FWCI
24.54
Type
journal article
Language
en
Open access
gold
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Funders
  • Svenska Vetenskapradet (Mining for Meaning 2018-05170)
  • Labex (ANR-11-LABX-0047)
  • Agence Nationale de la Recherche (ANR-19-P3IA-0001)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy. https://app.cochat.ai/papers/10.1177/00491241221134526
The paper
Do, S., Ollion, É., & Shen, R. (2024). The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy. Sociological Methods & Research. https://doi.org/10.1177/00491241221134526

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.