Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
The increasing capacities of large language models (LLMs) have been shown to present an unprecedented opportunity to scale up data analytics in the humanities and social sciences, by automating complex qualitative tasks otherwise typically carried out by human researchers. While numerous benchmarking studies have assessed the analytic prowess of LLMs, there is less focus on operationalizing this capacity for inference and hypothesis testing. Addressing this challenge, a systematic framework is argued for here, building on mixed methods quantitizing and converting design principles, and feature analysis from linguistics, to transparently integrate human expertise and machine scalability. Replicability and statistical robustness are discussed, including how to incorporate machine annotator error rates in subsequent inference. The approach is discussed and demonstrated in over a dozen LLM-assisted case studies, covering nine diverse languages, multiple disciplines and tasks, including analysis of themes, stances, ideas, and genre compositions; linguistic and semantic annotation, interviews, text mining and event cause inference in noisy historical data, literary social network construction, metadata imputation, and multimodal visual cultural analytics. Using hypothesis-driven topic classification instead of “distant reading” is discussed. The replications among the experiments also illustrate how tasks previously requiring protracted team effort or complex computational pipelines can now be accomplished by an LLM-assisted scholar in a fraction of the time. Importantly, the approach is not intended to replace, but to augment and scale researcher expertise and analytic practices. With these opportunities in sight, qualitative skills and the ability to pose insightful questions have arguably never been more critical.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This has led to their increasing adoption in social sciences where qualitative text analysis has traditionally been the dominant paradigm (see Karjus, 2025;Linegar et al., 2023).”
From Using large language models to analyze political texts through natural language understanding (2026) · cites this paper as Karjus, 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Tuning or adaptation approaches (Dettmers et al. 2023; Hu et al. 2021) can still be more ef fi cient on larger or repeated tasks.”
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Braun, V., & Clarke, V. (2012) · APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. · doi:10.1037/13620-004 · Cited by 10,299
Hubert, L., & Arabie, P. (1985) · Journal of Classification · doi:10.1007/bf01908075 · Cited by 8,219
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1×
“In an educational setting, LLMs may be useful for assessment and other tasks (Baidoo-Anu and Owusu Ansah 2023; Kasneci et al. 2023).”
Achieving integration in mixed methods designs-principles and practices
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013) · Health Services Research · doi:10.1111/1475-6773.12117 · Cited by 7,219 · cited 1×
““ Quantitization ” or “ quantizing ” (Fetters et al. 2013; Hesse-Biber 2010; Sandelowski et al. 2009) is a useful term to distinguish the process of annotating data with the explicit purpose of subsequent quanti fi cation — from annotation for any other purposes.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“These issues have been and will be discussed elsewhere (Bender et al. 2021; Binz and Schulz 2023; Rooij et al. 2024; Feng et al. 2023; Hagendorff et al. 2024; Liesenfeld et al. 2023; Lund et al. 2023; Motoki et al. 2023; Novozhilova et al. 2024; Ollion et al. 2024; Tomlinson et al. 2024).”
Research works printed but not linked: 129
- Aiyappa R, An J, Kwak H, Ahn Y-Y (2023) Can we trust the evaluation on ChatGPT? In: Proceedings of the 3rd workshop on trustworthy natural language processing (TrustNLP 2023). Association for Computational Linguistics, pp 47–54 · DOI · Paper page
- Angelov D (2020) Top2Vec: distributed representations of topics. Preprint at https://doi.org/10.48550/arXiv.2008.09470 · DOI
- Asadchy Y, Karjus A, Mukhina K, Schich M (2024) Perceived gendered self-representation on Tinder using machine learning. Humanit Soc Sci Commun 11(1):1–11 · DOI
- Ascari M (2014) The dangers of distant reading: reassessing Moretti’s approach to literary genres. Genre 47(1):1–19 · doi:10.1215/00166928-2392348
- Ash E, Gauthier G, Widmer P (2024) Relatio: text semantics capture political and economic narratives. Political Anal 32(1):115–132 · DOI
- Asher N, Bhar S, Chaturvedi A, Hunter J, Paul S (2023) Limits for learning with language models. In: Proceedings of the 12th joint conference on lexical and computational semantics (*SEM 2023). Association for Computational Linguistics, pp 236–248 · DOI
- Banha F, Flores A, Coelho LS (2022) Quantitizing qualitative data from semi-structured interviews: a methodological contribution in the context of public policy decision-making. Mathematics 10(19):3597 · DOI
- Beguš G, Dąbkowski M, Rhodes R (2023) Large linguistic models: analyzing theoretical linguistic abilities of LLMs. Preprint at https://doi.org/10.48550/arXiv.2305.00948 · DOI
- Beuls K, Van Eecke P (2025) Construction grammar and artificial intelligence. In: Fried M, Nikiforidou K (eds) The Cambridge handbook of construction grammar. Cambridge University Press, pp 543–571 · DOI
- Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022
- Boros E, Ehrmann M, Romanello M, Najem-Meyer S, Frédéric K (eds) (2024) Post-correction of historical text transcripts with large language models: an exploratory study. In: Proceedings of the 8th joint SIGHUM workshop on computational linguistics for cultural heritage, social sciences, humanities and literature (LaTeCH-CLfL 2024). pp 133–159
- Buscemi A, Proverbio D (2024) ChatGPT vs Gemini vs LLaMA on multilingual sentiment analysis. Preprint at https://doi.org/10.48550/arXiv.2402.01715 · DOI
- Carroll RJ, Ruppert D, Stefanski, LA, Crainiceanu CM (2006) Measurement error in nonlinear models: a modern perspective, second edition. CRC Press · DOI · Paper page
- Chaturvedi S, Srivastava S, Roth D (2018) Where have I heard this story before? Identifying narrative similarity in movie remakes. In: Proceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, vol 2 (short papers). Association for Computational Linguistics, pp 673–678 · doi:10.18653/v1/n18-2106
- Chen W, Li Y, Tian Z, Zhang F (2023) 2D and 3D object detection algorithms from images: a survey. Array 19:100305 · DOI
- Chi N, Malchev T, Kong R, Chi R, Huang L, Chi E et al. (2024) ModeLing: a novel dataset for testing linguistic reasoning in language models. In: Hahn M, Sorokin A, Kumar R, Shcherbakov A, Otmakhova Y, Yang J et al. (eds) Proceedings of the 6th workshop on research in computational linguistic typology and multilingual NLP. Association for Computational Linguistics, pp 113–119
- Clark TS, Linzer DA (2015) Should I use fixed or random effects? Political Sci Res Methods 3(2):399–408 · DOI · Paper page
- Creamer EG (2018) An introduction to fully integrated mixed methods research. SAGE Publications, Inc · DOI · Paper page
- De Paoli S (2023) Performing an inductive thematic analysis of semi-structured interviews with a large language model: an exploration and provocation on the limits of the approach. Soc Sci Comput Rev 42(4):997–1019 · DOI · Paper page
- de la Rosa J, Pérez Pozo A, Ros S, González-Blanco E (2023) ALBERTI, a multilingual domain specific language model for poetry analysis. Preprint at http://arxiv.org/abs/2307.01387 · arXiv
- Demszky D, Yang D, Yeager DS, Bryan CJ, Clapper M, Chandhok S (2023) Using large language models in psychology. Nat Rev Psychol 2(11):688–701
- Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L (2023) QLoRA: efficient finetuning of quantized LLMs. Preprint at http://arxiv.org/abs/2305.14314 · arXiv
- Dinh T, Zhao J, Tan S, Negrinho R, Lausen L, Zha S et al. (2023) Large language models of code fail at completing code with potential bugs. Preprint at http://arxiv.org/abs/2306.03438 · arXiv
- Dubossarsky H, Hengchen S, Tahmasebi N, Schlechtweg D (2019) Time-out: temporal referencing for robust modeling of lexical semantic change. In: Proceedings of the 57th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 457–470 · DOI
- Elson D, Dames N, McKeown K (2010) Extracting social networks from literary fiction. In: Proceedings of the 48th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 138–147
- Erscoi L, Kleinherenbrink A, Guest O (2023) Pygmalion displacement: When humanising AI dehumanises women. Preprint at https://doi.org/10.31235/osf.io/jqxb6 · DOI
- Fan Y, Jiang F, Li P, Li H (2024) Uncovering the potential of ChatGPT for discourse analysis in dialogue: an empirical study. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). Association for Computational Linguistics, pp 16998–17010
- Fofana F, Bazeley P, Regnault A (2020) Applying a mixed methods design to test saturation for qualitative data in health outcomes research. PLoS ONE 15(6):e0234898 · DOI
- Fonteyn L (2021) Varying abstractions: a conceptual vs. distributional view on prepositional polysemy. Glossa J Gen Linguist 6(1):90
- Fudolig MI, Alshaabi T, Cramer K, Danforth CM, Dodds PS (2023) A decomposition of book structure through ousiometric fluctuations in cumulative word-time. Humanit Soc Sci Commun 10:187 · DOI
- Futrell R, Wilcox E, Morita T, Qian P, Ballesteros M, Levy R (2019) Neural language models as psycholinguistic subjects: representations of syntactic state. In: Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, volume 1 (long and short papers). Association for Computational Linguistics, pp 32–42 · DOI · Paper page
- Galantucci B, Garrod S, Roberts G (2012) Experimental semiotics. Lang Linguist Compass 6(8):477–493 · DOI · Paper page
- Gienapp L, Kircheis W, Sievers B, Stein B, Potthast M (2023) A large dataset of scientific text reuse in Open-Access publications. Sci Data 10(1):58 · DOI
- Glynn D (ed) (2010) Testing the hypothesis: objectivity and verification in usage-based cognitive semantics. In: Quantitative methods in cognitive semantics: corpus-driven approaches. De Gruyter Mouton, pp 239–269 · DOI
- Glynn D, Fischer K (2010) Quantitative methods in cognitive semantics: corpus-driven approaches. Walter de Gruyter, p 404 · doi:10.1515/9783110226423
- Goldstein H, Kounali D, Robinson A (2008) Modelling measurement errors and category misclassifications in multilevel models. Stat Model 8:243–261 · DOI
- González P, Díez J, Chawla N, Del Coz JJ (2017) Why is quantification an interesting learning problem? Prog Artif Intell 6(1):53–58 · doi:10.1007/s13748-016-0103-3
- Goodenough WH (1956) Componential analysis and the study of meaning. Language 32(1):195–216 · DOI · Paper page
- Gries ST (2015) The most under-used statistical method in corpus linguistics: multi-level (and mixed-effects) models. Corpora 10(1):95–125 · DOI · Paper page
- Gries ST, Divjak D (2009) A corpus-based approach to cognitive semantic analysis: behavioral profiles. In: Evans V, Pourcel S (eds) New directions in cognitive linguistics, human cognitive processing. John Benjamins Publishing Company, pp 57–75 · DOI
- Guillory D, Shankar V, Ebrahimi S, Darrell T, Schmidt L (2021) Predicting with confidence on unseen distributions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp 1134–1144 · DOI
- Gulordava K, Baroni M (2011) A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In: Proceedings of the GEMS 2011 workshop on geometrical models of natural language semantics. Association for Computational Linguistics, pp 67–71
- Hamilton WL, Leskovec J, Jurafsky D (2016) Diachronic word embeddings reveal statistical laws of semantic change. In: Proceedings of the 54th annual meeting of the Association for Computational Linguistics, ACL 2016, August 7–12, 2016, Berlin, Germany, vol 1: long papers, pp 1489–1501 · DOI · Paper page
- Hau MF (2024) A practice-oriented framework for using large language model-powered chatbots. Acta Sociol https://doi.org/10.1177/00016993241264152 · DOI
- Hesse-Biber SN (2010) Mixed methods research: merging theory with practice. Guilford Press, New York, p 242
- Hrastinski S, Aghaee NM (2012) How are campus students using social media to support their studies? An explorative interview study. Educ Inf Technol 17(4):451–464 · DOI · Paper page
- Hsieh H-F, Shannon S (2005) Three approaches to qualitative content analysis. Qual Health Res 15:1277–88 · DOI · Paper page
- Huang F, Kwak H, An J (2023) Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech. In: Companion Proceedings of the ACM web conference 2023, WWW '23 companion. Association for Computing Machinery, New York, NY, USA, pp 294–297 · DOI · Paper page
- Inggs J (2011) Censorship and translated children’s literature in the Soviet Union: the example of the Wizards Oz and Goodwin. Target 23:77–91 · DOI
- Jacobs T, Tschötschel R (2019) Topic models meet discourse analysis: a quantitative tool for a qualitative approach. Int J Soc Res Methodol 22(5):469–485 · DOI · Paper page
- Jänicke S, Franzini G, Cheema MF, Scheuermann G (2015) On close and distant reading in digital humanities: a survey and future challenges. In: Borgo R, Ganovelli F, Viola I (eds) Eurographics conference on visualization (EuroVis) – STARs. The Eurographics Association
- Jelodar H, Wang Y, Yuan C, Feng X, Jiang X, Li Y (2019) Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey. Multimed Tools Appl 78(11):15169–15211 · DOI · Paper page
- Kane H, Kahwati L (2023) Mixed methods and qualitative comparative analysis. In: Tierney RJ, Rizvi F, Ercikan K (eds) International encyclopedia of education (fourth edition). Elsevier, Oxford, pp 581–587 · DOI
- Kanger L, Tinits P, Pahker A-K, Orru K, Tiwari AK, Sillak S (2022) Deep transitions: towards a comprehensive framework for mapping major continuities and ruptures in industrial modernity. Glob Environ Change 72:102447 · DOI
- Karjus A, Cuskley C (2024) Evolving linguistic divergence on polarizing social media. Humanit Soc Sci Commun 11(1):1–14 · DOI
- Karjus A, Blythe RA, Kirby S, Smith K (2020) Quantifying the dynamics of topical fluctuations in language. Lang Dyn Change 10(1):86–125 · DOI
- Karjus A, Blythe RA, Kirby S, Wang T, Smith K (2021) Conceptual similarity and communicative need shape colexification: an experimental study. Cogn Sci 45(9):e13035 · DOI
- Lai VD, Nguyen MV, Kaufman H, Nguyen TH (2021) Event extraction from historical texts: a new dataset for black rebellions. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, pp 2390–2400 · DOI
- Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: Xing EP, Jebara T (eds) Proceedings of the 31st international conference on machine learning, vol 32. PMLR, pp 1188–1196
- Lew R (2023) ChatGPT as a COBUILD lexicographer. Humanit Soc Sci Commun 10(1):704 · DOI
- Lew R (2024) Dictionaries and lexicography in the AI era. Humanit Soc Sci Commun 11(1):1–8 · DOI
- Limberg C, Gonçalves A, Rigault B, Prendinger H (2024) Leveraging YOLO-World and GPT-4V LMMs for zero-shot person detection and action recognition in drone imagery. Preprint at https://doi.org/10.48550/arXiv.2404.01571 · arXiv · doi:10.48550/arxiv.2404.01571
- Linder F, Desmarais B, Burgess M, Giraudy E (2020) Text as policy: measuring policy similarity through bill text reuse. Policy Stud J 48(2):546–574 · DOI
- Liu L, Zhu M (2023) Bertalign: Improved word embedding-based sentence alignment for Chinese–English parallel corpora of literary texts. Digit Scholarsh Humanit 38(2):621–634 · DOI
- Lyu H, Huang J, Zhang D, Yu Y, Mou X, Pan J et al. (2023) GPT-4V(ision) as a social media analysis engine. Preprint at https://doi.org/10.48550/arXiv.2311.07547 · arXiv · doi:10.48550/arxiv.2311.07547
- Majumder BP, Li S, Ni J, McAuley J (2020) Interview: large-scale modeling of media dialog with discourse patterns and knowledge grounding. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). Association for Computational Linguistics, pp 8129–8141 · DOI
- Manjavacas E, Long B, Kestemont M (2019) On the feasibility of automated detection of allusive text reuse. In: Proceedings of the 3rd joint SIGHUM workshop on computational linguistics for cultural heritage, social sciences, humanities and literature. Association for Computational Linguistics, pp 104–114 · DOI
- Mao H, Cheung M, She J (2017) DeepArt: learning joint representations of visual arts. In: Proceedings of the 25th ACM international conference on multimedia, MM ’17. Association for Computing Machinery, pp 1183–1191 · DOI
- Mertens DM (2008) Transformative research and evaluation. Guilford Press, New York, p 417
- Mets M, Karjus A, Ibrus I, Schich M (2024) Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media. PLoS ONE 19(4):e0302380 · DOI
- Michel J-B, Shen YK, Aiden AP, Veres A, Gray MK, Pickett JP (2011) Quantitative analysis of culture using millions of digitized books. Science 331(6014):176–182 · DOI · Paper page
- Mollick E (2024). Co-Intelligence. Penguin Random House
- Montanelli S, Periti F (2023) A survey on contextualised semantic shift detection. Preprint at https://doi.org/10.48550/arXiv.2304.01666 · DOI
- Morgan DL (1993) Qualitative content analysis: a guide to paths not taken. Qual Health Res 3(1):112–121 · DOI · Paper page
- Mulder AD, Fonteyn L, Kestemont M (2022) Linguistic value construction in 18th-century London auction advertisements: a quantitative approach. In: Proceedings of the computational humanities research conference 2022 (CHR 2022), vol 3. Antwerp, Belgium, 12–14 December, 2022, pp 92–113
- Nadi F, Naghavipour H, Mehmood T, Azman, AB, Nagantheran, JA, Ting, KSK et al. (2024) Sentiment analysis using large language models: a case study of GPT-3.5. In: Bee Wah Y, Al-Jumeily OBE, Berry MW (eds) Data Science and Emerging Technologies. Springer Nature, pp 161–168 · DOI
- Nida EA (1979) A componential analysis of meaning: an introduction to semantic structures. In: Approaches to Semiotics [AS], vol 57. De Gruyter · doi:10.1515/9783110828696
- Norman C, Wildman JM, Sowden S (2021) COVID-19 at the deep end: a qualitative interview study of primary care staff working in the most deprived areas of England during the COVID-19 pandemic. Int J Environ Res Public Health 18(16):8689 · DOI
- Novozhilova E, Mays K, Katz JE (2024) Looking towards an automated future: U.S. attitudes towards future artificial intelligence instantiations and their effect. Humanit Soc Sci Commun 11(1):1–11 · DOI
- O’Halloran KL, Tan S, Wignell P, Bateman JA, Pham D-S, Grossman M (2019) Interpreting text and image relations in violent extremist discourse: a mixed methods approach for big data analytics. Terrorism Political Violence 31(3):454–474 · DOI
- Oiva M, Mukhina K, Zemaityte V, Karjus A, Tamm M, Ohm T (2024) A framework for the analysis of historical newsreels. Humanit Soc Sci Commun 11(1):530 · DOI
- Oiva M, Ristilä A (2022) Mapping the pseudohistorical knowledge space in the Russian World Wide Web. In: Välimäki R (ed) Medievalism in Finland and Russia: twentieth- and twenty-first-century aspects. Bloomsbury, England, pp 57–71 · doi:10.5040/9781350232921.0011
- Ollion E, Shen R, Macanovic A, Chatelain, A (2023) ChatGPT for text annotation? Mind the Hype! Preprint at https://doi.org/10.31235/osf.io/x58kn · DOI · Paper page
- OpenAI (2023). GPT-4 technical report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 · DOI · Paper page
- Overos HD, Hlatky R, Pathak O, Goers H, Gouws-Dewar J, Smith K (2024) Coding with the machines: machine-assisted coding of rare event data. PNAS Nexus 3(5):165 · DOI
- Paasonen S, Sundén J, Tiidenberg K, Vihlman M (2023) About sex, open-mindedness, and cinnamon buns: exploring sexual social media. Social Media Soc 9(1):20563051221147324 · doi:10.1177/20563051221147324
- Palaniyappan L (2023) Conversational agents and psychiatric semiology v3.0. Sistemi intelligenti, Rivista quadrimestrale di scienze cognitive e di intelligenza artificiale (2): 445–454. 10.1422/108140 · DOI
- Parks L, Peters W (2023) Natural language processing in mixed-methods text analysis: a workflow approach. Int J Soc Res Methodol 26(4):377–389 · DOI
- Periti F, Dubossarsky H, Tahmasebi N (2024) (Chat)GPT v BERT Dawn of Justice for Semantic Change Detection. In: Graham Y, Purver M (eds) Findings of the Association for Computational Linguistics: EACL 2024. Association for Computational Linguistics, pp 420–436
- Pilny A, McAninch K, Slone A, Moore K (2024) From manual to machine: assessing the efficacy of large language models in content analysis. Commun Res Rep 41(2):61–70 · DOI
- Rasenberg M, Pouw W, Özyürek A, Dingemanse M (2022) The multimodal nature of communicative efficiency in social interaction. Sci Rep 12(1):19111 · DOI
- Ren Z, Caputo A, Jones G (2024) A few-shot learning approach for lexical semantic change detection using GPT-4. In: Tahmasebi N, Montariol S, Kutuzov A, Alfter D, Periti F, Cassotti P et al. (eds) Proceedings of the 5th workshop on computational approaches to historical language change. Association for Computational Linguistics, pp 187–192 · DOI
- Rosin GD, Radinsky K (2022) Temporal attention for language models. In: Findings of the Association for Computational Linguistics: NAACL 2022. Association for Computational Linguistics, pp 1498–1508 · DOI
- Salmi H, Paju P, Rantala H, Nivala A, Vesanto A, Ginter F (2021) The reuse of texts in Finnish newspapers and journals, 1771–1920: a digital humanities perspective. Historical Methods: J Quant Interdiscip Hist 54(1):14–28 · doi:10.1080/01615440.2020.1803166
- Sandelowski M, Voils CI, Knafl G (2009) On quantitizing. J Mixed Methods Res 3(3):208–222 · DOI · Paper page
- Schlechtweg D, McGillivray B., Hengchen, S., Dubossarsky, H., and Tahmasebi, N. (2020). SemEval-2020 Task 1: Unsupervised lexical semantic change detection. In: Proceedings of the fourteenth workshop on semantic evaluation. International Committee for Computational Linguistics, pp 1–23 · DOI · Paper page
- Schleiger E, Mason C, Naughtin C, Reeson A, Paris C (2023) Collaborative intelligence: a scoping review of current applications. Preprint at Qeios, https://doi.org/10.32388/RZGEPB · DOI
- Schrodt PA, Van Brackle D (2013) Automated coding of political event data. In: Subrahmanian V (ed) Handbook of computational approaches to counterterrorism. Springer, pp 23–49 · DOI
- Sen I, Quercia D, Capra L, Montecchi M, Šćepanović S (2023) Insider stories: analyzing internal sustainability efforts of major US companies from online reviews. Humanit Soc Sci Commun 10(1):309 · DOI
- Sherstinova T, Moskvina A, Kirina M, Zavyalova I, Karysheva A, Kolpashchikova E. et al. (2022) Topic modeling of literary texts using LDA: on the influence of linguistic preprocessing on model interpretability. In: 2022 31st conference of open innovations association (FRUCT), pp 305–312 · doi:10.23919/fruct54823.2022.9770887
- Smits T, Wevers M (2023) A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections. Digit Scholarsh Humanit 38(3):1267–1280 · DOI
- Sobchuk O, Šeļa A (2024) Computational thematics: comparing algorithms for clustering the genres of literary fiction. Humanit Soc Sci Commun 11(1):1–12 · DOI
- Sprugnoli R, Tonelli S (2019) Novel event detection and classification for historical texts. Comput Linguist 45(2):229–265 · DOI
- Szmrecsanyi B, Rosenbach A, Bresnan J, Wolk C (2014) Culturally conditioned language change? A multi-variate analysis of genitive constructions in ARCHER. In: Hundt M (ed) Late modern English syntax, studies in English language. Cambridge University Press, pp 133–152 · doi:10.1017/cbo9781139507226.011
- Tamariz M, Kirby S (2015) Culture: copying, compression, and conventionality. Cogn Sci 39(1):171–183 · DOI · Paper page
- Tang B, Lin B, Yan H, Li S (2024) Leveraging generative large language models with visual instruction and demonstration retrieval for multimodal sarcasm detection. In: Duh K, Gomez H, Bethard S (eds) Proceedings of the 2024 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, vol 1 (long papers). Association for Computational Linguistics, Mexico City, Mexico, pp 1732–1742 · DOI
- Tanzer G, Suzgun M, Visser E, Jurafsky D, Melas-Kyriazi L (2024) A benchmark for learning to translate a new language from one grammar book. Preprint at https://doi.org/10.48550/arXiv.2309.16575 · DOI
- Taylor JET, Taylor GW (2021) Artificial cognition: how experimental psychology can help generate explainable artificial intelligence. Psychon Bull Rev 28(2):454–475 · DOI · Paper page
- Törnberg P (2024) Best practices for text annotation with large language models. Sociologica, 18(2):67–85
- Torrent TT, Hoffmann T, Almeida AL, Turner M (2023) Copilots for linguists: AI, constructions, and frames. In: Elements in construction grammar. Cambridge University Press, Cambridge · DOI
- Trahan A, Stewart DM (2013) Toward a pragmatic framework for mixed-methods research in criminal justice and criminology. Appl Psychol Crim Justice 9(1):59–74
- Trauzettel-Klosinski S, Dietz K, the IReST Study Group (2012) Standardized assessment of reading performance: the new international reading speed texts IReST. Investig Ophthalmol Vis Sci 53(9):5452–5461 · DOI · Paper page
- Van Rooij I, Guest O, Adolfi F, de Haan R, Kolokolova A, Rich P (2024) Reclaiming AI as a theoretical tool for cognitive science. Comput Brain Behav 7:616–636 · DOI · Paper page
- Vis B (2012) The comparative advantages of fsQCA and regression analysis for moderately large-N analyses. Sociol Methods Res 41(1):168–198 · DOI · Paper page
- Wang C, Liu S, Li A, Liu J (2023a) Text dialogue analysis based ChatGPT for primary screening of mild cognitive impairment. Preprint at https://doi.org/10.1101/2023.06.27.23291884 · DOI
- Warrens MJ (2008) On the equivalence of Cohen’s Kappa and the Hubert-Arabie adjusted Rand index. J Classif 25(2):177–183 · DOI · Paper page
- Wendler C, Veselovsky V, Monea G, West R (2024) Do llamas work in English? On the latent language of multilingual transformers. In: Ku L-W, Martins A, Srikumar V (eds) Proceedings of the 62nd annual meeting of the Association for Computational Linguistics, vol 1 (long papers). Association for Computational Linguistics, pp 15366–15394 · DOI
- Wen-Yi AW, Adamson K, Greenfield N, Goldberg R, Babcock S, Mimno, D et al. (2024) Automate or assist? The role of computational models in identifying gendered discourse in US capital trial transcripts. Preprint at http://arxiv.org/abs/2407.12500 · DOI
- Wichmann S, Holman EW, Bakker D, Brown CH (2010) Evaluating linguistic distance measures. Phys A Stat Mech Appl 389(17):3632–3639 · DOI
- Winter (2020) Statistics for linguists: an introduction using R. Routledge, London · DOI · Paper page
- Wolk C, Bresnan J, Rosenbach A, Szmrecsanyi B (2013) Dative and genitive variability in late modern English: exploring cross-constructional variation and change. Diachronica 30(3):382–419 · DOI · Paper page
- Wu T, Zhu H, Albayrak M, Axon A, Bertsch A, Deng W. et al. (2023a) LLMs as workers in human-computational algorithms? Replicating crowdsourcing pipelines with LLMs. Preprint at https://doi.org/10.1145/3706599.3706690 · DOI
- Wu W, Yao H, Zhang M, Song Y, Ouyang W, Wang J (2023b) GPT4Vis: what can GPT-4 do for zero-shot visual recognition? Preprint at https://doi.org/10.48550/arXiv.2311.15732 · DOI
- Xiao Z, Zhou MX, Liao QV, Mark G, Chi C, Chen W (2020) Tell me about yourself: using an AI-powered chatbot to conduct conversational surveys with open-ended questions. ACM Trans Computer-Hum Interact 27(3):15:1–15:37 · DOI · Paper page
- Yadav S, Choppa T, Schlechtweg D (2024) Towards automating text annotation: a case study on semantic proximity annotation using GPT-4. Preprint at https://doi.org/10.48550/arXiv.2407.04130 · DOI
- Yi Q, Zhang G, Liu J, Zhang S (2023) Movie scene event extraction with graph attention network based on argument correlation information. Sensors 23(4):2285 · doi:10.3390/s23042285
- Young SD, Jaganath D (2013) Online social networking for HIV education and prevention: a mixed-methods analysis. Sexually Transmitted Dis 40(2):162 · DOI
- Zhang W, Deng Y, Liu B, Pan S, Bing L (2024) Sentiment analysis in the era of large language models: a reality check. In: Duh K, Gomez H, Bethard S (eds) Findings of the Association for Computational Linguistics: NAACL 2024. Association for Computational Linguistics, pp 3881–3906 · DOI · Paper page
- Zhu J-J, Jiang J, Yang M, Ren ZJ (2023) ChatGPT and environmental research. Environ Sci Technol 57(46):17667–17670 · DOI · Paper page
Cited by
Winners and losers of generative AI: Early Evidence of Shifts in Freelancer Demand
Teutloff, O., Einsiedler, J., Kässi, O., Braesemann, F., Mishkin, P., & del Rio-Chanona, R. M. (2025) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2024.106845 · Cited by 39
Using large language models to analyze political texts through natural language understanding
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 37
Metadata
- Authors
- Karjus, A.
- ORCID
- Karjus, Andres
- Venue
- Humanities and Social Sciences Communications
- Published
- 2025-02-28
- Volume
- 12
- Issue
- 1
- ISSN
- 2662-9992
- Cited by
- 32
- References
- 54
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence. https://app.cochat.ai/papers/10.1057/s41599-025-04503-w
- The paper
- Karjus, A. (2025). Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence. Humanities and Social Sciences Communications. https://doi.org/10.1057/s41599-025-04503-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.