CoChat
Sign Up
Cited by 7

The DOI matches its Crossref record.

Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis

Borse, Nikhil Sanjay, Chatta Subramaniam, Ravishankar, Rebello, N. Sanjay·2025·Physics Education Research Conference Proceedings
DOI 10.1119/perc.2025.pr.borse
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1119/perc.2025.pr.borse

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 42

  1. National Research Council, Next Generation Science Stan- dards: For States, By States (The National Academies Press, Washington, DC, 2013). · DOI
  2. M. Honey, G. Pearson, and E. Schweingruber, Heidi, STEM In- tegration in K-12 Education: Status, Prospects, and an Agenda for Research (National Academies Press, Washington, DC, 2014). · DOI
  3. F. Fischer, C. Wecker, A. Hetmanek, J. Osborne, C. A. Chinn, R. G. Duncan, R. W. Rinhart, S. A. Siler, D. Klahr, and W. A. Sandoval, The interplay of domain-specific and domain- general factors in scientific reasoning and argumentation, in Proceedings of the International Conference of the Learning Sciences (ICLS) 2014, Vol. 3, edited by J. L. Polman, E. A. Kyza, K. Schwarz, T. Amin, Y. Abu-El-Haj, and C. Maher (International Society of the Learning Sciences, Boulder, CO, 2014) pp. 1189-1198.
  4. N. R. Council, D. of Behavioral, S. Sciences, B. on Science Ed- ucation, and C. on a Conceptual Framework for New K-12 Sci- ence Education Standards, A framework for K-12 science edu- cation: Practices, crosscutting concepts, and core ideas (Na- tional Academies Press, 2012).
  5. M. Dalal, A. Carberry, and L. Archambault, Developing a ways of thinking framework for engineering education research, Studies in Engineering Education 1, 108 (2021). · DOI
  6. D. Slavit, E. Grace, and K. Lesseig, Stem ways of think- ing, North American Chapter of the International Group for the Psychology of Mathematics Education (2019), conference proceedings.
  7. D. Slavit, E. Grace, and K. Lesseig, Student ways of thinking in stem contexts: A focus on claim making and reasoning, School Science and Mathematics 121, 466 (2021). · DOI
  8. Y.-C. N. Lien, W.-J. Wu, and Y.-L. Lu, How well do teach- ers predict students' actions in solving an ill-defined problem in stem education: a solution using sequential pattern mining, IEEE Access 8, 134976 (2020).
  9. L. D. English, Ways of thinking in stem-based problem solv- ing, ZDM-Mathematics Education 55, 1219 (2023). · DOI
  10. V. Talanquer and J. Pollard, Let's teach how we think instead of what we know, Chemistry Education Research and Practice 11, 74 (2010).
  11. M. J. Luna, S. J. Selmer, and J. A. Rye, Teachers' noticing of students' thinking in science through classroom artifacts: In what ways are science and engineering practices evident?, Journal of Science Teacher Education 29, 148 (2018).
  12. I. Wilkinson, A. Soter, and P. Murphy, Developing a model of quality talk about literary text, in Bringing Reading Research to Life: Essays in Honor of Isabel L. Beck, edited by M. McK- eown and L. Kucan (The Guilford Press, New York, NY, 2010) pp. 142-169.
  13. E. Etkina and G. Planinšič, Thinking like a scientist, Physics world 27, 48 (2014).
  14. C. M. Firetto, E. Starrett, and M. E. Jordan, Embracing a cul- ture of talk: Stem teachers' engagement in small-group dis- cussions about photovoltaics, International Journal of STEM Education 10, 50 (2023). · DOI
  15. N. K. Denzin and Y. S. Lincoln, Handbook of qualitative re- search, Journal of Leisure Research 28, 132 (1996).
  16. V. Braun and V. Clarke, Using thematic analysis in psychology, Qualitative Research in Psychology 3, 77 (2006). · DOI · Paper page
  17. J. M. Morse, Critical analysis of strategies for determining rigor in qualitative inquiry, Qualitative health research 25, 1212 (2015).
  18. J. Saldana, The coding manual for qualitative researchers, Vol. 3 (Sage Publications, Thousand Oaks, CA, 2009).
  19. K. Jackson and P. Bazeley, Qualitative data analysis with NVivo (SAGE Publications Ltd, 2019).
  20. İ. Uysal and N. Dogan, How reliable is it to automatically score open-ended items? an application in the turkish language, Jour- nal of Measurement and Evaluation in Education and Psychol- ogy 12, 28 (2021).
  21. M. F. Turgut and Y. Baykul, Egitimde ölçme ve degerlendirme, Vol. 2 (Pegem Akademi, 2010).
  22. L. R. Aiken, Psychological testing and assessment (Pearson Education India, 2009).
  23. E. A. Siverling, T. J. Moore, E. Suazo-Flores, C. A. Mathis, and S. S. Guzey, What initiates evidence-based reasoning?: Situ- ations that prompt students to support their design ideas and decisions, Journal of Engineering Education 110, 294 (2021). · DOI
  24. H. E. Tinsley and D. J. Weiss, Interrater reliability and agree- ment of subjective judgments., Journal of Counseling Psychol- ogy 22, 358 (1975).
  25. L. A. Siiman, M. Rannastu-Avalos, J. Pöysä-Tarhonen, P. Häkkinen, and M. Pedaste, Opportunities and challenges for ai-assisted qualitative data analysis: An example from collab- orative problem-solving discourse data, in International Con- ference on Innovative Technologies and Learning (Springer, 2023) pp. 87-96.
  26. A. Katz, S. Wei, G. Nanda, C. Brinton, and M. Ohland, Ex- ploring the efficacy of chatgpt in analyzing student team- work feedback with an existing taxonomy, arXiv preprint arXiv:2305.11882 (2023). · arXiv
  27. D. Hitch, Artificial intelligence augmented qualitative analysis: the way of the future?, Qualitative Health Research 34, 595 (2024).
  28. W. Tabone and J. De Winter, Using chatgpt for human- computer interaction research: a primer, Royal Society Open Science 10, 231053 (2023).
  29. H. Zhang, C. Wu, J. Xie, Y. Lyu, J. Cai, and J. M. Carroll, Re- defining qualitative analysis in the ai era: Utilizing chatgpt for efficient thematic analysis, arXiv preprint arXiv:2309.10771 (2023). · arXiv
  30. Z. O. Dunivin, Scaling hermeneutics: a guide to qualitative coding with llms for reflexive content analysis, EPJ Data Sci- ence 14, 28 (2025).
  31. X. Liu, A. F. Zambrano, R. S. Baker, A. Barany, J. Ocumpaugh, J. Zhang, M. Pankiewicz, N. Nasiar, and Z. Wei, Qualitative coding with gpt-4: Where it works better., Journal of Learning Analytics 12, 169 (2025).
  32. Z. Xiao, X. Yuan, Q. V. Liao, R. Abdelghani, and P.-Y. Oudeyer, Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding, in Companion proceedings of the 28th international conference on intelligent user interfaces (2023) pp. 75-78.
  33. OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Alt- man, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Bal- com, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Cur- rier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Dev- ille, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecof- fet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gib- son, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gor- don, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jo- moto, B. Jonn, H. Jun, T. Kaftan, Łukasz Kaiser, A. Ka- mali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O'Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascan- dolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thomp- son, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph, Gpt-4 technical report (2024), arXiv:2303.08774 [cs.CL]. · arXiv
  34. T. B. Brown et al., Language models are few-shot learners, NeurIPS (2020).
  35. R. C. Subramaniam, C. LaFontaine, A. Bralin, J. W. Morphew, C. M. Rebello, and N. S. Rebello, Characterising stem ways of thinking in engineering design (ed)-based tasks, in 2024 PERC Proceedings [Boston, MA, July 10-11, 2024], edited by Q. X. · DOI
  36. R. Bijker, S. S. Merkouris, N. A. Dowling, and S. N. Rodda, Chatgpt for automated qualitative research: content analysis, Journal of medical Internet research 26, e59050 (2024).
  37. E. G. Guba, Y. S. Lincoln, et al., Competing paradigms in qualitative research, Handbook of qualitative research 2, 105 (1994).
  38. A. Mizumoto and M. F. Teng, Large language models fall short in classifying learners' open-ended responses, Research Meth- ods in Applied Linguistics 4, 100210 (2025).
  39. B. Chen, Z. Zhang, N. Langrené, and S. Zhu, Unleashing the potential of prompt engineering in large language mod- els: a comprehensive review, arXiv preprint arXiv:2310.14735 (2023). · arXiv
  40. S. Xu, X. Huang, C. K. Lo, G. Chen, and M. S.-y. Jong, Eval- uating the performance of chatgpt and gpt-4o in coding class- room discourse data: A study of synchronous online mathe- matics instruction, Computers and Education: Artificial Intel- ligence 7, 100325 (2024).
  41. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948 (2025). · arXiv
  42. P. Tschisgale, P. Wulff, and M. Kubsch, Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory, Phys. Rev. Phys. Educ. Res. 19, 020123 (2023). · DOI

Cited by

Sorted by citations

Metadata

Authors
Borse, N. S., Chatta Subramaniam, R., & Rebello, N. S.
ORCID
Venue
Physics Education Research Conference Proceedings
Publisher
American Association of Physics Teachers
Published
2025-10-28
Pages
92-98
ISSN
2377-2379
DOI
10.1119/perc.2025.pr.borse
Cited by
7
References
0

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review