Can ChatGPT and Bard generate aligned assessment items? A reliability analysis against human performance
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, the reliability of OpenAI’s ChatGPT and Google’s Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts is measured. Intraclass correlation (ICC) as a performance metric showed that the reliability of both ChatGPT and Bard was low against the gold standard of human ratings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806
Pavlik, J. V. (2023) · Journalism & Mass Communication Educator · doi:10.1177/10776958221149577 · Cited by 972
Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies
Cope, B., Kalantzis, M., & Searsmith, D. (2021) · Educational Philosophy and Theory · doi:10.1080/00131857.2020.1728732 · Cited by 518 · cited 1×
“One promising area that AI can be of great assistance to learning and assessment is automatic item generation for summative and particularly formative assessment, especially in self-assessment contexts and personalized learning through continuous feedback into the learning processes (Cope et al., 2021).”
Background(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.23 · Cited by 421
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420
The Future of AI in Medicine: A Perspective from a Chatbot
King, M. R. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-022-03121-w · Cited by 299
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.1 · Cited by 278 · cited 1×
“, 2023) and the GRE, though some have shown otherwise (Huh, 2023).”
BackgroundThe death of the short-form physics essay in the coming AI revolution
Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A., & Testrow, C. P. (2023) · Physics Education · doi:10.1088/1361-6552/acc5cf · Cited by 199
The Death of the Short-Form Physics Essay in the Coming AI Revolution
Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A., & Testrow, C. (2022) · arXiv · doi:10.48550/arxiv.2212.11661 · Cited by 199
Educational Research and AI-Generated Writing: Confronting the Coming Tsunami
Tate, T. P., Doroudi, S., Ritchie, D., Xu, Y., & uci, M. W. (2023) · doi:10.35542/osf.io/4mec3 · Cited by 137
Cited by
Role of AI chatbots in education: systematic literature review
Labadze, L., Grigolia, M., & Machaidze, L. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00426-1 · Cited by 1,043
“• ChatGPT, launched in 2022 by OpenAI, is a large language model chatbot that can generate text, produce diverse creative content, and deliver informative answers to questions (Dergaa et al., 2023; Khademi, 2023; Rudolph et al., 2023).”
BackgroundBoscardin, C. K., Gin, B., Golde, P. B., & Hauer, K. E. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005439 · Cited by 620
Yavuz, F., Çelik, Ö., & Yavaş Çelik, G. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13494 · Cited by 121
The use of Generative AI in qualitative analysis: Inductive thematic analysis with ChatGPT
(2024) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2024.7.1.22 · Cited by 70
Limna, P., & Kraiwanit, T. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/7zmssq60 · Cited by 13
“These entities have significantly impacted various fields through their exceptional capabilities in text generation, machine translation, and code synthesis, indicating a notable shift in how digital systems interface with human linguistic constructs (Khademi, 2023; Teubner et al., 2023).”
Background
Metadata
- Venue
- Journal of Applied Learning & Teaching
- Publisher
- Open Access Publishing Association
- Published
- 2023-05-10
- Volume
- 6
- Issue
- 1
- ISSN
- 2591-801X (ISSN-L)
- OpenAlex
- W4376139550
- Cited by
- 50
- References
- 15
- FWCI
- 11.75
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.