From substitution to redefinition: A framework of machine learning‐based science assessment
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
This study develops a framework to conceptualize the use and evolution of machine learning (ML) in science assessment. We systematically reviewed 47 studies that applied ML in science assessment and classified them into five categories: (a) constructed response, (b) essay, (c) simulation, (d) educational game, and (e) inter‐discipline. We compared the ML‐based and conventional science assessments and extracted 12 critical characteristics to map three variables in a three‐dimensional framework: construct, functionality, and automaticity. The 12 characteristics used to construct a profile for ML‐based science assessments for each article were further analyzed by a two‐step cluster analysis. The clusters identified for each variable were summarized into four levels to illustrate the evolution of each. We further conducted cluster analysis to identify four classes of assessment across the three variables. Based on the analysis, we conclude that ML has transformed—but not yet redefined—conventional science assessment practice in terms of fundamental purpose, the nature of the science assessment, and the relevant assessment challenges. Along with the three‐dimensional framework, we propose five anticipated trends for incorporating ML in science assessment practice for future studies: addressing developmental cognition, changing the process of educational decision making, personalized science learning, borrowing 'good' to advance 'good', and integrating knowledge from other disciplines into science assessment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
Reconsidering Research on Learning from Media
Clark, R. E. (1983) · Review of Educational Research · doi:10.3102/00346543053004445 · Cited by 2,715
Some studies in machine learning using the game of checkers
Samuel, A. L. (2000) · IBM Journal of Research and Development · doi:10.1147/rd.441.0206 · Cited by 1,439
Factors Affecting Technology Uses in Schools: An Ecological Perspective
Zhao, Y., & Frank, K. A. (2003) · American Educational Research Journal · doi:10.3102/00028312040004807 · Cited by 947
Applying machine learning in science assessment: A systematic review
Zhai, X., Yin, Y., Pellegrino, J. W., Haudek, K. C., & Shi, L. (2020) · Studies in Science Education · doi:10.1080/03057267.2020.1735757 · Cited by 257
Self-citationZhu, M., Liu, O. L., & Lee, H.-S. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103668 · Cited by 229
Gobert, J. D., Sao Pedro, M., Raziuddin, J., & Baker, R. S. (2013) · Journal of the Learning Sciences · doi:10.1080/10508406.2013.837391 · Cited by 210
Kuo, C.-Y., & Wu, H.-K. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.06.002 · Cited by 66
Cited by
Almasri, F. (2024) · Research in Science Education · doi:10.1007/s11165-024-10176-3 · Cited by 361
Kim, N. J., & Kim, M. K. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.755914 · Cited by 298
“Zhai et al. (2020) reviewed 47 studies that adopted AI algorithms (i.e., machine learning) as assessments in science education and found AI to be an effective and validated alternative for traditional science assessments.”
BackgroundApplying machine learning in science assessment: A systematic review
Zhai, X., Yin, Y., Pellegrino, J. W., Haudek, K. C., & Shi, L. (2020) · Studies in Science Education · doi:10.1080/03057267.2020.1735757 · Cited by 257
Self-citationChatgpt for next generation science learning
Zhai, X. (2023) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/3589649 · Cited by 178
Applying large language models and chain-of-thought for automatic scoring
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100213 · Cited by 186
Metadata
- Authors
- Zhai, X., C. Haudek, K., Shi, L., H. Nehm, R., & Urban‐Lurain, M.
- Venue
- Journal of Research in Science Teaching
- Publisher
- Wiley
- Published
- 2020-10-06
- Volume
- 57
- Issue
- 9
- Pages
- 1430-1459
- ISSN
- 0022-4308 (ISSN-L)
- 1098-2736
- OpenAlex
- W3091828467
- Cited by
- 131
- References
- 11
- FWCI
- 4.17
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: From substitution to redefinition: A framework of machine learning‐based science assessment. https://app.cochat.ai/papers/10.1002/tea.21658
- The paper
- Zhai, X., C. Haudek, K., Shi, L., H. Nehm, R., & Urban‐Lurain, M. (2020). From substitution to redefinition: A framework of machine learning‐based science assessment. Journal of Research in Science Teaching. https://doi.org/10.1002/tea.21658
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.