Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds.
Open accessThe DOI matches its DataCite record; 108 of the 108 references it cites were found in a registry or an index.
Abstract
We present Science Assistments, an interactive environment, which assesses students' inquiry skills as they engage in inquiry using science microworlds. We frame our variables, tasks, assessments, and methods of analyzing data in terms of evidence-centered design. Specifically, we focus on the student model, the task model, and the evidence model in the conceptual assessment framework. In order to support both assessment and the provision of scaffolding, the environment makes inferences about student inquiry skills using models developed through a combination of text replay tagging [cf. Sao Pedro et al. 2011], a method for rapid manual coding of student log files, and educational data mining. Models were developed for multiple inquiry skills, with particular focus on detecting if students are testing their articulated hypotheses, and if they are designing controlled experiments. Student-level cross-validation was applied to validate that this approach can automatically and accurately identify these inquiry skills for new students. The resulting detectors also can be applied at run-time to drive scaffolding intervention.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 2×
“They were validated using six-fold student-level cross validation and had their predictive goodness assessed using two metrics, A’ 1 [Hanley and McNeil 1982] and Cohen’s Kappa (κ) 2 [Cohen 1960].”
MethodData Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · cited 2×
“They were validated using six-fold student-level cross validation and had their predictive goodness assessed using two metrics, A’ 1 [Hanley and McNeil 1982] and Cohen’s Kappa (κ) 2 [Cohen 1960].”
MethodFast algorithms for mining association rules
Agrawal, R., & Srikant, R. (1998) · Cited by 11,307
Witten, I. H. (2008) · Cited by 9,143
National Science Education Standards
Shea, J. H. (1995) · Journal of Geological Education · doi:10.5408/0022-1368-43.2.102 · Cited by 7,888
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776 · cited 3×
“This scaffolding design lead to a guided system; that is, we aimed to strike a balance between openended and guided inquiry [cf., Kirschner et al. 2006; Hmelo-Silver et al. 2007] such that students can be assessed on inquiry skills and so that they are provided with support so they do not flounder…”
BackgroundAxten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology · doi:10.2307/2063712 · Cited by 6,834
Newell, A. (1972) · Cited by 6,222
Cited by
Items 101–120 of 120
Pedro, M. S. (2019) · American Educational Research Association Annual Meeting (AERA) · doi:10.3102/1433374 · Cited by 2
“• Forming questions: scored based on whether students could identify independent variables, dependent variables, and whether their IV and DV matched their experimental goal (Gobert et al., 2012)”
BackgroundMethodInfluentialMultimedia-based performance assessment in Dutch vocational education
de Klerk, S. (2015) · doi:10.3990/1.9789036539975 · Cited by 2
Boosting AI Innovation, Technology Transfer, and Industry Collaboration in Arab Higher Education
Al-Omari, H. M., Zabalawi, I., Malkawi, A. I. H., & Kordahji, H. (2026) · Springer eBooks · doi:10.1007/978-3-031-99068-7_17 · Cited by 1
Jiang, Y. (2018) · doi:10.7916/d82z2npw · Cited by 1
“estimate and predict the acquisition of CVS in computer-based learning environments (Gobert et al., 2012; Sao Pedro et al., 2013; Sao Pedro et al., 2014).”
BackgroundInfluentialBachmann, M. (2012) · Digital WPI · Cited by 1
Toutkoushian, E., & Ryoo, K. (2024) · Measurement Interdisciplinary Research and Perspectives · doi:10.1080/15366367.2023.2246754 · Cited by 1
From experience to explanation: an analysis of students’ use of a wildfire simulation
Lord, T., Horwitz, P., Pallant, A., & Lore, C. (2024) · Educational Technology Research and Development · doi:10.1007/s11423-024-10433-2 · Cited by 2
“A small group of researchers who study educational simulations have attempted to analyze the clickstream data generated by students’ use of simulations and other online tools (Buckley et al., 2006; Gobert et al., 2012; McElhaney & Linn, 2011; Wilson et al., 2018).”
Gobert, J. D., Raziuddin, J. J., & Koedinger, K. R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_109 · Cited by 1
Self-citationFurther Issues in Producing Valid Assessment Data
Maxwell, G. S. (2021) · The enabling power of assessment · doi:10.1007/978-3-030-63539-8_6 · Cited by 1
DiSCS: A New Sequence Segmentation Method for Open-Ended Learning Environments
Bywater, J. P., Floryan, M., & Chiu, J. L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78292-4_8 · Cited by 3
Educational Survey Assessments
Oranje, A., Keehner, M., Persky, H., Cayton-Hodges, G. A., & Feng, G. (2016) · John Wiley & Sons, Inc. eBooks · doi:10.1002/9781118956588.ch18
Mixed Agents Virtual Observation Lenses for Immersive Learning Environments
Peña-Ríos, A., Gardner, M., Felemban, S., & Callaghan, V. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.3217/jucs-024-02-0171
Investigating Epistemic Stances in Game Play Through Learning Analytics
Martinez-Garza, M. M., & Clark, D. B. (2018) · Advances in game-based learning book series · doi:10.4018/978-1-5225-7461-3.ch004
Searching for Predictors of Learning Outcomes in Non Abstract Eye Movement Logs
Gobert, J. D., Toto, E., Brigham, M., & Pedro, M. S. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_116
“Scoring of multiple choice pre- and post-test data was done automatically within the learning environment Science Assistments (Gobert et al., 2012), now referred to as Inq-ITS (www.inqITS).”
MethodSelf-citationDesigning and Testing Assessments and Scaffolds for Mathematics Practices in Science Inquiry
Olsen, J., & Gobert, J. D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78270-2_84
Self-citationAn Observation Framework for Recognising Learning Evidence in 3D Collaborative Virtual Environments
Felemban, S. (2021) · Open Access at Essex (University of Essex)
Investigating Epistemic Stances in Game Play Through Learning Analytics
Martinez-Garza, M. M., & Clark, D. B. (2021) · IGI Global eBooks · doi:10.4018/978-1-6684-3710-0.ch031
Modeling mathematics anxiety in middle school students: a machine learning approach
Mahaur, N., Lakshmi, C. V., & Ahuja, S. (2026) · SN Social Sciences · doi:10.1007/s43545-026-01685-1
Adapting the scheduling of illustrations and graphs to learners in conceptual physics tutoring
Lipschultz, M. (2015) · D-Scholarship@Pitt (University of Pittsburgh)
Metadata
- Authors
- Gobert, J. D., Sao Pedro, M. A., Baker, R. S. J. D., Toto, E., & Montalvo, O.
- Affiliations
- Worcester Polytechnic Institute (US)
- Worcester Polytechnic Institute
- Venue
- Zenodo
- Publisher
- Zenodo
- Published
- 2012-10-01
- Volume
- 4
- Issue
- 1
- Pages
- 111-143
- ISSN
- 2157-2100 (ISSN-L)
- OpenAlex
- W1485330246
- Cited by
- 122
- References
- 107
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 5 · 2023: 9 · 2022: 11 · 2021: 7 · 2020: 9 · 2019: 7 · 2018: 11 · 2017: 8 · 2016: 12 · 2015: 11 · 2014: 10 · 2013: 16 · 2012: 2
- FWCI
- 12.28
- Type
- journal article
- Language
- en
- Open access
- Locations
- Journal of Educational Data Mining (publishedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by-nc-nd)
- Zenodo (CERN European Organization for Nuclear Research)
- Educational Data Mining
- Funders
- National Science Foundation (0742503, 0733286, 1008649)
- U.S. Department of Education
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Education › Science Education and Pedagogy
- Keywords
- scientific inquiry skills, evidence-centered design, student model, task modeling, evidence model, hypothesis testing, controlled experiments, scaffolding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.