Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds.
Open accessThe DOI matches its DataCite record; 108 of the 108 references it cites were found in a registry or an index.
Abstract
We present Science Assistments, an interactive environment, which assesses students' inquiry skills as they engage in inquiry using science microworlds. We frame our variables, tasks, assessments, and methods of analyzing data in terms of evidence-centered design. Specifically, we focus on the student model, the task model, and the evidence model in the conceptual assessment framework. In order to support both assessment and the provision of scaffolding, the environment makes inferences about student inquiry skills using models developed through a combination of text replay tagging [cf. Sao Pedro et al. 2011], a method for rapid manual coding of student log files, and educational data mining. Models were developed for multiple inquiry skills, with particular focus on detecting if students are testing their articulated hypotheses, and if they are designing controlled experiments. Student-level cross-validation was applied to validate that this approach can automatically and accurately identify these inquiry skills for new students. The resulting detectors also can be applied at run-time to drive scaffolding intervention.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 2×
“They were validated using six-fold student-level cross validation and had their predictive goodness assessed using two metrics, A’ 1 [Hanley and McNeil 1982] and Cohen’s Kappa (κ) 2 [Cohen 1960].”
MethodData Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · cited 2×
“They were validated using six-fold student-level cross validation and had their predictive goodness assessed using two metrics, A’ 1 [Hanley and McNeil 1982] and Cohen’s Kappa (κ) 2 [Cohen 1960].”
MethodFast algorithms for mining association rules
Agrawal, R., & Srikant, R. (1998) · Cited by 11,307
Witten, I. H. (2008) · Cited by 9,143
National Science Education Standards
Shea, J. H. (1995) · Journal of Geological Education · doi:10.5408/0022-1368-43.2.102 · Cited by 7,888
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776 · cited 3×
“This scaffolding design lead to a guided system; that is, we aimed to strike a balance between openended and guided inquiry [cf., Kirschner et al. 2006; Hmelo-Silver et al. 2007] such that students can be assessed on inquiry skills and so that they are provided with support so they do not flounder…”
BackgroundAxten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology · doi:10.2307/2063712 · Cited by 6,834
Newell, A. (1972) · Cited by 6,222
Applied Optimal Control: Optimization, Estimation, and Control
Bryson, A. E., & Ho, Y.-C. (2018) · Routledge eBooks · doi:10.1201/9781315137667 · Cited by 6,015
Introduction to Classical and Modern Test Theory
Crocker, L. M., & Algina, J. J. (1986) · Cited by 5,090
(1991) · Choice Reviews Online · doi:10.5860/choice.28-5376 · Cited by 4,207
(1997) · Obesity Surgery · doi:10.1381/096089297765555359 · Cited by 3,916
Statistical Inference for Probabilistic Functions of Finite State Markov Chains
Baum, L. E., & Petrie, T. (1966) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177699147 · Cited by 2,775 · cited 2×
“[Baum and Petrie 1966; cf. Russell and Norvig 2009] for identifying learner trajectories through a series of activities.”
BackgroundHMELO-SILVER, C. E., DUNCAN, R. G., & CHINN, C. A. (2007) · Educational Psychologist · doi:10.1080/00461520701263368 · Cited by 2,641 · cited 2×
“…scaffolding design lead to a guided system; that is, we aimed to strike a balance between openended and guided inquiry [cf., Kirschner et al. 2006; Hmelo-Silver et al. 2007] such that students can be assessed on inquiry skills and so that they are provided with support so they do not flounder or…”
BackgroundKnowing What Students Know: The Science and Design of Educational Assessment
Pellegrino, J. W., Chudowsky, N., & Glaser, R. A. (2013) · Cited by 2,402
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324 · cited 4×
“In the future, we plan on doing so by employing Bayesian Knowledge-Tracing (BKT) [Corbett and Anderson 1995].”
MethodInfluentialEducational Data Mining: A review of the state of the art
Romero, C., & Ventura, S. (2010) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2010.2053532 · Cited by 2,251 · cited 3×
“We also focus on how we leveraged techniques from the educational data mining (EDM) literature [cf. Baker and Yacef 2009; Romero and Ventura 2010] to develop our evidence models, and reliably assess inquiry process skills associated with data collection [Sao Pedro et al., 2011].”
BackgroundMethodIntegrating classification and association rule mining
Liu, B., Hsu, W., & Ma, Y. (1998) · Cited by 2,247
The Atomic Components of Thought
Anderson, J. R., & Lebiere, C. J. (2014) · Psychology Press eBooks · doi:10.4324/9781315805696 · Cited by 2,173 · cited 2×
“…[1998] engineered a rule-based Adaptive Control of Thought—Rational (ACT-R) model of scientific inquiry based on an assessment of skill differences between experts and novices on formulating hypotheses, exploring, analyzing data, and generating conclusions [e.g., Anderson and Lebiere 1998].”
BackgroundMethodKuhn, D. (1991) · doi:10.1017/cbo9780511571350 · Cited by 2,094 · cited 2×
“2011], and even scientifically naive adults do not understand the control for variables strategy if not taught explicitly to do so [Kuhn 1991].”
BackgroundThe theory and practice of item response theory.
Ayala, D. (2009) · Guilford Press eBooks · Cited by 1,812
Self‐Explanations: How Students Study and Use Examples in Learning to Solve Problems
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989) · Cognitive Science · doi:10.1207/s15516709cog1302_1 · Cited by 1,714
Inquiry, modeling, and metacognition: making science accessible to all students
White, B. Y., & Frederiksen, J. R. (1998) · Cognition and Instruction · doi:10.1207/s1532690xci1601_2 · Cited by 1,394 · cited 1×
“Science Assistments also differs from other microworld-based discovery environments [e.g., White and Frederiksen 1998; van Joolingen and de Jong 2003; Buckley, Gobert, & Horwitz 2006; McElhaney and Linn 2008, 2010; Buckley et al. 2010; de Jong et al. 2010] in several ways.”
BackgroundChinn, C. A., & Brewer, W. F. (1993) · Review of Educational Research · doi:10.3102/00346543063001001 · Cited by 1,393 · cited 7×
“Finally, given the plethora of studies that have evidenced students’ difficulties in executing inquiry, including difficulties with hypothesis-formation [Chinn and Brewer 1993; Klahr and Dunbar 1988; Kuhn et al. 1995; Njoo and de Jong 1993; van Joolingen and de Jong 1997; Glaser et al. 1992], difficulties with conduct…”
BackgroundMethodInfluentialThe State of Educational Data Mining in 2009: A Review and Future Visions
Baker, R. S. J. D., & Yacef, K. (2009) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554657 · Cited by 1,359 · cited 4×
“We also focus on how we leveraged techniques from the educational data mining (EDM) literature [cf. Baker and Yacef 2009; Romero and Ventura 2010] to develop our evidence models, and reliably assess inquiry process skills associated with data collection [Sao Pedro et al., 2011].”
BackgroundMethodInfluentialSelf-citationThe interplay of evidence and consequences in the validation of performance assessments
Messick, S. (1994) · Educational Researcher · doi:10.3102/0013189x023002013 · Cited by 1,315 · cited 2×
“…that specifies features of tasks and questions that would elicit the evidence of learning, then to an evidence model that specifies the quality of student responses that would indicate levels of proficiency [Messick 1994; Mislevy et al. 2003; Mislevy et al., this issue; Pellegrino et al. 2001].”
BackgroundThe State of Educational Data Mining in 2009: A Review and Future Visions
Baker, R. S., & Yacef, K. (2009) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554658 · Cited by 1,296
Self-citationIntelligent Tutoring Goes To School in the Big City
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (2018) · doi:10.1184/r1/6470153 · Cited by 1,105
Langley, P., Simon, H. A., Bradshaw, G. L., & Zytkow, J. M. (1993) · Readings in Philosophy and Cognitive Science · doi:10.7551/mitpress/5782.003.0013 · Cited by 1,090
Dual space search during scientific reasoning
Klahr, D., & Dunbar, K. (1988) · Cognitive Science · doi:10.1207/s15516709cog1201_1 · Cited by 1,058 · cited 6×
“…and Schunn 2004; McElhaney and Linn 2008, 2010); difficulties with interpreting data, and linking hypotheses and data [Chinn and Brewer 1993; Klahr and Dunbar 1988; Kuhn et al. 1995], and difficulties with communicating findings [Krajcik et al. 1998; McNeill and Krajcik 2007], it seems…”
BackgroundMethodInfluentialFocus Article: On the Structure of Educational Assessments
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003) · Measurement: Interdisciplinary Research & Perspective · doi:10.1207/s15366359mea0101_02 · Cited by 802
Eliciting Self‐Explanations Improves Understanding
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994) · Cognitive Science · doi:10.1207/s15516709cog1803_3 · Cited by 757 · cited 1×
“We incorporate this inquiry step since there is evidence that explaining is a good strategy to get students to reify their understanding [Chi et al. 1989; Chi et al. 1994; Chi 2000].”
MethodAll Other Things Being Equal: Acquisition and Transfer of the Control of Variables Strategy
Chen, Z., & Klahr, D. (1999) · Child Development · doi:10.1111/1467-8624.00081 · Cited by 736 · cited 6×
“Regarding the designing and conducting experiments, a great deal of prior work has shown that students [Chen and Klahr 1999; McElhaney and Linn 2008, 2010; Sao Pedro et al. 2010, 2011], and even scientifically naive adults do not understand the control for variables strategy if not taught…”
BackgroundMethodInfluentialHambleton, R. K., & Jones, R. W. (1993) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.1993.tb00543.x · Cited by 731 · cited 1×
“…being assessed are psychologically multidimensional and as such, traditional psychometric methods such as classical test theory (CTT) [e.g., Crocker and Algina 2006] and unidimensional item response theory (UIRT) [de Ayala 2009; Hambleton and Jones 1993] cannot appropriately model their complexity.”
MethodInquiry in Project-Based Science Classrooms: Initial Attempts by Middle School Students
Krajcik, J., Blumenfeld, P. C., Marx, R. W., Bass, K. M., Fredricks, J., & Soloway, E. (1998) · Journal of the Learning Sciences · doi:10.1080/10508406.1998.9672057 · Cited by 688 · cited 1×
“1995], and difficulties with communicating findings [Krajcik et al. 1998; McNeill and Krajcik 2007], it seems reasonable to us that an environment like ours, which as designed to assess and assist with”
BackgroundOn the cognitive effects of learning computer programming
Pea, R. D., & Kurland, D. M. (1984) · New Ideas in Psychology · doi:10.1016/0732-118x(84)90018-7 · Cited by 544 · cited 3×
“Journal of Educational Data Mining, Article 5, Volume 4, No 1, October 2012 155 microworlds [Papert 1980], computerized models of real-world phenomena whose properties can be inspected and changed [Pea and Kurland 1984; Resnick 1997].”
BackgroundInfluentialImplications of Evidence‐Centered Design for Educational Testing
Mislevy, R. J., & Haertel, G. D. (2006) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.2006.00075.x · Cited by 543
Addressing the assessment challenge with an online system that tutors as it assesses
Feng, M., Heffernan, N., & Koedinger, K. (2009) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-009-9063-7 · Cited by 502 · cited 1×
“Journal of Educational Data Mining, Article 5, Volume 4, No 1, October 2012 174 and Corbett 2006; Baker et al. 2008a; Ritter et al. 2009; Feng et al. 2009; Baker et al. 2010; Pardos et al. 2010].”
BackgroundTechnological advances in inquiry learning
de Jong, T. (2006) · Science · doi:10.1126/science.1127750 · Cited by 469 · cited 3×
“This skill may be related to students’ planning and monitoring [de Jong 2006], particularly if they are testing more than one hypothesis.”
Baker, R. S. J. D., Corbett, A. T., & Aleven, V. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-69132-7_44 · Cited by 442 · cited 7×
“In this line of research, Walonoski and Heffernan [2006] as well as Baker et al. [2008b] successfully built and validated detectors of “gaming the system” (whereby students simply try to get as many hints as possible rather than learn the material) by triangulating qualitative field observations with features gleaned…”
BackgroundMethodInfluentialSelf-citationKuhn, D. (1986) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146818608700404 · Cited by 439 · cited 8×
“For example, research has shown that students have difficulty with inquiry, including choosing the correct variables to work with, forming testable hypotheses, drawing correct conclusions from experiments, and linking hypotheses and data [Chinn and Brewer 1993; Klahr and Dunbar 1988; Kuhn et al. 1995; Kuhn 2005].”
BackgroundMethodInfluentialStrategies of Knowledge Acquisition
Kuhn, D., Garcia-Mila, M., Zohar, A., Andersen, C., White, S. H., Klahr, D., & Carver, S. M. (1995) · Monographs of the Society for Research in Child Development · doi:10.2307/1166059 · Cited by 422
Knowing What Students Know: The Science and Design of Educational Assessment
(2001) · National Academies Press eBooks · doi:10.17226/10019 · Cited by 380
Perkins, D. N. (2013) · Routledge eBooks · doi:10.4324/9781315803067 · Cited by 351
Sensible Reasoning: A Hypothesis about Hypotheses
Tschirgi, J. E. (1980) · Child Development · doi:10.1111/j.1467-8624.1980.tb02502.x · Cited by 318
Students' Understanding of the Objectives and Procedures of Experimentation in the Science Classroom
Schauble, L., Glaser, R., Duschl, R. A., Schulze, S., & John, J. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_1 · Cited by 306 · cited 1×
“…often change too many variables when attempting to test a hypothesis [Tsirgi 1980; Shute and Glaser 1990; Schunn and Anderson 1999; McElhaney and Linn 2008, 2010], and consequently incorrectly draw conclusions from confounded data [Klahr and Dunbar 1988; Kuhn et al. 1992; Schauble et al. 1995].”
BackgroundA Large‐Scale Evaluation of an Intelligent Discovery World: Smithtown
Shute, V. J., & Glaser, R. (1990) · Interactive Learning Environments · doi:10.1080/1049482900010104 · Cited by 279 · cited 4×
“1992], difficulties with conducting experiments [Glaser et al. 1992; Reimann 1991; Tsirgi 1980; Shute and Glaser 1990; Kuhn 2005; Schunn and Anderson 1998, 1999; Harrison and Schunn 2004; McElhaney and Linn 2008, 2010); difficulties with interpreting data, and linking hypotheses and data [Chinn and Brewer 1993; Klahr…”
BackgroundResultInfluentialCognitive Tutors: Technology Bringing Learning Sciences to the Classroom
Koedinger, K. R., & Corbett, A. (2005) · The Cambridge Handbook of the Learning Sciences · doi:10.1017/cbo9780511816833.006 · Cited by 275
Cross-Domain Development of Scientific Reasoning
Kuhn, D., Schauble, L., & Garcia-Mila, M. (1992) · Cognition and Instruction · doi:10.1207/s1532690xci0904_1 · Cited by 267 · cited 1×
“…often change too many variables when attempting to test a hypothesis [Tsirgi 1980; Shute and Glaser 1990; Schunn and Anderson 1999; McElhaney and Linn 2008, 2010], and consequently incorrectly draw conclusions from confounded data [Klahr and Dunbar 1988; Kuhn et al. 1992; Schauble et al. 1995].”
BackgroundTesting: Friend or Foe?: Theory and Practice of Assessment and Testing
Black, P. (1997) · Cited by 227
Automated Scoring of Complex Tasks in Computer-Based Testing
(2006) · Routledge eBooks · doi:10.4324/9780415963572 · Cited by 227 · cited 3×
“…to Quellmalz et al. [2009], is principally due to the complexity of the simulations, and is summarized by their following four main characteristics [Williamson et al. 2006; see also Rupp et al. 2010]: 1) the task requires the learner to complete many, non-trivial, domain-relevant steps/processes,…”
BackgroundNjoo, M., & De Jong, T. (1993) · Journal of Research in Science Teaching · doi:10.1002/tea.3660300803 · Cited by 215
Developing a generalizable detector of when students game the system
Baker, R. S. J. D., Corbett, A. T., Roll, I., & Koedinger, K. R. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9045-6 · Cited by 203
Self-citationRupp, A. A., Gushta, M. M., Mislevy, R. J., & Shaffer, D. W. (2010) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 195
The Generality/Specificity of Expertise in Scientific Reasoning
Schunn, C. D., & Anderson, J. R. (1999) · Cognitive Science · doi:10.1016/s0364-0213(99)00006-3 · Cited by 185 · cited 2×
“…1997; Glaser et al. 1992], difficulties with conducting experiments [Glaser et al. 1992; Reimann 1991; Tsirgi 1980; Shute and Glaser 1990; Kuhn 2005; Schunn and Anderson 1998, 1999; Harrison and Schunn 2004; McElhaney and Linn 2008, 2010); difficulties with interpreting data, and linking…”
BackgroundResultSmall-group discussion in physics: Peer interaction modes in pairs and fours
Alexopoulou, E., & Driver, R. (1996) · Journal of Research in Science Teaching · doi:10.1002/(sici)1098-2736(199612)33:10<1099::aid-tea4>3.0.co;2-n · Cited by 182
Massachusetts science and technology/engineering curriculum framework
Secondary Education. Office for Career (2006) · Cited by 176
Student assessment using Bayesian nets
Martin, J. J., & VanLehn, K. A. (1995) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1995.1025 · Cited by 171
Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems
Walonoski, J. A., & Heffernan, N. T. (2006) · Lecture notes in computer science · doi:10.1007/11774303_38 · Cited by 164
Mislevy, R. J., Behrens, J. T., Dicerbo, K. E., & Levy, R. (2012) · Zenodo · doi:10.5281/zenodo.3554641 · Cited by 139
Contextual Slip and Prediction of Student Performance after Use of an Intelligent Tutor
Baker, R. S., Corbett, A. T., Gowda, S. M., Wagner, A. Z., MacLaren, B. A., Kauffman, L. R., Mitchell, A. P., & Giguere, S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13470-8_7 · Cited by 137 · cited 3×
“The performance of these detectors, as measured by A’ and Kappa, is comparable to detectors of gaming the system refined over several years [e.g., Baker and de Carvalho 2008; Baker et al. 2010].”
BackgroundMethodSelf-citationStudent Modelling Based on Belief Networks
Reye, J. (2004) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2004-14(1)04 · Cited by 136
McNeill, K. L., & Krajcik, J. S. (2012) · Cited by 135
Constraint-Based Tutors: A Success Story
Mitrovic, A., Mayo, M., Suraweera, P., & Martin, B. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-45517-5_103 · Cited by 125 · cited 1×
“Science Assistments enables a moderate degree of student control, less than in purely exploratory learning environments [Amershi and Conati 2009], but more than in classic model-tracing tutors [Koedinger and Corbett 2006] or constraint-based tutors [Mitrovic et al. 2001].”
BackgroundSupporting hypothesis generation by learners exploring an interactive computer simulation
van Joolingen, W., & de Jong, T. (1991) · Instructional Science · doi:10.1007/bf00116355 · Cited by 110
An extended dual search space model of scientific discovery learning
van Joolingen, W., & de Jong, T. (1997) · Instructional Science · doi:10.1023/a:1002993406499 · Cited by 109
Amershi, S., & Conati, C. (2009) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554659 · Cited by 109 · cited 1×
“Science Assistments enables a moderate degree of student control, less than in purely exploratory learning environments [Amershi and Conati 2009], but more than in classic model-tracing tutors [Koedinger and Corbett 2006] or constraint-based tutors [Mitrovic et al. 2001].”
BackgroundExtending the scope of the student model
Bull, S., Brna, P., & Pain, H. (1995) · User Modeling and User-Adapted Interaction · doi:10.1007/bf01101801 · Cited by 107 · cited 1×
“In particular, these groups do not separate the evidence model from the student model; the student model contains the definition of the latent variables and the ways they and their relationships are measured are combined [cf. Wenger 1987; Bull et al. 1995].”
BackgroundPedro, M. S., Baker, R. S., Gobert, J. D., Montalvo, O., & Nakama, A. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9101-0 · Cited by 105
Self-citationScientific Reasoning Across Different Domains
Glaser, R. A., Schauble, L., Raghavan, K., & Zeitz, C. M. (1992) · Springer eBooks · doi:10.1007/978-3-642-77228-3_16 · Cited by 104 · cited 2×
“…Dunbar 1988; Kuhn et al. 1995; Njoo and de Jong 1993; van Joolingen and de Jong 1997; Glaser et al. 1992], difficulties with conducting experiments [Glaser et al. 1992; Reimann 1991; Tsirgi 1980; Shute and Glaser 1990; Kuhn 2005; Schunn and Anderson 1998, 1999; Harrison and Schunn 2004; McElhaney…”
BackgroundRhetoric and reality in science performance assessments: An update
Ruiz-Primo, M. A., & Shavelson, R. J. (1996) · Journal of Research in Science Teaching · doi:10.1002/(sici)1098-2736(199612)33:10<1045::aid-tea1>3.0.co;2-s · Cited by 99
Learning by creating and exchanging objects: the SCY experience
De Jong, T., Van Joolingen, W. R., Giemza, A., Girault, I., Hoppe, U., Kindermann, J., Kluge, A., Lazonder, A. W., Vold, V., Weinberger, A., et al. (2010) · British Journal of Educational Technology · doi:10.1111/j.1467-8535.2010.01121.x · Cited by 98 · cited 1×
“Science Assistments also differs from other microworld-based discovery environments [e.g., White and Frederiksen 1998; van Joolingen and de Jong 2003; Buckley, Gobert, & Horwitz 2006; McElhaney and Linn 2008, 2010; Buckley et al. 2010; de Jong et al. 2010] in several ways.”
BackgroundScience performance assessments: benchmarks and surrogates
Baxter, G., & Shavelson, R. (1994) · International Journal of Educational Research · doi:10.1016/s0883-0355(06)80020-0 · Cited by 85 · cited 2×
“Hands on performance assessments are more authentic because they require specific skills to solve real problems [Baxter and Shavelson 1994; Ruiz-Primo and Shavelson 1996].”
BackgroundMethodSimQuest, Authoring educational simulations
van Joolingen, W., de Jong, T., Murray, T., Blessing, S. B., & Ainsworth, S. (2003) · University of Twente Research Information · Cited by 81
Note on Sources of Sampling Variability in Science Performance Assessments
Shavelson, R. J., Ruiz‐Primo, M. A., & Wiley, E. W. (1999) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1999.tb00546.x · Cited by 79 · cited 6×
“Our approach assesses each inquiry skill over multiple trials, in the context of rich science inquiry microworlds, thereby providing a solution to three previously acknowledged problems, (1) the amount of data required to establish reliable measurement of inquiry skills [Shavelson et al. 1999], (2) the capacity to ass…”
BackgroundMethodInfluentialLabeling Student Behavior Faster and More Precisely with Text Replays.
Baker, R. S., & de Carvalho, A. M. J. B. (2008) · Educational Data Mining · Cited by 77
Self-citationKaczorowski, J., & Perelli, A. (1999) · Cited by 76
Modeling the Development of Problem Solving Skills in Chemistry with a Web-Based Tutor
Stevens, R., Soller, A., Cooper, M. M., & Sprang, M. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30139-4_55 · Cited by 71
Using Fine-Grained Skill Models to Fit Student Performance with Bayesian Networks
Pardos, Z. A., Heffernan, N. T., Anderson, B. S., & Heffernan, C. (2010) · CRC Press eBooks · doi:10.1201/b10274-36 · Cited by 71
Detecting Gaming the System in Constraint-Based Tutors
Baker, R. S., Mitrović, A., & Mathews, M. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13470-8_25 · Cited by 67 · cited 3×
“The performance of these detectors, as measured by A’ and Kappa, is comparable to detectors of gaming the system refined over several years [e.g., Baker and de Carvalho 2008; Baker et al. 2010].”
BackgroundMethodSelf-citationRupp, A. A., Levy, R., DiCerbo, K. E., Sweet, S. J., Crawford, A., Caliço, T., Benson, M., Fay, D. M., Kunze, K. L., Mislevy, R. J., et al. (2012) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554643 · Cited by 61
Modeling User Knowledge with Dynamic Bayesian Networks in Interactive Narrative Environments
Rowe, J., & Lester, J. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v6i1.12403 · Cited by 61
Reducing the Knowledge Tracing Space
RITTER, S., Harris, T. K., Nixon, T. M., Dickison, D., Murray, R. C., & Towle, B. (2009) · Educational Data Mining · Cited by 58
Automatic detection of users’ skill levels using high-frequency user interface events
Ghazarian, A., & Noorhosseini, S. M. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9073-5 · Cited by 58
Looking inside the black box: assessing model-based learning and inquiry in BioLogica™
Buckley, B. C., Gobert, J. D., Horwitz, P., & O'Dwyer, L. M. (2010) · International Journal of Learning Technology · doi:10.1504/ijlt.2010.034548 · Cited by 49 · cited 5×
“Science Assistments also differs from other microworld-based discovery environments [e.g., White and Frederiksen 1998; van Joolingen and de Jong 2003; Buckley, Gobert, & Horwitz 2006; McElhaney and Linn 2008, 2010; Buckley et al. 2010; de Jong et al. 2010] in several ways.”
BackgroundMethodInfluentialSelf-citationComponent-Based Construction of a Science Learning Space
Koedinger, K. R., Suthers, D. D., & Forbus, K. D. (1998) · Defense Technical Information Center (DTIC) · doi:10.21236/ada638366 · Cited by 47
Assessment of Student Learning in Science Simulations and Games
Quellmalz, E., Timms, M. J., & Schneider, S. (2009) · Cited by 43
Assessment of Student Learning.
López, C. L. (1998) · Liberal education · Cited by 42
Detecting functional relations in a computerized discovery environment
Reimann, P. (1991) · Learning and Instruction · doi:10.1016/0959-4752(91)90018-4 · Cited by 39 · cited 1×
“…al. 1995; Njoo and de Jong 1993; van Joolingen and de Jong 1997; Glaser et al. 1992], difficulties with conducting experiments [Glaser et al. 1992; Reimann 1991; Tsirgi 1980; Shute and Glaser 1990; Kuhn 2005; Schunn and Anderson 1998, 1999; Harrison and Schunn 2004; McElhaney and Linn 2008,…”
BackgroundEnsembling predictions of student knowledge within intelligent tutoring systems
Baker, R. S., Pardos, Z. A., Gowda, S. M., Nooraei, B., & Heffernan, N. T. (2011) · International Conference on User Modeling, Adaptation, and Personalization · doi:10.5555/2021855.2021858 · Cited by 38
Self-citationUsing log files to track students' model-based inquiry
Buckley, B. C., Gobert, J. D., & Horwitz, P. (2006) · International Conference of Learning Sciences · Cited by 37
Self-citationUsing Text Replay Tagging to Produce Detectors of Systematic Experimentation Behavior Patterns
Pedro, M. S., Baker, R. S., Montalvo, O., Nakama, A., & Gobert, J. D. (2010) · Educational Data Mining · Cited by 36
Self-citationThe ASSISTment Builder: Towards an Analysis of Cost Effectiveness of ITS Creation.
Heffernan, N. T., Turner, T. E., Lourenco, A. L. N., Macasek, M. A., Nuzzo-Jones, G., & Koedinger, K. R. (2006) · The Florida AI Research Society · Cited by 33
Identifying students' inquiry planning using machine learning
Montalvo, O., Baker, R. S., Pedro, M. S., Nakama, A., & Gobert, J. D. (2010) · Educational Data Mining · Cited by 28
Self-citationDiscovering and Recognizing Student Interaction Patterns in Exploratory Learning Environments
Bernardini, A., & Conati, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_17 · Cited by 26
Measuring students' scientific content and inquiry reasoning
Gotwals, A. W., & Songer, N. B. (2006) · International Conference of Learning Sciences · Cited by 24
Impacts of students' experimentation using a dynamic visualization on their understanding of motion
McElhaney, K. W., & Linn, M. C. (2008) · International Conference of Learning Sciences · Cited by 21
Helping students make controlled experiments more informative
McElhaney, K. W., & Linn, M. C. (2010) · International Conference of Learning Sciences · Cited by 18
Gobert, J. D., & Koedinger, K. R. (2011) · Society for Research on Educational Effectiveness · Cited by 10
Self-citationThe transfer of logically general scientific reasoning skills
Harrison, A. M., & Schunn, C. D. (2004) · eScholarship (California Digital Library) · Cited by 10
Determinants of discovery learning in a complex simulation learning environment
de Jong, T., Beishuizen, J. J., Hulshof, C., Prins, F. J., van Rijn, H., van Someren, M., Veenman, M. V. J., Wilhelm, P., Gärdenfors, P., & Johansson, P. (2005) · VU Research Portal · Cited by 10
Carelessness and Goal Orientation in a Science Microworld
Hershkovitz, A., Wixon, M., Baker, R. S., Gobert, J. D., & Pedro, M. S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21869-9_70 · Cited by 9
Self-citationTracking Students' Inquiry Paths through Student Transition Analysis.
Bachmann, M., Gobert, J. D., & Beck, J. B. (2010) · Cited by 6
Self-citationScience ASSISTments: Tutoring Inquiry Skills in Middle School Students
Richardson, J. O. (2008) · Digital WPI · Cited by 2
Bachmann, M. (2012) · Digital WPI · Cited by 1
Cited by
Challenges and Future Directions of Big Data and Artificial Intelligence in Education
Luan, H., Geczy, P., Lai, H., Gobert, J., Yang, S. J. H., Ogata, H., Baltes, J., Guerra, R., Li, P., & Tsai, C.-C. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580820 · Cited by 667
“With the availability of large quantity and high-quality input training data, machine learning processes can achieve accurate results and facilitate informed decision making (Manyika et al., 2011; Gobert et al., 2012, 2013; Gobert and Sao Pedro, 2017).”
BackgroundSelf-citationEducational data mining: A survey and a data mining-based analysis of recent works
Peña-Ayala, A. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.042 · Cited by 660
“119 Gobert et al. (2012) Machine learning,, probability Predictive Classification”
BackgroundInfluentialAldowah, H., Al-Samarraie, H., & Fauzy, W. M. (2019) · Telematics and Informatics · doi:10.1016/j.tele.2019.01.007 · Cited by 564
Mining Big Data in Education: Affordances and Challenges
Fischer, C., Pardos, Z. A., Baker, R. S., Williams, J. J., Smyth, P., Yu, R., Slater, S., Baker, R., & Warschauer, M. (2020) · Review of Research in Education · doi:10.3102/0091732x20903304 · Cited by 412
“…strategies, affective states, or self-regulated learning (SRL) behaviors, and they are sometimes validated based on real-time observations of student actions (Botelho et al., 2017; DeFalco et al., 2018; Pardos et al., 2014) or retrospective hand coding of data subsets (Gobert et al., 2012).”
BackgroundSelf-citationAcademic and Administrative Role of Artificial Intelligence in Education
Ahmad, S. F., Alam, M. M., Rahmat, M. K., Mubarik, M. S., & Hyder, S. I. (2022) · Sustainability · doi:10.3390/su14031101 · Cited by 309
“Such tools automatically set criteria for the desired candidates and information gathering which provides guidelines for interviews, etc. [11].”
Background
Metadata
- Authors
- Gobert, J. D., Sao Pedro, M. A., Baker, R. S. J. D., Toto, E., & Montalvo, O.
- Affiliations
- Worcester Polytechnic Institute (US)
- Worcester Polytechnic Institute
- Venue
- Zenodo
- Publisher
- Zenodo
- Published
- 2012-10-01
- Volume
- 4
- Issue
- 1
- Pages
- 111-143
- ISSN
- 2157-2100 (ISSN-L)
- OpenAlex
- W1485330246
- Cited by
- 122
- References
- 107
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 5 · 2023: 9 · 2022: 11 · 2021: 7 · 2020: 9 · 2019: 7 · 2018: 11 · 2017: 8 · 2016: 12 · 2015: 11 · 2014: 10 · 2013: 16 · 2012: 2
- FWCI
- 12.28
- Type
- journal article
- Language
- en
- Open access
- Locations
- Journal of Educational Data Mining (publishedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by-nc-nd)
- Zenodo (CERN European Organization for Nuclear Research)
- Educational Data Mining
- Funders
- National Science Foundation (0742503, 0733286, 1008649)
- U.S. Department of Education
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Education › Science Education and Pedagogy
- Keywords
- scientific inquiry skills, evidence-centered design, student model, task modeling, evidence model, hypothesis testing, controlled experiments, scaffolding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.