Statistical Techniques to Explore the Quality of Constraints in Constraint-Based Modeling Environments
Open accessThe DOI matches its Crossref record; 40 of the 42 references it cites were found in a registry or an index.
Abstract
One of the most popular student modeling approaches is Constraint-Based Modeling (CBM). It is an efficient approach that can be easily applied inside an Intelligent Tutoring System (ITS). Even with these characteristics, building new ITSs requires carefully designing the domain model to be taught because different sources of errors could affect the efficiency of the system. In this paper a novel mechanism for studying the quality of the elements in the domain model of CBM systems is presented. This mechanism combines CBM with the Item Response Theory (IRT), a data-driven technique for automatic assessment. The goal is to improve the quality of the elements that are used in problem solving environments for assessment or instruction. In this paper we propose a set of statistical techniques, i.e., the analysis of the point-biserial correlation, the Cronbach’s alpha and the information function, to explore the quality of constraints. Two different tools have been used to test this approach: a problem solving environment designed to assess students in project investment analysis; and an independent component that performs assessments using CBM and IRT. Results suggest that the three methods produce consistent diagnosis and may be complementary in some cases. In the experiments we have carried out they were able to detect faulty, bad and good quality constraints.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures













Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“There is only one study [18] that clearly addresses aspects related to pedagogical models that support the work to be developed subsequently.”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“Only one study [18] makes clear the use of the pedagogical model behind the choice of activities that AI will then evaluate.”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · cited 2×
“Cronbach ' s Alpha ( Cronbach 1951 ) is a measure of the internal consistency of a set of items .”
BackgroundStatistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022 · cited 10×
“(4), where A(i,θ) represents the area of the IIF of the item i with respect to the knowledge.”
InfluentialRozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.3102/00028312006001112 · Cited by 8,014
Cognitive Architecture and Instructional Design
Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998) · Educational Psychology Review · doi:10.1023/a:1022193728205 · Cited by 6,023 · cited 1×
“The system interface tries to reduce the cognitive overload (Sweller et al. 1998), otherwise calculus inherent in these kinds of problems would affect the student’s working memory.”
BackgroundFundamentals of item response theory
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4185 · Cited by 3,414
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324 · cited 2×
“Knowledge Tracing ( KT ) ( Corbett and Anderson 1995 ) is a technique to model student knowledge and learning over time that have been primarily included in CT .”
Method(1968) · Addison-Wesley eBooks · Cited by 2,001
Cognitive Tutors: Lessons Learned
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · Cited by 1,982 · cited 1×
“CT (Anderson et al. 1995) are based on a theory that claims that there are two long-term memory stores: declarative and procedural.”
BackgroundThe basics of item response theory
Baker, F. B. (1985) · Cited by 1,362
“Learning factors analysis-a general method for cognitive model evaluation and improvement,”
Cen, H., Koedinger, K., & Junker, B. (2006) · Lecture notes in computer science · doi:10.1007/11774303_17 · Cited by 572 · cited 3×
“Learning Factor Analysis (Cen et al. 2006), propose a semi-automated method combining a multiple logistic regression model of learning with combinatory techniques to calculate the parameters of the model.”
MethodDeMars, C. (2010) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195377033.001.0001 · Cited by 497
A BRIEF INTRODUCTION TO EVIDENCE‐CENTERED DESIGN
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003) · ETS Research Report Series · doi:10.1002/j.2333-8504.2003.tb01908.x · Cited by 473 · cited 2×
“3 Evidence-centered design framework (adapted from (Mislevy et al. 2004)) 28 Int J Artif Intell Educ (2013) 23:22–49”
BackgroundLearning from performance errors.
Ohlsson, S. (1996) · Psychological Review · doi:10.1037//0033-295x.103.2.241 · Cited by 387
A method of scaling psychological and educational tests.
Thurstone, L. L. (1925) · Journal of Educational Psychology · doi:10.1037/h0073357 · Cited by 298 · cited 1×
“IRT was conceived by Thurstone (1925) as a theory used in testing environments to measure certain latent traits, such as the student's knowledge.”
MethodConstraint-Based Student Modeling
Ohlsson, S. (1994) · Student Modelling: The Key to Individualized Knowledge-Based Instruction · doi:10.1007/978-3-662-03037-0_7 · Cited by 227
KT-IDEM: introducing item difficulty to the knowledge tracing model
Pardos, Z. A., & Heffernan, N. T. (2011) · International Conference on User Modeling, Adaptation, and Personalization · doi:10.5555/2021855.2021877 · Cited by 206
SIETTE: A Web-Based Tool for Adaptive Testing
Conejo, R., Guzmán, E., Millán, E., Trella, M., Pérez-de-la-Cruz, J.-L., & Ríos, A. (2004) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2004-14(1)03 · Cited by 193 · cited 3×
“To perform the experiment we used three systems , each one for a different purpose : the first one is Siette ( Conejo et al .”
MethodSelf-citationOptimising ITS Behaviour with Bayesian Networks and Decision Theory
Mayo, M., & Mitrović, A. (2001) · Research Commons (University of Waikato) · Cited by 192
SIETTE: A Web-Based Tool for Adaptive Testing
Conejo, R., Guzmán, E., Millán, E., Trella, M., Pérez-de-la-Cruz, J.-L., & Ríos, A. (2004) · doi:10.5555/1434852.1434855 · Cited by 184
Self-citationKT-IDEM: Introducing Item Difficulty to the Knowledge Tracing Model
Pardos, Z. A., & Heffernan, N. T. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-22362-4_21 · Cited by 174 · cited 1×
“Pardos and Heffernan ( 2011 ) have extended the standard KT model to take into account different item difficulties .”
BackgroundA Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling
Mitrovic, A., Koedinger, K. R., & Martin, B. (2003) · Lecture Notes in Computer Science · doi:10.1007/3-540-44963-9_42 · Cited by 167 · cited 1×
“It is easier to apply than other approaches, such as Model Tracing (Mitrovic et al. 2003) since CBM does not require the identification of all possible steps a student could take to reach a solution to a problem.”
MethodFifteen years of constraint-based tutors: what we have achieved and where we are going
Mitrović, A. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9105-9 · Cited by 151 · cited 2×
“As collected in (Mitrovic 2012; Mitrovic et al. 2007), results obtained during the last 16 years from research in a number of areas within ITSs (affect, collaboration, open learner modeling, tutorial dialogs, etc.”
BackgroundUsing Evaluation to Shape ITS Design: Results and Experiences with SQL-Tutor
Mitrović, A., Martin, B., & Mayo, M. (2002) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1015022619307 · Cited by 141 · cited 1×
“Probabilistic methods based on Bayesian networks have also been implemented (Mayo and Mitrovic 2001; Mitrovic et al. 2002).”
MethodComparing Two IRT Models for Conjunctive Skills
Cen, H., Koedinger, K. R., & Junker, B. W. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69132-7_111 · Cited by 136 · cited 1×
“In the area of CT and KT there are some proposals of multidimensional IRT models (Cen et al. 2008).”
BackgroundIntelligent Tutors for All: The Constraint-Based Approach
Mitrovic, A., Martin, B., & Suraweera, P. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.74 · Cited by 134 · cited 2×
“As collected in (Mitrovic 2012; Mitrovic et al. 2007), results obtained during the last 16 years from research in a number of areas within ITSs (affect, collaboration, open learner modeling, tutorial dialogs, etc.”
BackgroundEvaluating and improving adaptive educational systems with learning curves
Martin, B., Mitrović, A., Koedinger, K. R., & Mathan, S. A. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9084-2 · Cited by 98 · cited 2×
“They concluded in these studies that groups of constraints, linked to more general concepts, would be more effective for learning than single constraints in some cases and, in others, splitting constraints into more specific concepts was more effective (Martin and Mitrovic 2005, 2006; Martin et al. 2011).”
BackgroundMeyer, J. P., & Shi, Z. (2013) · Research & Practice in Assessment · Cited by 64
Improving Student Performance Using Self-Assessment Tests
Guzman, E., Conejo, R., & Perez-de-la-Cruz, J.-L. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.71 · Cited by 62 · cited 3×
“To perform the experiment we used three systems , each one for a different purpose : the first one is Siette ( Conejo et al .”
MethodSelf-citationAssessment and Learning of Mathematics.
Leder, G. C. (1992) · Cited by 61
The Point Biserial Coefficient of Correlation
Lev, J. (1949) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730103 · Cited by 58 · cited 17×
“PBC performs very well detecting faulty constraints , but does not provide much information about good quality constraints .”
BackgroundInfluentialRevisiting Ill-Definedness and the Consequences for ITSs
Antonija, M., & Weerasinghe, A. (2009) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-60750-028-5-375 · Cited by 50 · cited 1×
“In the particular case of the PIPSE system , according to the categorization made by Mitrovic and Weerasinghe ( 2009 ) , we are dealing with a well - defined domain where problems as well as tasks are well - defined .”
MethodAuthoring Constraint-Based Tutors in ASPIRE
Mitrović, A., Suraweera, P., Martin, B., Zakharov, K. P., Milik, N., & Holland, J. (2006) · Lecture notes in computer science · doi:10.1007/11774303_5 · Cited by 47 · cited 1×
“Designing the constraints set in new learning environments can be performed very easily using authoring tools such as ASPIRE (Mitrovic et al. 2006) as no programming skills are needed.”
MethodConstraint-based knowledge representation for individualized instruction
Ohlsson, S., & Mitrović, A. (2006) · Computer Science and Information Systems · doi:10.2298/csis0601001s · Cited by 39 · cited 1×
“Although CBM establishes how to model the domain and the student in an easy and efficient way, it suffers from a major drawback which its authors pointed out in (Ohlsson and Mitrovic 2006): it is necessary to have a long-term student model in order to improve the pedagogical actions a tutor could take.”
BackgroundERIC clearinghouse on assessment and evaluation
(2000) · Choice Reviews Online · doi:10.5860/choice.37sup-495 · Cited by 38
Student Knowledge Diagnosis Using Item Response Theory and Constraint-Based Modeling
Gálvez, J., Guzmán, E., Conejo, R., & Millán, E. (2009) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-60750-028-5-291 · Cited by 20
Self-citationThe Effect of Adapting Feedback Generality in ITS
Martin, B., & Mitrović, A. (2006) · Lecture notes in computer science · doi:10.1007/11768012_21 · Cited by 13 · cited 2×
“…in these studies that groups of constraints , linked to more general concepts , would be more effective for learning than single constraints in some cases and , in others , splitting constraints into more specific concepts was more effective ( Martin and Mitrovic 2005 , 2006 ; Martin et al .”
BackgroundUsing Learning Curves to Mine Student Models
Martin, B., & Mitrović, A. (2005) · Lecture notes in computer science · doi:10.1007/11527886_12 · Cited by 8 · cited 2×
“…in these studies that groups of constraints , linked to more general concepts , would be more effective for learning than single constraints in some cases and , in others , splitting constraints into more specific concepts was more effective ( Martin and Mitrovic 2005 , 2006 ; Martin et al .”
BackgroundData-Driven Student Knowledge Assessment through Ill-Defined Procedural Tasks
Gálvez, J., Guzmán, E., & Conejo, R. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14264-2_24 · Cited by 4
Self-citationA SOA-Based Framework for Constructing Problem Solving Environments
Gálvez, J., Guzmán, E., & Conejo, R. (2008) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2008.230 · Cited by 4 · cited 1×
“The CBM-Engine is a Service Oriented Architecture-based component following the same idea as (Gálvez et al. 2008) that implements CBM with IRT assessment.”
MethodSelf-citationExploring Quality of Constraints for Assessment in Problem Solving Environments
Gálvez, J., Guzmán, E., & Conejo, R. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30950-2_41 · Cited by 3
Self-citation
Research works printed but not linked: 2
- Khan, M. & Jain, P. (1999). Theory and Problems in Financial Management. McGraw Hill Education.
- Pavlik, P. I., Cen, H. & Koedinger, K. R. (2009). Performance factors analysis - a new alternative to knowledge tracing. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, pp. 531–538.
Cited by
Artificial Intelligence for Student Assessment: A Systematic Review
González-Calatayud, V., Prendes-Espinosa, P., & Roig-Vila, R. (2021) · Applied Sciences · doi:10.3390/app11125467 · Cited by 503
“Only one study [18] makes clear the use of the pedagogical model behind the choice of activities that AI will then evaluate.”
BackgroundInfluentialThe SIETTE Automatic Assessment Environment
Conejo, R., Guzmán, E., & Trella, M. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0078-4 · Cited by 34
“Furthermore, we have recently added composite items which, in combination with external items and CBM techniques, have allowed us to assess declarative knowledge (Gálvez 2009, 2012; Gálvez et al. 2009a, b, c, 2010, 2012, 2013) in problem-solving items.”
MethodInfluentialSelf-citationGalindo‐Domínguez, H., Delgado, N., de la Maza, M. S., & Expósito, E. (2024) · Edutec Revista Electrónica de Tecnología Educativa · doi:10.21556/edutec.2024.89.3509 · Cited by 18
Improving student model for individualized learning
Chen, Y. (2015) · theses.fr (ABES) · doi:10.70675/d7c5845dzb4eez4709z9979z4964b5e5ca91 · Cited by 1
“Gálvez et al. (2013) combined an IRT model with Constraint-Based Modeling (CBM) for student modeling.”
A Study of Possible AI Applications within Childhood Educations
Kubde, B. R., & Dekate, M. V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27160-0_6
Metadata
- Authors
- Gálvez, J., Conejo, R., & Guzmán, E.
- Affiliations
- Universidad de Málaga (ES)
- Corresponding author
- Gálvez, Jaime
- Venue
- International Journal of Artificial Intelligence in Education
- Publisher
- Elsevier BV
- Published
- 2013-11
- Volume
- 23
- Issue
- 1-4
- Pages
- 22-49
- ISSN
- 1560-4292 (ISSN-L)
- OpenAlex
- W2028383195
- Cited by
- 6
- References
- 42
- Citations by year
- 2026: 1 · 2024: 1 · 2021: 1 · 2015: 3
- FWCI
- 1.11
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor, 4555 d)
- Locations
- International Journal of Artificial Intelligence in Education (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- constraint-based modeling, student modeling, intelligent tutoring systems, item response theory, point-biserial correlation, Cronbach's alpha, information function, problem solving environments, automated assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.