Mathematical Language Processing: Automatic Grading and Feedback for Open Response Mathematical Questions
Abstract
While computer and communication technologies have provided effective means to scale up many aspects of education, the submission and grading of assessments such as homework assignments and tests remains a weak link. In this paper, we study the problem of automatically grading the kinds of open response mathematical questions that figure prominently in STEM (science, technology, engineering, and mathematics) courses. Our data-driven framework for mathematical language processing (MLP) leverages solution data from a large number of learners to evaluate the correctness of their solutions, assign partial-credit scores, and provide feedback to each learner on the likely locations of any errors. MLP takes inspiration from the success of natural language processing for text data and comprises three main steps. First, we convert each solution to an open response mathematical question into a series of numerical features. Second, we cluster the features from several solutions to uncover the structures of correct, partially correct, and incorrect solutions. We develop two different clustering approaches, one that leverages generic clustering algorithms and one based on Bayesian nonparametrics. Third, we automatically grade the remaining (potentially large number of) solutions based on their assigned cluster and one instructor-provided grade per cluster. As a bonus, we can track the cluster assignment of each step of a multistep solution and determine when it departs from a cluster of correct solutions, which enables us to indicate the likely locations of errors to learners. We test and validate MLP on real-world MOOC data to demonstrate how it can substantially reduce the human effort required in large-scale educational platforms.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
Advances in Neural Information Processing Systems 14
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 9,184 · see citing works
On Spectral Clustering: Analysis and an algorithm
Ng, A. Y., Jordan, M. I., & Weiss, Y. (2001) · Cited by 8,159 · see citing works
Probabilistic Models for Some Intelligence and Attainment Tests
Psychology, M. (1961) · Information and Control · doi:10.1016/s0019-9958(61)80061-2 · Cited by 6,936 · see citing works
Clustering by Passing Messages Between Data Points
Frey, B. J., & Dueck, D. (2007) · Science · doi:10.1126/science.1136800 · Cited by 6,733 · see citing works
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009 · see citing works
Encyclopedia of Machine Learning
Sammut, C., & Webb, G. I. (2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8 · Cited by 3,499 · see citing works
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 16
Storkey, A. (2004) · neural information processing systems · Cited by 2,022 · see citing works
Probabilistic Models for some Intelligence and Attainment Tests.
Wolins, L., Wright, B. D., & Rasch, G. (1982) · Journal of the American Statistical Association · doi:10.2307/2287805 · Cited by 2,009 · see citing works
Handbook of Latent Semantic Analysis
(2007) · Psychology Press eBooks · doi:10.4324/9780203936399 · Cited by 1,571 · see citing works
Steyvers, M., & Griffiths, T. L. (2015) · Routledge eBooks · doi:10.4324/9780203936399.ch21 · Cited by 1,027 · see citing works
Hierarchical Topic Models and the Nested Chinese Restaurant Process
Griffiths, T. L., Jordan, M. I., Tenenbaum, J. B., & Blei, D. M. (2003) · Cited by 912 · see citing works
Estimating a Dirichlet Distribution
Minka, T. P. (2000) · Cited by 758 · see citing works
Building Intelligent Interactive Tutors : student-centered strategies for revolutionizing e-learning
Woolf, B. P. (2009) · HAL · Cited by 716 · see citing works
The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies
Blei, D. M., Griffiths, T. L., & Jordan, M. I. (2010) · Journal of the ACM · doi:10.1145/1667053.1667056 · Cited by 644 · see citing works
A dirichlet multinomial mixture model-based approach for short text clustering
Yin, J., & Wang, J. (2014) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623715 · Cited by 576 · see citing works
The Impact of Self- and Peer-Grading on Student Learning
Sadler, P., & Good, E. (2006) · Educational Assessment · doi:10.1207/s15326977ea1101_1 · Cited by 535 · cited 1× · see citing works
“While peer grading appears to have some pedagogical value for learners [30], each learner typically needs to grade several solutions from other learners for each question they solve, in order to obtain an accurate grade estimate.”
BackgroundTest format and corrective feedback modify the effect of testing on long-term retention
Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007) · European Journal of Cognitive Psychology · doi:10.1080/09541440601056620 · Cited by 513 · see citing works
Automated feedback generation for introductory programming assignments
Singh, R. K., Gulwani, S., & Solar-Lezama, A. (2013) · Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2491956.2462195 · Cited by 388 · see citing works
Demographic differences in how students navigate through MOOCs
Guo, P. J., & Reinecke, K. (2014) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2556325.2566247 · Cited by 319 · see citing works
Powergrading: a Clustering Approach to Amplify Human Effort for Short Answer Grading
Basu, S., Jacobs, C., & Vanderwende, L. (2013) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00236 · Cited by 163 · see citing works
CodeHint: dynamic and interactive synthesis of code snippets
Galenson, J., Reames, P., Bodík, R., et al. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568250 · Cited by 120 · see citing works
Methods for ordinal peer grading
Raman, K., & Joachims, T. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623654 · Cited by 111 · see citing works
Nguyen, A., Piech, C., Huang, J., & Guibas, L. J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568023 · Cited by 108 · see citing works
A system to grade computer programming skills using machine learning
Srikant, S., & Aggarwal, V. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623377 · Cited by 104 · see citing works
Student skill and goal achievement in the mapping with google MOOC
Wilkowski, J., Deutsch, A., & Russell, D. M. (2014) · Proceedings of the first ACM conference on Learning @ scale conference · doi:10.1145/2556325.2566240 · Cited by 89 · see citing works
Divide and correct: using clusters to grade short answers at scale
Brooks, M., Basu, S., Jacobs, C., & Vanderwende, L. (2014) · ACM Conference on Learning @ Scale · doi:10.1145/2556325.2566243 · Cited by 77 · see citing works
Tuned Models of Peer Assessment in MOOCs.
Piech, C., Huang, J., Chen, Z., et al. (2013) · Educational Data Mining · Cited by 72 · see citing works
Feedback generation for performance problems in introductory programming assignments
Gulwani, S., Radiček, I., & Zuleger, F. (2014) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/2635868.2635912 · Cited by 70 · see citing works
Analysis of collaborative writing processes using revision maps and probabilistic topic models
Southavilay, V., Yacef, K., Reimann, P., & Calvo, R. A. (2013) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2460296.2460307 · Cited by 69 · see citing works
Time-varying learning and content analytics via sparse factor analysis
Lan, A., Studer, C., & Baraniuk, R. G. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623631 · Cited by 61 · see citing works
Self-citationNaumowicz, A., & Korniłowicz, A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03359-9_5 · Cited by 60 · see citing works
Automating Hint Generation with Solution Space Path Construction
Rivers, K., & Koedinger, K. R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_41 · Cited by 56 · see citing works
The Naproche Project Controlled Natural Language Proof Checking of Mathematical Texts
Cramer, M., Fisseni, B., Koepke, P., et al. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14418-9_11 · Cited by 51 · see citing works
Correlating skill and improvement in 2 MOOCs with a student's time on tasks
Champaign, J., Colvin, K. F., Liu, A., Fredericks, C., Seaton, D., & Pritchard, D. E. (2014) · Proceedings of the first ACM conference on Learning @ scale conference · doi:10.1145/2556325.2566250 · Cited by 50 · see citing works
CONSTRUCT VALIDITY OF E‐RATER® IN SCORING TOEFL® ESSAYS
Attali, Y. (2007) · ETS Research Report Series · doi:10.1002/j.2333-8504.2007.tb02063.x · Cited by 44 · see citing works
A Canonicalizing Model for Building Programming Tutors
Rivers, K., & Koedinger, K. R. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30950-2_80 · Cited by 24 · see citing works
Automated Scoring of Constructed-Response Literacy and Mathematics Items
Elliot, R. (2011) · Cited by 19 · see citing works
Khan Academy: the world's free virtual school
Dijksman, J. A., & Khan, S. (2011) · Bulletin of the American Physical Society · Cited by 13 · see citing works
Test-size Reduction for Concept Estimation
Vats, D., Studer, C., Lan, A., et al. (2013) · Cited by 10 · see citing works
Self-citationA Bayesian nonparametric approach for the analysis of multiple categorical item responses
Waters, A. E., Fronczyk, K. M., Guindani, M., Baraniuk, R. G., & Vannucci, M. (2014) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2014.07.004 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Proactive and reactive engagement of artificial intelligence methods for education: a review
Mallik, S., & Gangopadhyay, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1151391 · Cited by 139 · see citing works
Automated Feedback for Student Math Responses Based on Multi-Modality and Fine-Tuning
Li, H., Li, C., Xing, W., Baral, S., & Heffernan, N. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636860 · Cited by 29 · see citing works
“Automatic assessment of student answers for geometric theorem proving questions,”
Mendis, C., Lahiru, D., Pamudika, N., Madushanka, S., Ranathunga, S., & Dias, G. (2017) · 2017 Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon.2017.7980520 · Cited by 3 · see citing works
- Authors
- Lan, A. S., Vats, D., Waters, A. E., & Baraniuk, R. G.
- Affiliations
- Rice University (US)
- Venue
- Proceedings of the Second (2015) ACM Conference on Learning @ Scale
- Publisher
- ACM
- Published
- 2015-03-14
- Pages
- 167-176
- OpenAlex
- W2963133108
- Cited by
- 60
- References
- 42
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 11 · 2023: 11 · 2022: 3 · 2021: 8 · 2020: 5 · 2019: 2 · 2018: 7 · 2017: 2 · 2016: 4
- FWCI
- 3.10
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Learning @ Scale (L@S) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- automated grading, open-ended questions, partial credit scoring, solution clustering, Bayesian nonparametric clustering, error localization, student feedback, MOOCs, STEM education, large-scale assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
