Automated Feedback for Student Math Responses Based on Multi-Modality and Fine-Tuning
Open accessAbstract
Open-ended mathematical problems are a commonly used method for assessing students’ abilities by teachers. In previous automated assessments, natural language processing focusing on students’ textual answers has been the primary approach. However, mathematical questions often involve answers containing images, such as number lines, geometric shapes, and charts. Several existing computer-based learning systems allow students to upload their handwritten answers for grading. Yet, there are limited methods available for automated scoring of these image-based responses, with even fewer multi-modal approaches that can simultaneously handle both texts and images. In addition to scoring, another valuable scaffolding to procedurally and conceptually support students while lacking automation is comments. In this study, we developed a multi-task model to simultaneously output scores and comments using students’ multi-modal artifacts (texts and images) as inputs by extending BLIP, a multi-modal visual reasoning model. Benchmarked with three baselines, we fine-tuned and evaluated our approach on a dataset related to open-ended questions as well as students’ responses. We found that incorporating images with text inputs enhances feedback performance compared to using texts alone. Meanwhile, our model can effectively provide coherent and contextual feedback in mathematical settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Heffernan, N. T., & Heffernan, C. L. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0024-x · Cited by 616 · see citing works
Self-citationAutomatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440 · see citing works
CHALLENGES DURING THE PANDEMIC: USE OF E-LEARNING IN MATHEMATICS LEARNING IN HIGHER EDUCATION
Irfan, M., Kusumaningrum, B., Yulia, Y., & Widodo, S. A. (2020) · Infinity Journal · doi:10.22460/infinity.v9i2.p147-158 · Cited by 220 · see citing works
Using synchronous peer tutoring system to promote elementary students’ learning in mathematics
Tsuei, M. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.11.025 · Cited by 107 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 86 · see citing works
Jacobs, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., & Sumner, T. (2022) · Teaching and Teacher Education : An International Journal of Research and Studies · doi:10.1016/j.tate.2022.103631 · Cited by 84 · see citing works
Cook, S. C., Collins, L. W., Morin, L. L., & Riccomini, P. (2020) · Learning Disability Quarterly · doi:10.1177/0731948718823080 · Cited by 77 · see citing works
Lan, A. S., Vats, D., Waters, A. E., & Baraniuk, R. G. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724664 · Cited by 60 · see citing works
Automated Grading Systems for Programming Assignments: A Literature Review
Aldriye, H., Alkhalaf, A., & Alkhalaf, M. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0100328 · Cited by 55 · see citing works
The automated grading of student open responses in mathematics
Erickson, J. A., Botelho, A. F., McAteer, S., Varatharaj, A., & Heffernan, N. T. (2020) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3375462.3375523 · Cited by 53 · see citing works
Self-citationUsing BERT and XLNET for the Automatic Short Answer Grading Task
Ghavidel, H. A., Zouaq, A., & Desmarais, M. C. (2020) · International Conference on Computer Supported Education · doi:10.5220/0009422400580067 · Cited by 47 · see citing works
Automated Grading for Handwritten Answer Sheets using Convolutional Neural Networks
Shaikh, E., Mohiuddin, I., Manzoor, A., et al. (2019) · International Conference on New Trends in Computing Sciences (ICTCS) · doi:10.1109/ictcs.2019.8923092 · Cited by 42 · see citing works
Bahar, A. K., & Maker, C. J. (2020) · Journal of Advanced Academics · doi:10.1177/1932202x20906130 · Cited by 41 · see citing works
Academic development of multimodal learning analytics: a bibliometric analysis
Pei, B., Xing, W., & Wang, M. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2021.1936075 · Cited by 38 · see citing works
Self-citationNguyen, C. T., Khuong, V. T. M., Nguyen, H. T., et al. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.12.015 · Cited by 34 · see citing works
Du, H., Xing, W., & Pei, B. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2021.1993932 · Cited by 27 · see citing works
Self-citationTowards the Future of AI-Augmented Human Tutoring in Math Learning
Aleven, V., Baraniuk, R. G., Brunskill, E., Crossley, S. A., Demszky, D., Fancsali, S. E., Gupta, S., Koedinger, K. R., Piech, C., RITTER, S., et al. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-36336-8_3 · Cited by 25 · see citing works
Self-citationLi, C., Xing, W., & Leite, W. L. (2021) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3448139.3448200 · Cited by 17 · see citing works
Self-citationImproving Automated Scoring of Student Open Responses in Mathematics
Baral, S., Botelho, A. F., Erickson, J. A., Benachamardi, P., & Heffernan, N. T. (2021) · Educational Data Mining · Cited by 17 · see citing works
Self-citationLevonian, Z., Li, C., Zhu, W., Gade, A., Henkel, O., Postle, M.-E., & Xing, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.03184 · Cited by 17 · see citing works
Self-citationAutomatic Short Math Answer Grading via In-context Meta-learningPreprint
Zhang, M., Baral, S., Heffernan, N. T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.15219 · Cited by 13 · see citing works
Self-citationLachner, A., & Nückles, M. (2015) · Instructional Science · doi:10.1007/s11251-015-9365-6 · Cited by 11 · see citing works
Baral, S., Seetharaman, K., Botelho, A. F., Wang, A., Heineman, G. T., & Heffernan, N. T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11644-5_68 · Cited by 11 · see citing works
Self-citationGold, C., & Zesch, T. (2020) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr2020.2020.00054 · Cited by 8 · see citing works
One on One Numeracy Intervention: A Pilot Project in Low SES Communities.
Thornton, S., Quinane, M., Galluzzo, G., et al. (2010) · Mathematics Education Research Group of Australasia · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 10
- [4]4. Sami Baral, Anthony Botelho, Abhishek Santhanam, Ashish Gurung, Li Cheng, and Neil Heffernan. 2023. Auto-scoring Student Responses with Images in Math- ematics. In The Proceedings of the 16th International Conference on Educational Data Mining.
- [8]8. Anthony Botelho, Sami Baral, John A Erickson, Priyanka Benachamardi, and Neil T Heffernan. 2023. Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics. Journal of Computer Assisted Learning (2023).
- [14]14. Sean P Goggins, Wanli Xing, Xin Chen, Bodong Chen, and Bob Wadholm. 2015. Learning Analytics at" Small" Scale: Exploring a Complexity-Grounded Model for Assessment Automation. J. Univers. Comput. Sci. 21, 1 (2015), 66-92.
- [16]16. Ashish Gurung, Anthony Botelho, Russell Thompson, Adam Sales, Sami Baral, and Neil Heffernan. 2022. Considerate, Unfair, or Just Fatigued? Examining Factors that Impact Teacher. In Proceedings of the 30th International Conference on Computers in Education. Asia-Pacific Society for Computers in Education.
- [20]20. Giuseppe Jurman, Samantha Riccadonna, Roberto Visintainer, and Cesare Furlanello. 2009. Canberra distance on ranked lists. In Proceedings of advances in ranking NIPS 09 workshop. Citeseer, 22-27.
- [25]25. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning. PMLR, 12888-12900.
- [31]31. Raysa Rivera-Bergollo, Sami Baral, Anthony Botelho, and Neil Heffernan. 2022. Leveraging auxiliary data from similar problems to improve automatic open response scoring. In Conference for Educational Data Mining.
- [36]36. Khoon Yoong Wong, Kwang-Shin Oh, Qiu Ting Yvonne Ng, and Jim Siew Kuan Cheong. 2012. Linking IT-based semi-automatic marking of student mathematics responses and meaningful feedback to pedagogical objectives. Teaching Math- ematics and its Applications: An International Journal of the IMA 31, 1 (2012), 57-63.
- [37]37. Benjamin D Wright. 1977. Solving measurement problems with the Rasch model. Journal of educational measurement (1977), 97-116.
- [39]39. Yuan Zhang, Chen Lin, and Min Chi. 2020. Going deeper: Automatic short- answer grading by combining student and question models. User modeling and user-adapted interaction 30 (2020), 51-80.
Cited by
Sorted by citations
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119 · see citing works
GenAI Analytics and Pedagogical Configurations: Featuring New Generative AI Systems in Education
Ortega-Arranz, A., Topali, P., García-Zarza, P., Bote-Lorenzo, M. L., Asensio-Pérez, J. I., & Molenaar, I. (2026) · Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference · doi:10.1145/3785022.3785054 · Cited by 2 · see citing works
- Authors
- Li, H., Li, C., Xing, W., Baral, S., & Heffernan, N.
- Affiliations
- University of Florida (US)
- University of Utah (US)
- Worcester Polytechnic Institute (US)
- Venue
- Proceedings of the 14th Learning Analytics and Knowledge Conference
- Publisher
- ACM
- Published
- 2024-03-18
- Pages
- 763-770
- OpenAlex
- W4392445324
- Cited by
- 29
- References
- 26
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 6
- FWCI
- 8.57
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Learning Analytics and Knowledge (LAK) (publishedVersion)
- Funders
- Jaffe Foundation (AGR00026932)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- automated feedback, multimodal learning, automated grading, multi-task learning, BLIP, fine-tuning, scaffolding, student comments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
