CoChat
Sign Up
PreprintCited by 47Topic ModelingIntelligent Tutoring Systems and Adaptive LearningMathematics, Computing, and Information Processing

MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education

Open access
Shen, Jia Tracy, Yamashita, Michiharu, Prihar, Ethan, Heffernan, Neil, Wu, Xintao, Graff, Ben, Lee, Dongwon·2021·arXiv
DOI 10.48550/arxiv.2106.07340arXiv 2106.07340
Open this paper in CoChat — free accountOpen PDF

Abstract

Since the introduction of the original BERT (i.e., BASE BERT), researchers have developed various customized BERT models with improved performance for specific domains and tasks by exploiting the benefits of transfer learning. Due to the nature of mathematical texts, which often use domain specific vocabulary along with equations and math symbols, we posit that the development of a new BERT model for mathematics would be useful for many mathematical downstream tasks. In this resource paper, we introduce our multi-institutional effort (i.e., two learning platforms and three academic institutions in the US) toward this need: MathBERT, a model created by pre-training the BASE BERT model on a large mathematical corpus ranging from pre-kindergarten (pre-k), to high-school, to college graduate level mathematical content. In addition, we select three general NLP tasks that are often used in mathematics education: prediction of knowledge component, auto-grading open-ended Q&A, and knowledge tracing, to demonstrate the superiority of MathBERT over BASE BERT. Our experiments show that MathBERT outperforms prior best methods by 1.2-22% and BASE BERT by 2-8% on these tasks. In addition, we build a mathematics specific vocabulary 'mathVocab' to train with MathBERT. We discover that MathBERT pre-trained with 'mathVocab' outperforms MathBERT trained with the BASE BERT vocabulary (i.e., 'origVocab'). MathBERT is currently being adopted at the participated leaning platforms: Stride, Inc, a commercial educational resource provider, and ASSISTments.org, a free online educational platform. We release MathBERT for public usage at: https://github.com/tbs17/MathBERT.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.07340

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In long-context applications such as legal analysis, healthcare, and education, documents from a common pool are often reused across prompts.

    “Additionally, in long-context applications such as legal analysis [10], healthcare [50], and education [47], documents from a common pool are often reused across prompts.”

    From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [47]

    The Introduction (p. 1) discusses mathematics-education tasks, and Table 3 (p. 4) lists sources used for MathBERT’s training corpus. Neither those passages nor searches for document reuse across prompts address whether educational applications repeatedly draw documents from a common pool. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • CDN caching systems such as Akamai and Cloudflare target network-level efficiency rather than computational cost, making them unsuitable for latency-sensitive, variable-cost KV cache management in LLM inference.

    “CDN caching systems, like Akamai [46] and Cloudflare [45], focuse on network level efficiency, rather than computational cost, making them inherently unsuitable for managing the latency sensitive, variable computational cost KV cache management in LLM inference.”

    From HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems (2025) · cites this paper as [46]

    The Introduction (pp. 1–2) discusses MathBERT and mathematics-education tasks, not CDN caching or LLM inference. Searches for Akamai, Cloudflare, network-level efficiency, and latency-sensitive KV-cache management found no passage addressing the claim. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. [3][3] Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, and Jongwuk Lee. 2021. MelBERT : Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics . -
  2. [4][4] Albert T Corbetr and John R Anderson. 1995. Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction 4 (1995), 253-278. -
  3. [6][6] John A Erickson, Anthony F Botelho, Steven Mcateer, Ashvini Varatharaj, and Neil T Heffernan. 2020. The Automated Grading of Student Open Responses in Mathematics ACM Reference Format. In Proceedings of the 10th Learning Analytics and Knowledge Conference . -
  4. [7][7] Weiwei Guo, Xiaowei Liu, Sida Wang, Huiji Gao, Ananth Sankar, Zimeng Yang, Qi Guo, Liang Zhang, Bo Long, Bee-Chung Chen, and Deepak Agarwal. 2020. DeText: A Deep Text Ranking Framework with BERT. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management . -
  5. [10][10] Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Finetuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics . 328-339. -
  6. [11][11] Kexin Huang and Jaan Altosaar. [n.d.]. ClinicalBert: Modeling Clinical Notes and Predicting Hospital Readmission. In arXiv preprint arXiv:1904.05342v2 . - · arXiv
  7. [13][13] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Data and text mining BioBERT: a pretrained biomedical language representation model for biomedical text mining. Bioinformatics (2020), 1234-1240. https://doi.org/10.1093/bioinformatics/btz682 - · DOI · Paper page
  8. [14][14] Youngnam Lee, Youngduck Choi, Junghyun Cho, Alexander R Fabbri, Hyunbin Loh, Chanyou Hwang, Yongku Lee, Sang-Wook Kim, and Dragomir Radev. 2019. Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess. In arXiv preprint arXiv:1906.10910v2 . - · arXiv
  9. [15][15] Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. EKT: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1 (2019), 100-115. -
  10. [16][16] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, and Paul G Allen. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. In arXiv preprint arXiv:1907.11692v1 . - · arXiv
  11. [17][17] Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020. FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence Special Track on AI in FinTech . -
  12. [18][18] Shalini Pandey and George Karypis. 2019. A Self-Attentive model for Knowledge Tracing. In Proceedings of The 12th International Conference on Educational Data Mining . -
  13. [19][19] Zachary A Pardos. 2017. Imputing KCs with Representations of Problem Content and Context. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization . 148-155. https://doi.org/10.1145/3079628.3079689 - · DOI
  14. [21][21] Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. [n.d.]. MathBERT: A PreTrained Model for Mathematical Formula Understanding. In arXiv preprint arXiv:2105.00377v1 . - · arXiv
  15. [23][23] Chris Piech, Jonathan Spencer, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, Jascha Sohl-Dickstein, Stanford University, and Khan Academy. 2015. Deep Knowledge Tracing. In Advances in Neural Information Processing Systems . -
  16. [24][24] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing . 3982-3992. -
  17. [25][25] Carolyn Rosé, Pinar Donmez, Gahgene Gweon, Andrea Knight, Brian Junker, William Cohen, Kenneth Koedinger, and Neil Heffernan. 2005. Automatic and Semi-Automatic Skill Coding With a View Towards Supporting On-Line Assessment. In Proceedings of the conference on Artificial Intelligence in Education . 571-578. -
  18. [26][26] Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil Heffernan, Xintao Wu, Sean Mcgrew, and Dongwon Lee. 2021. Classifying Math Knowledge Components via Task-Adaptive Pre-Trained BERT. In Proceedings of the Conference on Artificial Intelligence in Education . -
  19. [27][27] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to Fine-Tune BERT for Text Classification? Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11856 LNAI, 2 (2019), 194-206. -
  20. [28][28] Nguyen Thai-Nghe, Lucas Drumond, Artus Krohn-Grimberghe, and Lars SchmidtThieme. 2010. Recommender system for predicting student performance. Procedia Computer Science 1, 2 (2010), 2811-2819. -
  21. [30][30] Denghui Zhang, Zixuan Yuan, Yanchi Liu, Zuohui Fu, Fuzhen Zhuang, Pengyang Wang, Haifeng Chen, and Hui Xiong. 2020. E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce. In arXiv preprint arXiv:2009.02835v2 . - · arXiv
  22. [31][31] Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. 2017. Dynamic KeyValue Memory Networks for Knowledge Tracing. In International World Wide Web Conference Committee (IW3C2) .

Cited by

Sorted by citations

Authors
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., & Lee, D.
Affiliations
  • Pennsylvania State University (US)
  • Worcester Polytechnic Institute (US)
  • University of Arkansas at Fayetteville (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.07340
arXiv
2106.07340
OpenAlex
W3167022671
Cited by
47
References
5
Citations by year
2026: 4 · 2025: 8 · 2024: 18 · 2023: 9 · 2022: 6 · 2021: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1940236, 1940076, 1822830, 1917713, 1759229, 1917808, 1940093, 1903304, 1931523)
  • Office of Naval Research (N00014-18, N00014, N00014-18-1-2768)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Mathematics, Computing, and Information Processing
Keywords
BERT pre-training, mathematics education, mathematical texts, domain-specific terminology, knowledge tracing, transfer learning, ASSISTments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review