Bridging large language model disparities: skill tagging of multilingual educational content
Open accessAbstract
The adoption of large language models (LLMs) in education holds much promise. However, like many technological innovations before them, adoption and access can often be inequitable from the outset, creating more divides than they bridge. In this paper, we explore the magnitude of the country and language divide in the leading open‐source and proprietary LLMs with respect to knowledge of K‐12 taxonomies in a variety of countries and their performance on tagging problem content with the appropriate skill from a taxonomy, an important task for aligning open educational resources and tutoring content with state curricula. We also experiment with approaches to narrowing the performance divide by enhancing LLM skill tagging performance across four countries (the USA, Ireland, South Korea and India–Maharashtra) for more equitable outcomes. We observe considerable performance disparities not only with non‐English languages but with English and non‐US taxonomies. Our findings demonstrate that fine‐tuning GPT‐3.5 with a few labelled examples can improve its proficiency in tagging problems with relevant skills or standards, even for countries and languages that are underrepresented during training. Furthermore, the fine‐tuning results show the potential viability of GPT as a multilingual skill classifier. Using both an open‐source model, Llama2‐13B, and a closed‐source model, GPT‐3.5, we also observe large disparities in tagging performance between the two and find that fine‐tuning and…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multilingual imbalances in AI systems affect confidence and usability.
“Kwak and Pardos (2024) who show that multilingual imbalances in AI systems affect the confidence and usability of the system further supported by systemic biases present in the AI models Lee et al. (2024).”
From Mapping the impact of generative AI in higher education: A scoping review of psychological and equity dimensions. (2026) · cites this paper as Kwak and Pardos (2024)
Cites
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · see citing works
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993 · see citing works
Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012) · Cognitive Science · doi:10.1111/j.1551-6709.2012.01245.x · Cited by 772 · see citing works
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · see citing works
Cultural bias and cultural alignment of large language models
Tao, Y., Viberg, O., Baker, R. S., & Kizilcec, R. F. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae346 · Cited by 383 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · arXiv · doi:10.48550/arxiv.2304.03738 · Cited by 282
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Vaughan, J. W. (2023) · arXiv · doi:10.48550/arxiv.2306.01941 · Cited by 209 · see citing works
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169 · see citing works
Learning gain differences between ChatGPT and human tutor generated algebra hints
Pardos, Z. A., & Bhandari, S. (2023) · arXiv · doi:10.48550/arxiv.2302.06871 · Cited by 70 · see citing works
Self-citationKnowledge of cultural moral norms in large language models
Ramezani, A., & Xu, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.26 · Cited by 54 · see citing works
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Cultural bias and cultural alignment of large language models
Tao, Y., Viberg, O., Baker, R. S., & Kizilcec, R. F. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae346 · Cited by 383 · see citing works
“This can improve cultural alignment [41, 34] but requires resources that render this approach accessible to only a few.”
BackgroundThe life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119 · see citing works
Ni, W., & Li, J. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1856854 · Cited by 6 · see citing works
Tran, L. K., Nguyen, L. T., & Pham, K. M. (2026) · Student Success · doi:10.63608/ssj.4067
- Authors
- Kwak, Y., & Pardos, Z. A.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Pardos, Zachary A.
- Venue
- British Journal of Educational Technology
- Publisher
- Wiley
- Published
- 2024-05-08
- Volume
- 55
- Issue
- 5
- Pages
- 2039-2057
- ISSN
- 0007-1013 (ISSN-L)
- 1467-8535
- OpenAlex
- W4396735702
- Cited by
- 25
- References
- 17
- Citations by year
- 2026: 11 · 2025: 9 · 2024: 5
- FWCI
- 3.68
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- British Journal of Educational Technology (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- large language models, educational taxonomy, multilingual LLMs, fine-tuning, GPT-3.5, Llama 2, open educational resources, language bias, underrepresented languages, prompt engineering, open-source vs closed-source models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
