Machine Learning–Driven Language Assessment
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index.
Abstract
We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales based on a given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with other high-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
(1991) · Cited by 15,554
The standards for educational and psychological testing.
Eignor, D. R. (2013) · APA handbook of testing and assessment in psychology, Vol. 1: Test theory and testing and assessment in industrial and organizational psychology. · doi:10.1037/14047-013 · Cited by 6,988
Probabilistic Models for Some Intelligence and Attainment Tests
(1961) · Information and Control · doi:10.1016/s0019-9958(61)80061-2 · Cited by 6,939
Common European Framework of Reference for Languages: learning, teaching, assessment
Verhelst, N. D., Avermaet, P. V., Takala, S., Figueras, N., & North, B. (2009) · Ghent University Academic Bibliography (Ghent University) · Cited by 6,120
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,007
A Rasch model for partial credit scoring
Masters, G. N. (1982) · Psychometrika · doi:10.1007/bf02296272 · Cited by 3,782
A Tutorial on the Cross-Entropy Method
de Boer, P.-T., Kroese, D. P., Mannor, S., & Rubinstein, R. Y. (2005) · Annals of Operations Research · doi:10.1007/s10479-005-5724-z · Cited by 3,024
A Rating Formulation for Ordered Response Categories
Andrich, D. (1978) · Psychometrika · doi:10.1007/bf02293814 · Cited by 2,970
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Research works printed but not linked: 21
- Brown J. D., 1989, JALT Journal, 11, 46
- Cambridge English. 2012. Preliminary wordlist.
- Cau S., 2015, Global Times
- Curto P., 2015, Proceedings of the International Conference on Computer Supported Education, 36
- Dudley R., 2016, Reuters Investigates
- Freedle R., 1999, Language Testing, 16, 2
- Graves A., 2014, arXiv
- Heafield K., 2013, Proceedings of the Association for Computational Linguistics (ACL), 690
- Lin W. Y., 2008, Journal of Pan-Pacific Association of Applied Linguistics, 12, 61
- Linacre J. M., 2014, Rasch Measurement Transactions, 27, 1441
- Loukina A., 2016, Proceedings of the International Conference on Computational Linguistics (COLING), 3245
- McNamara D. S., 2011, Annual Meeting of the American Educational Research Association (AERA)
- Mostow J., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 136
- Napoles C., 2010, Proceedings of the Workshop on Computational Linguistics and Writing: Writing Processes and Authoring Aids, 42
- Perez-Beltrachini L., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 147
- Reichert M., 2010, The C-Test: Contributions from Current Research, 205
- Sculley D., 2010, Proceedings of the Conference on Knowledge Discovery and Data Mining (KDD), 979
- Susanti Y., 2016, Proceedings of the International Conference on Computer Supported Education, 267
- Vajjala S., 2016, Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity, 38
- Vajjala S., 2018, Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, 147
- Volodina E., 2016, Proceedings of EUROCALL, 456
Cited by
Items 101–106 of 106
Fostering Creativity Through AI-Empowered Assessment
Gao, X. (2026) · Palgrave studies in teaching and learning Chinese · doi:10.1007/978-3-032-31122-1_5
Xiong, J., Tang, C., Liu, Q., & Li, F. (2026) · Journal of Intelligence · doi:10.3390/jintelligence14090216
Suria Thilagam, P., Rekha, D., Sheeba, K., Mohana Kannan, N., Esakkiammal, S., & Mary, D. S. E. (2026) · International Conference on Green Technologies and Sustainable Solutions (GTSS) · doi:10.1109/gtss68743.2026.11679851
Daruka, A., & Vishwakarma, V. (2026) · International Conference on Modern Sustainable Systems (CMSS) · doi:10.1109/cmss69636.2026.11689420
Hijazi, A. R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7496279
Nguyen, A., Huynh, T.-N., Gardner, L. A., Kremantzis, M. D., & Pham, H. (2026) · Evaluation Review · doi:10.1177/0193841x261501223
Metadata
- Authors
- Settles, B., T. LaFlair, G., & Hagiwara, M.
- Affiliations
- Duolingo (United States) (US)
- Duolingo, Pittsburgh, PA USA. geoff@duolingo.com
- Octanove Labs, Seattle, WA USA. masato@octanove.com
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 247-263
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3019082520
- Cited by
- 106
- References
- 77
- Citations by year
- 2026: 12 · 2025: 21 · 2024: 22 · 2023: 17 · 2022: 18 · 2021: 10 · 2020: 6
- FWCI
- 5.01
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- language proficiency assessment, computerized adaptive testing, item difficulty estimation, test security, item bank, Duolingo English Test, English proficiency, test reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.