Machine Learning–Driven Language Assessment
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index.
Abstract
We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales based on a given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with other high-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · cited 3×
“This index is isomorphic with the AUC (Fawcett, 2006), which we use as the graded response r i .”
BackgroundMethodInfluential(1991) · Cited by 15,554
The standards for educational and psychological testing.
Eignor, D. R. (2013) · APA handbook of testing and assessment in psychology, Vol. 1: Test theory and testing and assessment in industrial and organizational psychology. · doi:10.1037/14047-013 · Cited by 6,988 · cited 1×
“Lane et al. (2016) and the Standards for Educational and Psychological Testing (AERA et al., 2014) describe many of the procedures and requirements for planning, creating, revising, administering, analyzing, and reporting on high-stakes tests and their development.”
MethodProbabilistic Models for Some Intelligence and Attainment Tests
(1961) · Information and Control · doi:10.1016/s0019-9958(61)80061-2 · Cited by 6,939
Common European Framework of Reference for Languages: learning, teaching, assessment
Verhelst, N. D., Avermaet, P. V., Takala, S., Figueras, N., & North, B. (2009) · Ghent University Academic Bibliography (Ghent University) · Cited by 6,122
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,007 · cited 1×
“In psychometrics, item response theory (IRT) is a paradigm for designing and scoring measures of ability and other cognitive variables (Lord, 1980).”
BackgroundA Rasch model for partial credit scoring
Masters, G. N. (1982) · Psychometrika · doi:10.1007/bf02296272 · Cited by 3,782
A Tutorial on the Cross-Entropy Method
de Boer, P.-T., Kroese, D. P., Mannor, S., & Rubinstein, R. Y. (2005) · Annals of Operations Research · doi:10.1007/s10479-005-5724-z · Cited by 3,024
A Rating Formulation for Ordered Response Categories
Andrich, D. (1978) · Psychometrika · doi:10.1007/bf02293814 · Cited by 2,970
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Research works printed but not linked: 21
- Brown J. D., 1989, JALT Journal, 11, 46
- Cambridge English. 2012. Preliminary wordlist.
- Cau S., 2015, Global Times
- Curto P., 2015, Proceedings of the International Conference on Computer Supported Education, 36
- Dudley R., 2016, Reuters Investigates
- Freedle R., 1999, Language Testing, 16, 2
- Graves A., 2014, arXiv
- Heafield K., 2013, Proceedings of the Association for Computational Linguistics (ACL), 690
- Lin W. Y., 2008, Journal of Pan-Pacific Association of Applied Linguistics, 12, 61
- Linacre J. M., 2014, Rasch Measurement Transactions, 27, 1441
- Loukina A., 2016, Proceedings of the International Conference on Computational Linguistics (COLING), 3245
- McNamara D. S., 2011, Annual Meeting of the American Educational Research Association (AERA)
- Mostow J., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 136
- Napoles C., 2010, Proceedings of the Workshop on Computational Linguistics and Writing: Writing Processes and Authoring Aids, 42
- Perez-Beltrachini L., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 147
- Reichert M., 2010, The C-Test: Contributions from Current Research, 205
- Sculley D., 2010, Proceedings of the Conference on Knowledge Discovery and Data Mining (KDD), 979
- Susanti Y., 2016, Proceedings of the International Conference on Computer Supported Education, 267
- Vajjala S., 2016, Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity, 38
- Vajjala S., 2018, Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, 147
- Volodina E., 2016, Proceedings of EUROCALL, 456
Cited by
Wang, X., Liu, Q., Pang, H., Tan, S. C., Lei, J., Wallace, M. P., & Li, L. (2023) · Computers & Education · doi:10.1016/j.compedu.2022.104703 · Cited by 299
“…students ’ language output and diagnose their output by comparing it with the AI ’ s linguistic models, and then used its recommendation system and speech synthesis techniques to generate personalized feedback for the students (Engwall & Lopes, 2020; Heift & Schulze, 2007; Settles et al., 2020).”
Shanmugasundaram, M., & Tamilarasu, A. (2023) · Frontiers in Cognition · doi:10.3389/fcogn.2023.1203077 · Cited by 291
“Such applications can also be used for language proficiency tests (Settles et al., 2020).”
BackgroundTransforming Assessment: The Impacts and Implications of Large Language Models and Generative AI
Hao, J., von Davier, A. A., Yaneva, V., Lottridge, S., von Davier, M., & Harris, D. J. (2024) · Educational Measurement: Issues and Practice · doi:10.1111/emip.12602 · Cited by 93
The interactive reading task: Transformer-based automatic item generation
Attali, Y., Runge, A., LaFlair, G. T., Yancey, K. P., Goodwin, S., Park, Y., & von Davier, A. A. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.903077 · Cited by 127
“Much prior work has explored ways to estimate the difficulty of reading passages and comprehension questions based on a wide range of lexical, syntactic, and discourse properties (Xia et al., 2016; Huang et al., 2017; Settles et al., 2020).”
BackgroundTest Review: Current options in at-home language proficiency tests for making high-stakes decisions
Isbell, D. R., & Kremmel, B. (2020) · Language Testing · doi:10.1177/0265532220943483 · Cited by 82
Metadata
- Authors
- Settles, B., T. LaFlair, G., & Hagiwara, M.
- Affiliations
- Duolingo (United States) (US)
- Duolingo, Pittsburgh, PA USA. geoff@duolingo.com
- Octanove Labs, Seattle, WA USA. masato@octanove.com
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 247-263
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3019082520
- Cited by
- 106
- References
- 77
- Citations by year
- 2026: 12 · 2025: 21 · 2024: 22 · 2023: 17 · 2022: 18 · 2021: 10 · 2020: 6
- FWCI
- 5.01
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- language proficiency assessment, computerized adaptive testing, item difficulty estimation, test security, item bank, Duolingo English Test, English proficiency, test reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.