Machine Learning–Driven Language Assessment
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index.
Abstract
We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales based on a given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with other high-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
(1991) · Cited by 15,554
The standards for educational and psychological testing.
Eignor, D. R. (2013) · APA handbook of testing and assessment in psychology, Vol. 1: Test theory and testing and assessment in industrial and organizational psychology. · doi:10.1037/14047-013 · Cited by 6,988
Probabilistic Models for Some Intelligence and Attainment Tests
(1961) · Information and Control · doi:10.1016/s0019-9958(61)80061-2 · Cited by 6,939
Common European Framework of Reference for Languages: learning, teaching, assessment
Verhelst, N. D., Avermaet, P. V., Takala, S., Figueras, N., & North, B. (2009) · Ghent University Academic Bibliography (Ghent University) · Cited by 6,120
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,007
A Rasch model for partial credit scoring
Masters, G. N. (1982) · Psychometrika · doi:10.1007/bf02296272 · Cited by 3,782
A Tutorial on the Cross-Entropy Method
de Boer, P.-T., Kroese, D. P., Mannor, S., & Rubinstein, R. Y. (2005) · Annals of Operations Research · doi:10.1007/s10479-005-5724-z · Cited by 3,024
A Rating Formulation for Ordered Response Categories
Andrich, D. (1978) · Psychometrika · doi:10.1007/bf02293814 · Cited by 2,970
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Validating the interpretations and uses of test scores
Kane, M. T. (2013) · Journal of Educational Measurement · doi:10.1111/jedm.12000 · Cited by 2,110
Probabilistic Models for some Intelligence and Attainment Tests.
Wolins, L., Wright, B. D., & Rasch, G. (1982) · Journal of the American Statistical Association · doi:10.2307/2287805 · Cited by 1,979
Introduction to Semi-Supervised Learning
Zhu, X., & Goldberg, A. B. (2009) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01548-9 · Cited by 1,766
Exploiting Generative Models in Discriminative Classifiers
Jaakkola, T., & Haussler, D. H. (1998) · Cited by 1,271
Settles, B. (2012) · Cited by 1,183
Self-citationComputerized Adaptive Testing: A Primer
Wainer, H., Dorans, N. J., Flaugher, R. L., Green, B. F., & Mislevy, R. J. (1990) · Medical Entomology and Zoology · Cited by 1,169
Psychological Testing: Principles and Applications
Murphy, K. R., & Davidshofer, C. O. (1988) · Cited by 1,162
Educational Assessment of Students
Nitko, A. J. (1996) · Cited by 1,033
Perceptual adaptation to non-native speech
Bradlow, A. R., & Bent, T. (2008) · Cognition · doi:10.1016/j.cognition.2007.04.005 · Cited by 906
Language test construction and evaluation
Alderson, J. C., Clapham, C., & Wall, D. (1995) · Cited by 734
Lane, S. (2015) · Routledge eBooks · doi:10.4324/9780203102961 · Cited by 728
Language Assessment in Practice
Palmer, A. S., & Bachman, L. F. (2010) · Cited by 709
Proceedings of the International Conference on Language Resources and Evaluation (LREC)
Wentland, W., Knopp, J., Silberer, C., & Härtung, M. (2008) · Cited by 662
OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles
Lison, P., & Tiedemann, J. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3fi26b3nobqg · Cited by 656
Computerized Adaptive Testing: A Primer
Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., & Mislevy, R. J. (2000) · Routledge eBooks · doi:10.4324/9781410605931 · Cited by 654
Language Assessment in Practice
Clark‐Gareca, B. (2010) · Studies in Applied Linguistics and TESOL · doi:10.7916/d8ht31z2 · Cited by 652
Scalable Modified Kneser-Ney Language Model Estimation
Heafield, K., Pouzyrevsky, I., Clark, J. H., & Koehn, P. (2013) · Cited by 647
Encyclopedia of social measurement
(2005) · Choice Reviews Online · doi:10.5860/choice.42-5629 · Cited by 616
Vocabulary size and the skills of listening, reading and writing
Stæhr, L. S. (2008) · Language Learning Journal · doi:10.1080/09571730802389975 · Cited by 540
Settles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01560-1 · Cited by 502
Self-citationAPPLICATION OF COMPUTERIZED ADAPTIVE TESTING TO EDUCATIONAL PROBLEMS
WEISS, D. J., & KINGSBURY, G. G. (1984) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1984.tb01040.x · Cited by 501
The clear speech effect for non-native listeners
Bradlow, A. R., & Bent, T. (2002) · The Journal of the Acoustical Society of America · doi:10.1121/1.1487837 · Cited by 343
On the Reliability of Testlet‐Based Tests
Sireci, S. G., Thissen, D., & Wainer, H. (1991) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1991.tb00356.x · Cited by 301
Advances in Neural Information Processing Systems 11 (NIPS 1998)
Giancarlo Ferrari-trecate, A., Williams, C. K. I., & Opper, M. (1999) · The MIT Press eBooks · Cited by 235
Bartning, I., Martin, M., & Vedder, I. (2011) · UvA-DARE (University of Amsterdam) · Cited by 215
Elicited imitation:a brief overview
Vinther, T. (2002) · International Journal of Applied Linguistics · doi:10.1111/1473-4192.00024 · Cited by 212
Smart Language: Readers, Readability, and the Grading of Text
DuBay, W. H. (2007) · Cited by 198
Text Readability Assessment for Second Language Learners
Xia, M., Kochmar, E., & Briscoe, T. (2016) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w16-0502 · Cited by 176
C-Tests in the context of reduced redundancy testing: an appraisal
Klein-Braley, C. (1997) · Language Testing · doi:10.1177/026553229701400104 · Cited by 171
Combined regression and ranking
Sculley, D. (2010) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1835804.1835928 · Cited by 149
Freedle, R. O., & Kostin, I. W. (1999) · Language Testing · doi:10.1177/026553229901600102 · Cited by 145
Wainer, H. (2010) · The Corsini Encyclopedia of Psychology · doi:10.1002/9780470479216.corpsy0213 · Cited by 130
A1–B2 vocabulary: insights and issues arising from the English Profile Wordlists project
Capel, A. (2010) · English Profile Journal · doi:10.1017/s2041536210000048 · Cited by 116
A psycholinguistic approach to oral language assessment
Moere, A. V. (2012) · Language Testing · doi:10.1177/0265532211424478 · Cited by 114
Protecting the Integrity of Computerized Testing Item Pools
Way, W. D. (1998) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1998.tb00632.x · Cited by 108
Completing the English Vocabulary Profile: C1 and C2 vocabulary
Capel, A. (2012) · English Profile Journal · doi:10.1017/s2041536212000013 · Cited by 100
Segall, D. O. (2005) · Elsevier eBooks · doi:10.1016/b0-12-369398-5/00444-8 · Cited by 93
Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with the CEFR
SUNG, Y., LIN, W., DYSON, S. B., CHANG, K., & CHEN, Y. (2015) · The Modern Language Journal · doi:10.1111/modl.12213 · Cited by 87
The Meaning of Cloze Test Scores: An Item Difficulty Perspective
Abraham, R. G., & Chapelle, C. A. (1992) · Modern Language Journal · doi:10.1111/j.1540-4781.1992.tb05394.x · Cited by 87
Chapter 6. Aural Word Recognition and Oral Competence in English as a Foreign Language
Milton, J., Wade, J., & Hopkins, N. (2010) · Multilingual Matters eBooks · doi:10.21832/9781847692900-007 · Cited by 83
AN ANALYSIS OF FACTORS AFFECTING THE DIFFICULTY OF DIALOGUE ITEMS IN TOEFL LISTENING COMPREHENSION
Nissan, S., DeVincenzi, F., & Tang, K. L. (1995) · ETS Research Report Series · doi:10.1002/j.2333-8504.1995.tb01671.x · Cited by 81
The Relationship Between Item Exposure and Test Overlap in Computerized Adaptive Testing
Chen, S., Ankenmann, R. D., & Spray, J. A. (2003) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2003.tb01100.x · Cited by 77
THREE PRACTICAL ISSUES FOR MODERN ADAPTIVE TESTING ITEM POOLS1
Stocking, M. L. (1994) · ETS Research Report Series · doi:10.1002/j.2333-8504.1994.tb01578.x · Cited by 71
Predicting the Difficulty of Language Proficiency Tests
Beinborn, L., Zesch, T., & Gurevych, I. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00200 · Cited by 67
(1978) · Physics Bulletin · doi:10.1088/0031-9112/29/6/013 · Cited by 65
Elicited Imitation in Second Language Acquisition Research
Jessop, L., Suzuki, W., & Tomita, Y. (2007) · The Canadian Modern Language Review · doi:10.3138/cmlr.64.1.215 · Cited by 62
Zimmerman, J., Broder, P. K., Shaughnessy, J. J., & Underwood, B. J. (1977) · Intelligence · doi:10.1016/0160-2896(77)90025-3 · Cited by 57
Experiments with Universal CEFR Classification
Vajjala, S., & Rama, T. (2018) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w18-0515 · Cited by 56
Deriving TF-IDF as a Fisher Kernel
Elkan, C. (2005) · Lecture notes in computer science · doi:10.1007/11575832_33 · Cited by 55
Textual complexity as a predictor of difficulty of listening items in language proficiency tests.
Loukina, A., Yoon, S., Sakano, J., Wei, Y., & Sheehan, K. H. (2016) · International Conference on Computational Linguistics · Cited by 51
Generating Diagnostic Multiple Choice Comprehension Cloze Questions
Mostow, J., & Jang, H. (2012) · Cited by 48
Learning Simple Wikipedia: A Cogitation in Ascertaining Abecedarian Language
Napoles, C., & Dredze, M. H. (2010) · Cited by 45
Aural Word Recognition and Oral Competence in English as a Foreign Language
Milton, J., Wade, J., & Hopkins, N. (2010) · Channel View Publications eBooks · doi:10.2307/jj.29308428.9 · Cited by 44
EXPLORING ITEM CHARACTERISTICS THAT ARE RELATED TO THE DIFFICULTY OF TOEFL DIALOGUE ITEMS
Kostin, I. W. (2004) · ETS Research Report Series · doi:10.1002/j.2333-8504.2004.tb01938.x · Cited by 39
Freedle, R. O., & Kostin, I. W. (1993) · ETS Research Report Series · doi:10.1002/j.2333-8504.1993.tb01524.x · Cited by 38
Challenges and Opportunities of the CEFR for Reimagining Foreign Language Pedagogy
Westhoff, G. J. (2007) · Modern Language Journal · doi:10.1111/j.1540-4781.2007.00627_9.x · Cited by 37
A comparison of three test formats to assess word difficulty
Culligan, B. (2015) · Language Testing · doi:10.1177/0265532215572268 · Cited by 34
Perez-Beltrachini, L., Gardent, C., Kruszewski, G., Exercises, G. G., & Lorraine, U. ́. D. (2012) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 27
Vajjala, S., Meurers, D., Eitel, A., & Scheiter, K. (2016) · Iowa State University Digital Repository (Iowa State University) · Cited by 22
Isbell, D. R. (2017) · Assessing Writing · doi:10.1016/j.asw.2017.08.004 · Cited by 21
Curto, P., Mamede, N., & Baptista, J. (2015) · International Conference on Computer Supported Education · doi:10.5220/0005428300360044 · Cited by 20
Item Difficulty Analysis of English Vocabulary Questions
Susanti, Y., Nishikawa, H., Tokunaga, T., & Hiroyuki, O. (2016) · Proceedings of the 8th International Conference on Computer Supported Education · doi:10.5220/0005775502670274 · Cited by 18
Classification of Swedish learner essays by CEFR levels
Volodina, E., Pilán, I., & Alfter, D. (2016) · doi:10.14705/rpnet.2016.eurocall2016.606 · Cited by 18
Construct Validity of C-tests: A Factorial Approach
Khodadady, E. (2014) · Journal of Language Teaching and Research · doi:10.4304/jltr.5.6.1353-1362 · Cited by 16
The C-test, the TCF and the CEFR: a validation study.
Reichert, M., Keller, U., & Martin, R. (2010) · Open Repository and Bibliography (University of Luxembourg) · Cited by 7
Language Reduced Redundancy Tests: A Reexamination of Cloze Test and C-test
Lin, W.-Y., Yuan, H.-C., & Feng, H.-P. (2008) · Journal of Pan-Pacific Association of Applied Linguistics · doi:10.6384/ciq.201004.0189 · Cited by 3
Research works printed but not linked: 21
- Brown J. D., 1989, JALT Journal, 11, 46
- Cambridge English. 2012. Preliminary wordlist.
- Cau S., 2015, Global Times
- Curto P., 2015, Proceedings of the International Conference on Computer Supported Education, 36
- Dudley R., 2016, Reuters Investigates
- Freedle R., 1999, Language Testing, 16, 2
- Graves A., 2014, arXiv
- Heafield K., 2013, Proceedings of the Association for Computational Linguistics (ACL), 690
- Lin W. Y., 2008, Journal of Pan-Pacific Association of Applied Linguistics, 12, 61
- Linacre J. M., 2014, Rasch Measurement Transactions, 27, 1441
- Loukina A., 2016, Proceedings of the International Conference on Computational Linguistics (COLING), 3245
- McNamara D. S., 2011, Annual Meeting of the American Educational Research Association (AERA)
- Mostow J., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 136
- Napoles C., 2010, Proceedings of the Workshop on Computational Linguistics and Writing: Writing Processes and Authoring Aids, 42
- Perez-Beltrachini L., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 147
- Reichert M., 2010, The C-Test: Contributions from Current Research, 205
- Sculley D., 2010, Proceedings of the Conference on Knowledge Discovery and Data Mining (KDD), 979
- Susanti Y., 2016, Proceedings of the International Conference on Computer Supported Education, 267
- Vajjala S., 2016, Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity, 38
- Vajjala S., 2018, Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, 147
- Volodina E., 2016, Proceedings of EUROCALL, 456
Cited by
Wang, X., Liu, Q., Pang, H., Tan, S. C., Lei, J., Wallace, M. P., & Li, L. (2023) · Computers & Education · doi:10.1016/j.compedu.2022.104703 · Cited by 299
Shanmugasundaram, M., & Tamilarasu, A. (2023) · Frontiers in Cognition · doi:10.3389/fcogn.2023.1203077 · Cited by 291
“Such applications can also be used for language proficiency tests (Settles et al., 2020).”
BackgroundTransforming Assessment: The Impacts and Implications of Large Language Models and Generative AI
Hao, J., von Davier, A. A., Yaneva, V., Lottridge, S., von Davier, M., & Harris, D. J. (2024) · Educational Measurement: Issues and Practice · doi:10.1111/emip.12602 · Cited by 93
The interactive reading task: Transformer-based automatic item generation
Attali, Y., Runge, A., LaFlair, G. T., Yancey, K. P., Goodwin, S., Park, Y., & von Davier, A. A. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.903077 · Cited by 127
Test Review: Current options in at-home language proficiency tests for making high-stakes decisions
Isbell, D. R., & Kremmel, B. (2020) · Language Testing · doi:10.1177/0265532220943483 · Cited by 82
Metadata
- Authors
- Settles, B., T. LaFlair, G., & Hagiwara, M.
- Affiliations
- Duolingo (United States) (US)
- Duolingo, Pittsburgh, PA USA. geoff@duolingo.com
- Octanove Labs, Seattle, WA USA. masato@octanove.com
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 247-263
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3019082520
- Cited by
- 106
- References
- 77
- Citations by year
- 2026: 12 · 2025: 21 · 2024: 22 · 2023: 17 · 2022: 18 · 2021: 10 · 2020: 6
- FWCI
- 5.01
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- language proficiency assessment, computerized adaptive testing, item difficulty estimation, test security, item bank, Duolingo English Test, English proficiency, test reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.