CoChat

Machine Learning–Driven Language Assessment

Open access
Settles, Burr, T. LaFlair, Geoffrey, Hagiwara, Masato · 2020 · Transactions of the Association for Computational Linguistics

The DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index.

Abstract

We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales based on a given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with other high-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements.

Overview

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00310

Cites

76 of 76 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 76 references

Research works printed but not linked: 21

  1. Brown J. D., 1989, JALT Journal, 11, 46
  2. Cambridge English. 2012. Preliminary wordlist.
  3. Cau S., 2015, Global Times
  4. Curto P., 2015, Proceedings of the International Conference on Computer Supported Education, 36
  5. Dudley R., 2016, Reuters Investigates
  6. Freedle R., 1999, Language Testing, 16, 2
  7. Graves A., 2014, arXiv
  8. Heafield K., 2013, Proceedings of the Association for Computational Linguistics (ACL), 690
  9. Lin W. Y., 2008, Journal of Pan-Pacific Association of Applied Linguistics, 12, 61
  10. Linacre J. M., 2014, Rasch Measurement Transactions, 27, 1441
  11. Loukina A., 2016, Proceedings of the International Conference on Computational Linguistics (COLING), 3245
  12. McNamara D. S., 2011, Annual Meeting of the American Educational Research Association (AERA)
  13. Mostow J., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 136
  14. Napoles C., 2010, Proceedings of the Workshop on Computational Linguistics and Writing: Writing Processes and Authoring Aids, 42
  15. Perez-Beltrachini L., 2012, Proceedings of the Workshop on Building Educational Applications Using NLP, 147
  16. Reichert M., 2010, The C-Test: Contributions from Current Research, 205
  17. Sculley D., 2010, Proceedings of the Conference on Knowledge Discovery and Data Mining (KDD), 979
  18. Susanti Y., 2016, Proceedings of the International Conference on Computer Supported Education, 267
  19. Vajjala S., 2016, Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity, 38
  20. Vajjala S., 2018, Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, 147
  21. Volodina E., 2016, Proceedings of EUROCALL, 456

Cited by

Metadata

Authors
Settles, B., T. LaFlair, G., & Hagiwara, M.
Affiliations
  • Duolingo (United States) (US)
  • Duolingo, Pittsburgh, PA USA. geoff@duolingo.com
  • Octanove Labs, Seattle, WA USA. masato@octanove.com
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2020-12
Volume
8
Pages
247-263
ISSN
2307-387X (ISSN-L)
OpenAlex
W3019082520
Cited by
106
References
77
Citations by year
2026: 12 · 2025: 21 · 2024: 22 · 2023: 17 · 2022: 18 · 2021: 10 · 2020: 6
FWCI
5.01
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
language proficiency assessment, computerized adaptive testing, item difficulty estimation, test security, item bank, Duolingo English Test, English proficiency, test reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.