Dependency parsing of learner English
Open accessAbstract
Current syntactic annotation of large-scale learner corpora mainly resorts to “standard parsers” trained on native language data. Understanding how these parsers perform on learner data is important for downstream research and application related to learner language. This study evaluates the performance of multiple standard probabilistic parsers on learner English. Our contributions are three-fold. Firstly, we demonstrate that the common practice of constructing a gold standard – by manually correcting the pre-annotation of a single parser – can introduce bias to parser evaluation. We propose an alternative annotation method which can control for the annotation bias. Secondly, we quantify the influence of learner errors on parsing errors, and identify the learner errors that impact on parsing most. Finally, we compare the performance of the parsers on learner English and native English. Our results have useful implications on how to select a standard parser for learner English.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Punctuation and capitalization errors lower the accuracy of automated analysis.
“the first two error types have been demonstrated to lower the accuracy of automated analysis (Huang, Murakami, Alexopoulou, & Korhonen, 2018)”
From Effects of Availability, Contingency, and Formulaicity on the Accuracy of English Grammatical Morphemes in Second Language Writing (2022) · cites this paper as Huang, Murakami, Alexopoulou, & Korhonen, 2018
The passages labeled Abstract contain bibliographic metadata rather than findings from the paper; no article page is available in the returned passages. Searches for punctuation, capitalization, and parsing accuracy found no passage addressing whether those errors lower automated-analysis accuracy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Advances in Neural Information Processing Systems 14
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 9,184 · see citing works
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
Common European Framework of Reference for Languages: learning, teaching, assessment
Verhelst, N. D., Avermaet, P. V., Takala, S., Figueras, N., & North, B. (2009) · Ghent University Academic Bibliography (Ghent University) · Cited by 6,088 · see citing works
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1075096.1075150 · Cited by 3,040 · see citing works
Generating Typed Dependency Parses from Phrase Structure Parses
de Marneffe, M.-C., MacCartney, B., & Manning, C. D. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3wtkaidk6ea8 · Cited by 2,289 · see citing works
CoNLL-X shared task on multilingual dependency parsing
Buchholz, S., & Marsi, E. (2006) · doi:10.3115/1596276.1596305 · Cited by 971 · see citing works
Errors in Language Learning and Use: Exploring Error Analysis
Broussard, K. M., & James, C. (1999) · TESOL Quarterly · doi:10.2307/3588202 · Cited by 906 · see citing works
Coarse-to-fine n -best parsing and MaxEnt discriminative reranking
Charniak, E., & Johnson, M. S. (2005) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1219840.1219862 · Cited by 876 · see citing works
MaltParser: A language-independent system for data-driven dependency parsing
Nivre, J., Hall, J., Nilsson, J., Chanev, A., Eryiğit, G., Kübler, S., Marinov, S., & Marsi, E. (2007) · Natural Language Engineering · doi:10.1017/s1351324906004505 · Cited by 801 · see citing works
Fast Exact Inference with a Factored Model for Natural Language Parsing
Klein, D., & Manning, C. D. (2002) · Cited by 743 · see citing works
Stanford typed dependencies manual
de Marnee, M.-C., & Manning, C. D. (2010) · Cited by 728 · see citing works
Improved Inference for Unlexicalized Parsing
Petrov, S., & Klein, D. (2007) · Cited by 606 · see citing works
Building a Large Annotated Corpus of English: The
MarcusMitchell, P., Ann, M., & SantoriniBeatrice (1993) · Computational Linguistics · doi:10.5555/972470.972475 · Cited by 495 · see citing works
Proceedings of the corpus linguistics 2003 conference.
Archer, D., Rayson, P., Wilson, A., & McEnery, T. (2003) · Lancaster EPrints (Lancaster University) · Cited by 378 · see citing works
Errors in Language Learning and Use
James, C. (2013) · Routledge eBooks · doi:10.4324/9781315842912 · Cited by 281 · see citing works
Common European Framework of Reference for Languages
Little, D. (2020) · The TESOL Encyclopedia of English Language Teaching · doi:10.1002/9781118784235.eelt0114.pub2 · Cited by 256 · see citing works
Turning on the Turbo: Fast Third-Order Non-Projective Turbo Parsers
Martins, A. F. T., Almeida, M., & Smith, N. A. (2018) · KiltHub Repository · doi:10.1184/r1/6473801 · Cited by 156 · see citing works
Parsing to Stanford Dependencies: Trade-offs between Speed and Accuracy
Cer, D. M., de Marneffe, M.-C., Jurafsky, D., & Manning, C. D. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3aixhxswiwe3 · Cited by 127 · see citing works
Quantitative research methods and study quality in learner corpus research
Paquot, M., & Plonsky, L. (2017) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.3.1.03paq · Cited by 111 · see citing works
Geertzen, J., Alexopoulou, T., & Korhonen, A. (2014) · Cited by 67 · see citing works
Self-citationUniversal Dependencies for Learner English
Berzak, Y., Kenney, J., Spadine, C., Wang, J. X., Lam, L., Mori, K. S., Garza, S., & Katz, B. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1070 · Cited by 59 · see citing works
Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications
Dzikovska, M. O., Farrow, E., & Moore, J. D. (2013) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 49 · see citing works
Evaluating and automating the annotation of a learner corpus
Rosen, A., Hana, J., Štindlová, B., & Feldman, A. (2013) · Language Resources and Evaluation · doi:10.1007/s10579-013-9226-3 · Cited by 46 · see citing works
Semantically motivated subcategorization acquisition
Korhonen, A. (2002) · doi:10.3115/1118627.1118634 · Cited by 46 · see citing works
Self-citationThe International Corpus of Learner English. Version 2. Handbook and CD-Rom
Granger, S., Dagneaux, E., Meunier, F., & Paquot, M. (2009) · Digital Access to Libraries (Université catholique de Louvain (UCL), l'Université de Namur (UNamur) and the Université Saint-Louis (USL-B)) · Cited by 43 · see citing works
Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 2
Matsumoto, Y. (2003) · Cited by 39 · see citing works
Evaluating dependency parsing performance on german learner language
Ott, N., & Ziai, R. (2010) · DSpace repository (University of Tartu) · doi:10.5281/zenodo.8100415 · Cited by 37 · see citing works
Anchoring and Agreement in Syntactic Annotations
Berzak, Y., Huang, Y., Barbu, A., Korhonen, A., & Katz, B. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1239 · Cited by 25 · see citing works
Self-citationComparing rule-based and data-driven dependency parsing of learner language
Krivanek, J., & Meurers, D. (2011) · doi:10.5281/zenodo.8100423 · Cited by 23 · see citing works
Focus on learner writing at the beginning and intermediate stages: The ICCI corpus
Tono, Y., & Díez-Bedmar, M. B. (2014) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.19.2.01ton · Cited by 22 · see citing works
Inter-annotator Agreement for Dependency Annotation of Learner Language
Ragheb, M., & Dickinson, M. (2013) · Cited by 12 · see citing works
Modifying Corpus Annotation to Support the Analysis of Learner Language
Dickinson, M., & Lee, C. (2009) · CALICO Journal · doi:10.1558/cj.v26i3.545-561 · Cited by 12 · see citing works
Rankin, T. (2015) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.1.2.07ran · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 36 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Gaillat, T., Simpkin, A. J., Ballier, N., Stearns, B., Sousa, A., Bouyé, M., & Zarrouk, M. (2021) · ReCALL · doi:10.1017/s095834402100029x · Cited by 38 · see citing works
Learning the language of Dentistry
Crosthwaite, P. R., & Cheung, L. (2019) · Studies in corpus linguistics · doi:10.1075/scl.93 · Cited by 41 · see citing works
From early to future learner corpus research
Granger, S. (2024) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.00050.gra · Cited by 28 · see citing works
Du, X., Afzaal, M., & Alfadda, H. A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.752134 · Cited by 28 · see citing works
Sato, M., Thorne, S. L., Michel, M. C., Alexopoulou, T., & Hellermann, J. (2025) · Modern Language Journal · doi:10.1111/modl.12976 · Cited by 27 · see citing works
Self-citationNatural language processing for learner corpus research
Kyle, K. D. (2020) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.00019.int · Cited by 27 · see citing works
Using phrase-frames to trace the language development of L1 Chinese learners of English
Tan, Y., & Römer, U. (2022) · System · doi:10.1016/j.system.2022.102844 · Cited by 27 · see citing works
Murakami, A., & Ellis, N. C. (2022) · Language Learning · doi:10.1111/lang.12500 · Cited by 26 · see citing works
Self-citationChen, X., Alexopoulou, T., & Tsimpli, I. M. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01456-7 · Cited by 25 · see citing works
Self-citationGilquin, G. (2020) · Springer eBooks · doi:10.1007/978-3-030-46216-1_13 · Cited by 22 · see citing works
Kim, M., & Lu, X. (2024) · Modern Language Journal · doi:10.1111/modl.12907 · Cited by 21 · see citing works
Shatz, I. (2020) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.20009.sha · Cited by 20 · see citing works
Uchida, S., & Negishi, M. (2025) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2025.100199 · Cited by 22 · see citing works
Evaluating NLP models with written and spoken L2 samples
Kyle, K. D., & Eguchi, M. (2024) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2024.100120 · Cited by 14 · see citing works
Xia, D., Ai, H., & Pae, H. K. (2022) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.20019.xia · Cited by 14 · see citing works
Back to grammar: Using grammatical error correction to automatically assess L2 speaking proficiency
Bannò, S., & Matassoni, M. (2023) · Speech Communication · doi:10.1016/j.specom.2023.103025 · Cited by 17 · see citing works
A Supervised Learning Model for the Automatic Assessment of Language Levels Based on Learner Errors
Ballier, N., Gaillat, T., Simpkin, A. J., Stearns, B., Bouyé, M., & Zarrouk, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29736-7_23 · Cited by 12 · see citing works
Shatz, I., Alexopoulou, T., & Murakami, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281137 · Cited by 11 · see citing works
Self-citationStudying children's writing development with a corpus
Durrant, P. (2022) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2022.100026 · Cited by 11 · see citing works
Linguistic Features Distinguishing Students’ Writing Ability Aligned with CEFR Levels
Ma, H., Wang, J., & He, L. (2023) · Applied Linguistics · doi:10.1093/applin/amad054 · Cited by 10 · see citing works
Eguchi, M., & Kyle, K. D. (2024) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2024.100153 · Cited by 10 · see citing works
The CELI corpus: Design and linguistic annotation of a new online learner corpus
Spina, S., Fioravanti, I., Forti, L., & Zanda, F. (2023) · Second language Research · doi:10.1177/02676583231176370 · Cited by 9 · see citing works
Utility of Kolmogorov complexity measures: Analysis of L2 groups and L1 backgrounds
Alzahrani, A. A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301806 · Cited by 8 · see citing works
McCallum, L., & Durrant, P. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009070461 · Cited by 8 · see citing works
Xia, D., Sulzer, M., & Pae, H. K. (2023) · Text and Talk · doi:10.1515/text-2022-0027 · Cited by 8 · see citing works
Using the Relative Entropy of Linguistic Complexity to Assess L2 Language Proficiency Development
Sun, K., & Wang, R. (2021) · Entropy · doi:10.3390/e23081080 · Cited by 7 · see citing works
Rubin, R. (2020) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.20005.rub · Cited by 7 · see citing works
Klebanov, B. B., & Madnani, N. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01121ed1v01y202108hlt052 · Cited by 7 · see citing works
Assessing receptive vocabulary using state‑of‑the‑art natural language processing techniques
Crossley, S. A., & Holmes, L. (2022) · Journal of Second Language Studies · doi:10.1075/jsls.22006.cro · Cited by 7 · see citing works
TellMeWhy: Learning to Explain Corrective Feedback for Second Language Learners
Lai, Y.-H., & Chang, J. S. (2019) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d19-3040 · Cited by 7 · see citing works
Crosthwaite, P. R., & Cheung, L. (2019) · Cited by 7 · see citing works
Towards an Italian Learner Treebank in Universal Dependencies.
Nuovo, E. D., Bosco, C., Mazzei, A., & Sanguinetti, M. (2019) · Institutional Research Information System University of Turin (University of Turin) · Cited by 5 · see citing works
Towards the Prediction of Semantic Complexity Based on Concept Graphs
Venant, R., & d’Aquin, M. (2019) · HAL · Cited by 4 · see citing works
Unravelling interlanguage facts via explainable machine learning
Berti, B., Esuli, A., & Sebastiani, F. (2023) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqad019 · Cited by 4 · see citing works
Sun, K., & Lu, X. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.672243 · Cited by 5 · see citing works
Zhang, Y., & Zhang, L. J. (2024) · TESOL Quarterly · doi:10.1002/tesq.3370 · Cited by 3 · see citing works
Murakami, A. (2025) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2025.100182 · Cited by 2 · see citing works
Self-citationFrom Linguistic Research Projects to Language Technology Platforms: A Case Study in Learner Data
Sousa, A., Ballier, N., Gaillat, T., Stearns, B., Zarrouk, M., Simpkin, A. J., & Bouyé, M. (2020) · HAL · Cited by 2 · see citing works
Subcategorization frame identification for learner English
Huang, Y., Murakami, A., Alexopoulou, T., & Korhonen, A. (2020) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.18097.hua · Cited by 1 · see citing works
Self-citationLestari, F. (2024) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2024.100097 · Cited by 1 · see citing works
Shatz, I., Alexopoulou, T., & Murakami, A. (2024) · Corpora · doi:10.3366/cor.2024.0305 · Cited by 1 · see citing works
Self-citationExploring the cross-lingual influence of linguistic complexity in second language writing assessment
Geremia, S., Gaillat, T., Ballier, N., & Simpkin, A. J. (2025) · Assessing Writing · doi:10.1016/j.asw.2025.100951 · Cited by 1 · see citing works
Fukuta, J., Murakami, A., Terai, M., & Tamura, Y. (2026) · Bilingualism Language and Cognition · doi:10.1017/s1366728925100965 · Cited by 1 · see citing works
Self-citationLexical Bias In Essay Level PredictionPreprint
Balikas, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.08935 · Cited by 1 · see citing works
Using Classifier Features to Determine Language Transfer on Morphemes
Lavrentovich, A. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/n18-4009
Hyland, K. (2019) · Studies in corpus linguistics · doi:10.1075/scl.93.for
Yan, H., & Li, Y. (2023) · Linguistics Vanguard · doi:10.1515/lingvan-2022-0122
Coding all clauses in L2 data: A call for consistency
Vercellotti, M. L., & Hall, S. (2024) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2024.100132
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.r2
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.c3
Putting it into (dental) practice
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.c6
‘Extracting’ the linguistic dimensions of dentistry experimental research articles
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.c2
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.ind
Language Features in EFA Model (Chapter 2)
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.app
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.toc
Understanding disciplinary vs. non-disciplinary terminology
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.c4
Getting to the ‘root’ of the problem
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.c1
Hyland, K. (2019) · Studies in corpus linguistics · doi:10.1075/scl.93.foreword
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.prelim
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.c5
Reflections on practice, and future directions
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.c7
(2019) · Studies in corpus linguistics · doi:10.1075/scl.93.ack
A framework for verb annotation of Spanish L2 written corpora using local LLMs
Zimotti, G. (2026) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2026.100308
Durrant, P. (2026) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.00965-0
Big Educational Language Learning data (BELL)
Alexopoulou, T. (2026) · Language acquisition & language disorders · doi:10.1075/lald.71.15ale
Self-citationComplexity, Accuracy, and Fluency in L2 Writing
Rokoszewska, K. (2026) · Second language learning and teaching · doi:10.1007/978-3-032-10318-5_4
Syntactic complexity in Asian EFL writing: A gold-standard phrase structure analysisPreprint
Le, N. V. A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7404740
- Authors
- Huang, Y., Murakami, A., Alexopoulou, T., & Korhonen, A.
- Affiliations
- University of Cambridge (GB)
- University of Tübingen (DE)
- Venue
- International Journal of Corpus Linguistics
- Publisher
- John Benjamins Publishing Company
- Published
- 2018-05-31
- Volume
- 23
- Issue
- 1
- Pages
- 28-54
- ISSN
- 1384-6655 (ISSN-L)
- 1569-9811
- OpenAlex
- W2805577724
- Cited by
- 68
- References
- 36
- Citations by year
- 2026: 6 · 2025: 4 · 2024: 9 · 2023: 7 · 2022: 7 · 2021: 4 · 2020: 8 · 2019: 21 · 2018: 2
- FWCI
- 4.90
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- Locations
- International Journal of Corpus Linguistics (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Apollo (University of Cambridge) (submittedVersion)
- Apollo
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Second Language Acquisition and Learning
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- dependency parsing, learner English, interlanguage, gold standard annotation, annotation bias, student errors, parsing errors, parser evaluation, second language acquisition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
