Bootstrapping in Applied Linguistics: Assessing its Potential Using Shared Data
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Parametric analyses such as t tests and ANOVAs are the norm - if not the default - statistical tests found in quantitative applied linguistics research (Gass 2009). Applied statisticians and one applied linguist (Larson-Hall 2010, 2012; Larson-Hall and Herrington 2010), however, have argued that this approach may not be appropriate for small samples and/or nonnormally distributed data (e.g. Wilcox 2003), both common in second language (L2) research. They recommend instead 'robust statistics' such as bootstrapping, a nonparametric procedure that randomly resamples from an observed data set to produce a simulated but more stable and statistically accurate outcome. The present study tests the usefulness of bootstrapping by reanalyzing raw data from 26 studies of applied linguistics research. Our results found no evidence of Type II error (false negative). However, 4 out of 16 statistically significant results were not replicated (i.e. a Type I error 'misfit' five times higher than an alpha of. 05). We discuss empirically justified suggestions for the use of bootstrapping in the context of broader methodological issues and reforms in applied linguistics (see Plonsky 2013, 2014).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350 · cited 1×
“The analyses were conducted using the boot package version 1.3-4 (Davison and Hinkley 1997; Angelo and Ripley 2012).”
MethodInfluentialEffectiveness of L2 instruction: A research synthesis and quantitative meta-analysis
Norris, J. M., & Ortega, L. (2000) · Language Learning · doi:10.1111/0023-8333.00136 · Cited by 2,569
How Big Is “Big”? Interpreting Effect Sizes in L2 Research
Plonsky, L., & Oswald, F. L. (2014) · Language Learning · doi:10.1111/lang.12079 · Cited by 1,960
Self-citationWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 2×
“…this practice as a Type M (for magnitude) error, a notion closely related to the author- and editorially-rooted bias in favor publishing statistically significant results usually discussed in the context of meta-analysis (see also Norris and Ortega 2006; Ioannidis 2008; Author and Co-author 2012).”
BackgroundResultMooney, C., & Duval, R. (1993) · doi:10.4135/9781412983532 · Cited by 893
The Philosophy of Multiple Comparisons
Tukey, J. W. (1991) · Statistical Science · doi:10.1214/ss/1177011945 · Cited by 773
Book Review: Discovering Statistics Using SPSS
Field, A., & Gaitanou, C. (2005) · PsyPag Quarterly · doi:10.53841/bpspag.2005.1.56.31 · Cited by 679
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 4×
“Based on our results, we agree with Wicherts et al.’s (2006) recommendation that”
BackgroundResultInfluentialFundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy
Wilcox, R. R. (2010) · doi:10.1007/978-1-4419-5525-8 · Cited by 404
Cited by
How Big Is “Big”? Interpreting Effect Sizes in L2 Research
Plonsky, L., & Oswald, F. L. (2014) · Language Learning · doi:10.1111/lang.12079 · Cited by 1,960
Self-citationCorpus Use in Language Learning: A Meta‐Analysis
Boulton, A., & Cobb, T. (2017) · Language Learning · doi:10.1111/lang.12224 · Cited by 580
The Effectiveness of Second Language Pronunciation Instruction: A Meta-Analysis
Lee, J., Jang, J., & Plonsky, L. (2015) · Applied Linguistics · doi:10.1093/applin/amu040 · Cited by 431
Self-citationMultiple Regression in L2 Research: A Methodological Synthesis and Guide to Interpreting R2Values
PLONSKY, L., & GHANBAR, H. (2018) · The Modern Language Journal · doi:10.1111/modl.12509 · Cited by 321
Self-citationMarsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
Mizumoto, A. (2023) · Language Learning · doi:10.1111/lang.12518 · Cited by 176
Plonsky, L., & Gonulal, T. (2015) · Language Learning · doi:10.1111/lang.12111 · Cited by 159
Self-citationAn A–Z of Applied Linguistics Research Methods
Loewen, S., & Plonsky, L. (2016) · doi:10.1007/978-1-137-40322-3 · Cited by 156
Self-citationEffects of task repetition on learners’ attention orientation in L2 oral production
Fukuta, J. (2016) · Language Teaching Research · doi:10.1177/1362168815570142 · Cited by 147
“…conducted further analysis by ‘robust statistics’ (Larson-Hall & Herrington, 2010; Plonsky, Egbert & LaFlair, 2014), namely, bootstrapping,5 which ‘randomly resamples from an observed data set to produce a simulated but more stable and statistically accurate outcome’ (Plonsky et al., 2014, p. 1).”
Method
Metadata
- Authors
- Plonsky, L., Egbert, J., & Laflair, G. T.
- Venue
- Applied Linguistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-02-14
- Pages
- amu001
- ISSN
- 0142-6001 (ISSN-L)
- 1477-450X
- OpenAlex
- W2028109586
- Cited by
- 89
- References
- 14
- FWCI
- 60.01
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.