Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results)
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Applied linguists have turned increasingly in recent years to meta-analysis as the preferred means of synthesizing quantitative research. The first step in the meta-analytic process involves defining a domain of interest. Despite its apparent simplicity, this step involves a great deal of subjectivity on the part of the meta-analyst. This article problematizes the importance of clearly defining and operationalizing meta-analytic domains. Toward that end, we present a critical review of one particular domain, corrective feedback, which has been subject to 18 unique meta-analyses. Specifically, we examine the unique approach each study has taken in defining their domain of interest. In order to demonstrate the critical role of this stage in the meta-analytic process, we also examine variability in summary effects as a function of the unique subdomains in the sample. Because different techniques used to identify candidate studies carry assumptions about the type of research that falls within the domain of interest (e.g. published vs. unpublished), we also include a brief review of search techniques employed in a set of 81 meta-analyses of second language research. Building on the work of In’nami and Koizumi (2010) and Oswald and Plonsky (2010), the results for this phase of the analysis show that L2 meta-analysts generally rely on a stable but very limited set of search strategies, none of which is likely to yield unpublished studies. Based on our findings related both to domain definition and search techniques employed by L2 researchers, we make specific recommendations for future meta-analytic practice in the field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis
Norris, J. M., & Ortega, L. (2000) · Language Learning · doi:10.1111/0023-8333.00136 · Cited by 2,569 · cited 3×
“Seven such studies were included in the present sample: Biber et al., 2011; Lee et al., in press; Lyster and Saito, 2010; Mackey and Goo, 2007; Norris and Ortega, 2000; Poltavtchenko and Johnson, 2009; Truscott, 2007.”
BackgroundResultPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“In cases where multiple meta-analytic d values were reported, following previous review of this type (e.g. Lipsey and Wilson, 1993; Plonsky and Oswald, in press), these effects were then averaged such that each meta-analysis would contribute a single effect size when calculating summary statistics.”
MethodResearch Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716
The effectiveness of corrective feedback in SLA: A meta-analysis
Li, S. (2010) · Language Learning · doi:10.1111/j.1467-9922.2010.00561.x · Cited by 941 · cited 6×
“For the formulas for these indices, see Lipsey and Wilson (2001). Number of studies of CF that contributed to the meta-analytic results presented here; b ‘Not applicable’ here denotes meta-analyses that included studies of CF among larger domains (e.”
BackgroundInfluentialORAL FEEDBACK IN CLASSROOM SLA: A Meta-Analysis
Lyster, R., & Saito, K. (2010) · Studies in Second Language Acquisition · doi:10.1017/s0272263109990520 · Cited by 838
Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00863.x · Cited by 318
Quantitative Research Methods, Study Quality, and Outcomes: The Case of Interaction Research
Plonsky, L., & Gass, S. (2011) · Language Learning · doi:10.1111/j.1467-9922.2011.00640.x · Cited by 267
Self-citationMeta-analysis in Second Language Research: Choices and Challenges
Oswald, F. L., & Plonsky, L. (2010) · Annual Review of Applied Linguistics · doi:10.1017/s0267190510000115 · Cited by 237
Self-citationThe Effectiveness of feedback for L1-English and L2-writing development: a meta-analysis.
Biber, D., Nekrasova, T., & Horn, B. (2011) · ETS Research Report Series · doi:10.1002/j.2333-8504.2011.tb02241.x · Cited by 175
Cited by
How Big Is “Big”? Interpreting Effect Sizes in L2 Research
Plonsky, L., & Oswald, F. L. (2014) · Language Learning · doi:10.1111/lang.12079 · Cited by 1,960
Self-citationSECOND LANGUAGE ANXIETY AND ACHIEVEMENT: A META-ANALYSIS
Teimouri, Y., Goetze, J., & Plonsky, L. (2019) · Studies in Second Language Acquisition · doi:10.1017/s0272263118000311 · Cited by 611
Self-citationCorpus Use in Language Learning: A Meta‐Analysis
Boulton, A., & Cobb, T. (2017) · Language Learning · doi:10.1111/lang.12224 · Cited by 580
The Effectiveness of Second Language Pronunciation Instruction: A Meta-Analysis
Lee, J., Jang, J., & Plonsky, L. (2015) · Applied Linguistics · doi:10.1093/applin/amu040 · Cited by 431
Self-citationInteraction and instructed second language acquisition
Loewen, S., & Sato, M. (2018) · Language Teaching · doi:10.1017/s0261444818000125 · Cited by 377
Metadata
- Authors
- Plonsky, L., & Brown, D.
- Venue
- Second Language Research
- Publisher
- SAGE Publications
- Published
- 2014-07-14
- Volume
- 31
- Issue
- 2
- Pages
- 267-278
- ISSN
- 0165-9960 (ISSN-L)
- 0267-6583
- 1477-0326
- OpenAlex
- W1971557093
- Cited by
- 111
- References
- 10
- FWCI
- 4.80
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results). https://app.cochat.ai/papers/10.1177/0267658314536436
- The paper
- Plonsky, L., & Brown, D. (2015). Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results). Second Language Research. https://doi.org/10.1177/0267658314536436
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.