The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 5 of the 47 sources its claims cite were checked for support.
How do language education researchers attend to quality in qualitative studies?
Open accessAbstract
Abstract The steady expansion in qualitative research in the area of language education over the last two decades indicates the growing recognition of its importance to investigating issues of language teaching and learning. Along with this recognition, understanding and assessing the quality of qualitative studies in this area has gained increasing significance. Addressing this concern, in this research synthesis, we qualitatively explore how 236 qualitative language education studies published in seven leading journals explicitly foreground the issue of ‘research quality’. We conducted a qualitative content analysis of how authors of these studies addressed the main quality concepts proposed by well-known frameworks of qualitative research quality. Our findings, presented as ten major themes, show that qualitative researchers' overt treatment of research quality is realised based on three distinct orientations: no explicit quality criteria, positivist views of quality, and interpretive quality conceptions. We discuss aspects of these orientations and their implications for qualitative research in language education.
Overview
CoChat Summary
TL;DR
This research synthesis qualitatively analyzed how 236 qualitative language education studies published in seven leading journals (1999–2021) explicitly addressed research quality. It identified ten major quality themes and three orientations—no explicit quality criteria, positivist views of quality, and interpretive quality conceptions—with positivist terms such as validity and reliability still prevalent.
Key findings
- Across 236 studies, the most frequent quality term was triangulation (105 mentions across 78 articles), followed by reliability (73/42), validity (63/46), trustworthiness (30/24), member checking (27/16), credibility (17/15), corroboration (15/12), rigor (14/12), thick description (9/8), transparency (8/8), reflexivity (8/4), transferability (7/7), dependability (3/3), and confirmability (1/1).
- The authors grouped findings into three orientations: no explicit quality criteria; positivist views of quality, including inter-coder/inter-rater reliability, Cohen’s kappa, and Cronbach’s alpha; and interpretive quality conceptions, especially trustworthiness, credibility, member checking, and triangulation.
- Triangulation was often described as merely collecting multiple data types, while validity and reliability were frequently used vaguely or in positivist senses; just one study made complete use of Lincoln and Guba’s componential trustworthiness model, and no studies formally adhered to other well-known frameworks such as Richardson, Spencer et al., Stewart, or Tracy.
- The sample included 236 articles from Applied Linguistics (17), Foreign Language Annals (37), Language Learning (1), Language Teaching Research (29), Modern Language Journal (28), System (74), and TESOL Quarterly (50); 56% were single-author, 35% two-author, and 8% three-author works.
Methods
- Research synthesis using qualitative content analysis of manifest content in 236 qualitative language education articles published in seven journals from 1999 to 2021.
- Scopus searches generated 741 records; alternate records were excluded, 134 studies were eliminated, and full texts of the final 236 studies were imported into NVivo 12.
- The authors searched lemma forms of quality concepts, exported 801 mentions, cleaned them to 380 relevant instances, coded them, and constellated codes into ten themes.
Limitations
- Stated in the paper: the analysis was limited to manifest content rather than latent meaning; search terms depended on authors explicitly positioning studies as qualitative or ethnographic; cross-paradigmatic approaches such as feminist and participatory action research were excluded; specialist journals, non-primary-research journals, and book chapters were excluded; concepts with diverse linguistic operationalizations (e.g., prolonged engagement, evocative representation, contextual detail) were not feasible to include; member reflection and multivocality generated no hits; publication word limits constrained authors and the authors’ own reporting; and the researchers could not query authors directly about their use of terms.
- Evident from the text: the sample was restricted to seven English-language journals indexed in Scopus, to articles labelled as “articles” and “final” versions, and to 1999–2021; distribution was uneven across journals and time (e.g., Language Learning contributed only 1 study); and the manifest-content approach may miss implicit quality work.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 0 Not valid · 48 Not checked · 79 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Yet, the prevailing view held by qualitative scholars from a social constructivist standpoint (Kress, 2011), that we would align ourselves with, is that the idea of foundational criteria (i.e., fixed and universal) against which the quality of qualitative research can be judged is flawed (Barbour, 2001; Hammersley, 2007; Lazaraton, 2003; Richards, 2009; Shohamy, 2004; Tracy, 2020).”
What the sources say
- Supportedcites Barbour, 2001 source
Barbour argues that prescriptive, “one size fits all” approaches cannot resolve the context-dependent choices involved in qualitative research. The Conclusion (p. 1117) says reducing qualitative research to a list of technical procedures is overly prescriptive and that no such procedure confers rigour by itself. This supports the objection to fixed, universal criteria; the separate description of a social constructivist standpoint is attributed to Kress, not Barbour. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites Hammersley, 2007 source
- Not checked — no full text of the sourcecites Lazaraton, 2003 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We agree with the position of Barbour (2001) in that double coding of transcripts offers value more in how the content of coding disagreements and the discussions that follow alert coders to alternative interpretations and shape the evolution of coding categories, rather than to indicate the exact degree of intersubjectivity.”
What the sources say
- Supportedcites Barbour, 2001 source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With these conceptions of reliability and measures undertaken to calculate and ensure them, the absence of such measures is expectedly acknowledged by some researchers as shortcomings and limitations of their research.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition, authors of a few of the studied articles appear to subsume strategies like triangulation and member checking under an attempt to take care of reliability (and validity) in their purportedly qualitative studies.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Credibility, as the most frequent one, is primarily used in relation with ' prolonged engagement with the participants ' [295] and establishing trust and in-depth familiarity and understanding”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
58 more claims
Create a free account to see all 132 claims and every source verdict.
- Probably not a substantial claim. 50
- It may state two separate claims as one. 35
Citations
Cites
Qualitative Data Analysis: An Expanded Sourcebook
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80231-2 · Cited by 48,420
Lincoln, Y. S., Guba, E. G., & Pilotta, J. J. (1985) · International Journal of Intercultural Relations · doi:10.1016/0147-1767(85)90062-8 · Cited by 45,787
Qualitative research and evaluation methods
Patton, M. Q. (1980) · Cited by 35,734
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
Graneheim, U. H., & Lundman, B. (2004) · Nurse Education Today · doi:10.1016/j.nedt.2003.10.001 · Cited by 20,772 · cited 4×
“…issues of quality within the limits of journal articles, hence our decision (and, in a sense, struggle) to stay focused on the manifest content of explicit propositions about research quality in the examined articles rather than reading between-the-line latent content (Graneheim & Lundman, 2004).”
Influential(1996) · Cited by 11,584
The SAGE handbook of qualitative research
(2005) · Choice Reviews Online · doi:10.5860/choice.43-1330 · Cited by 10,183
Qualitative Quality: Eight “Big-Tent” Criteria for Excellent Qualitative Research
Tracy, S. J. (2010) · Qualitative Inquiry · doi:10.1177/1077800410383121 · Cited by 7,536 · cited 10×
“…processes that feature implications for study quality (Mills et al., 2010), allowing greater retrospective monitoring and assessment of research (Tracy, 2010), enhancing trustworthiness (Hammersley, 2013), and facilitating ‘ thick interpretations ’ of the nature and value of the research at the…”
InfluentialInterViews: An introduction to qualitative research interviewing
Flinders, D. J. (1997) · Evaluation and Program Planning · doi:10.1016/s0149-7189(97)89858-8 · Cited by 7,238
Verification strategies for establishing reliability and validity in qualitative research
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100202 · Cited by 4,978
Research works printed but not linked: 2
- Mayring, 2019, Qualitative content analysis: Demarcation, varieties, developments, Forum: Qualitative Social Research, 20, Art. 16
- Lincoln, 2011, Sage qualitative research methods, 399
Cited by
Systematic review, systematic bias? An example from EMI research
Hampson, T., & McKinley, J. (2024) · Language Teaching · doi:10.1017/s0261444824000338 · Cited by 20
Ullah, M. M., Fatema, S., Alam, M. S., & Siddiqua, F. (2026) · Integrated Journal for Research in Arts and Humanities · doi:10.55544/ijrah.6.5.6 · Cited by 1
Research Methods for Language Education
Phakiti, A., & Lin, K. C. (2025) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.00369-0 · Cited by 1
Metadata
- Authors
- Mirhosseini, S.-A., & Pearson, W. S.
- Affiliations
- University of Hong Kong (HK)
- University of Exeter (GB)
- Corresponding author
- Mirhosseini, Seyyed-Abdolhamid
- Venue
- Language Teaching
- Publisher
- Cambridge University Press (CUP)
- Published
- 2024-04-09
- Volume
- 58
- Issue
- 1
- Pages
- 99-115
- ISSN
- 0261-4448 (ISSN-L)
- 1475-3049
- OpenAlex
- W4394605836
- Semantic Scholar
- 43dce848e5df2c3479cdf6c013232fc3cbf8a49d
- Cited by
- 17
- References
- 72
- Citations by year
- 2026: 9 · 2025: 5 · 2024: 3
- FWCI
- 21.49
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Language Teaching (publishedVersion, cc-by)
- The HKU Scholars Hub (University of Hong Kong) (submittedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Education › Educational Research and Methods
- Social Sciences › Sociology and Political Science › Qualitative Research Methods and Ethics
- Keywords
- research quality, qualitative research quality, language teaching research, quality standards, research reporting practices, quality framework
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How do language education researchers attend to quality in qualitative studies? https://app.cochat.ai/papers/10.1017/s0261444824000053
- The paper
- Mirhosseini, S.-A., & Pearson, W. S. (2025). How do language education researchers attend to quality in qualitative studies? Language Teaching. https://doi.org/10.1017/s0261444824000053
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.