The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
Should We Exclude Inadequately Reported Studies From Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews
Abstract
The role of critical appraisal of qualitative studies in systematic reviews remains an ongoing cause for debate. Key to such a debate is whether quality assessment can or should be used to exclude studies. In our study, we extended the use of existing criteria to assess the quality of reporting of studies included in two qualitative systematic reviews. We then excluded studies deemed to be inadequately reported from the subsequent analysis. We tested the impact of these exclusions on the overall findings of the synthesis and its depth or thickness. Exclusion of so-called inadequately reported studies had no meaningful effect on the synthesis. There was a correlation between quality of reporting of a study and its values as a source for the final synthesis. We propose that there is a possible case for excluding inadequately reported studies from qualitative evidence synthesis.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results of a study that are reported can be trustworthy (3, 22).”
From The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018) · cites this paper as [22]
In the citing paper
“We used the Quality of Reporting Tool [36] to critically appraise the included sources of evidence.”
From Experiences of Using Digital Mindfulness-Based Interventions: Rapid Scoping Review and Thematic Synthesis (2023) · cites this paper as [36]
In the citing paper
“To account for the methodological variety of studies, the quality of reporting tool by Carroll et al [22] was used to assess quality.”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [22]
Citations
Cites
Qualitative evaluation and research methods
Ellis, D. J. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,943
Qualitative Data Analysis: A Sourcebook of New Methods
Lotto, L. S., Miles, M. B., & Huberman, A. M. (1986) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737008003329 · Cited by 10,357
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
(1990) · Cited by 8,425
Qualitative Research: Rigour and qualitative research
Mays, N., & Pope, C. (1995) · BMJ · doi:10.1136/bmj.311.6997.109 · Cited by 2,715
Checklists for improving rigour in qualitative research: a case of the tail wagging the dog?
Barbour, R. S. (2001) · BMJ · doi:10.1136/bmj.322.7294.1115 · Cited by 2,613
Methods for the synthesis of qualitative research: a critical review
Barnett-Page, E., & Thomas, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-59 · Cited by 2,386
Validity in qualitative research
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001) · Qualitative Health Research · doi:10.1177/104973201129119299 · Cited by 2,144
Synthesising qualitative and quantitative evidence: A review of possible methods
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005) · Journal of Health Services Research & Policy · doi:10.1177/135581960501000110 · Cited by 2,132
Research works printed but not linked: 8
- Conole G., Hall M., Smith S. (2002). An evaluation of an online course for medical practitioners. Educational Technology & Society, 5, 66-75.
- 10.1258/1355819052801804 · DOI · Paper page
- 10.1177/1049732311402096 · DOI
- 10.2105/AJPH.89.2.18 · DOI
- Nelson M., Quinney D. (1997). Evaluating a school-based health clinic. Health Visitor, 70, 419-421.
- 10.1136/bmj.322.7294.1115 · DOI · Paper page
- Tanner K., Kirton A., Stone N., Ingham R. (2003). Evaluating the effectiveness of Time 4U services and associated educational interventions in Worcestershire: Summary of main findings. Worcester, UK: Worcester County Council.
- Washkansky G. (2008). Sexual health drop in service. London: London Borough of Hammersmith & Fulham.
Cited by
Sorted by citations
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069
“To account for the methodological variety of studies, the quality of reporting tool by Carroll et al [22] was used to assess quality.”
MethodLong, H. A., French, D. P., & Brooks, J. M. (2020) · Research Methods in Medicine & Health Sciences · doi:10.1177/2632084320947559 · Cited by 1,112
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107
Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.008 · Cited by 1,082
“Compared to reporting quality criteria, methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results that are reported can be trustworthy [43].”
Background
Metadata
- Authors
- Carroll, C., Booth, A., & Lloyd-Jones, M.
- Affiliations
- University of Sheffield (GB)
- Venue
- Qualitative Health Research
- Publisher
- SAGE Publications
- Published
- 2012-08-03
- Volume
- 22
- Issue
- 10
- Pages
- 1425-1434
- ISSN
- 1049-7323 (ISSN-L)
- 1552-7557
- OpenAlex
- W2161865743
- Cited by
- 248
- References
- 49
- Citations by year
- 2026: 16 · 2025: 22 · 2024: 26 · 2023: 14 · 2022: 21 · 2021: 22 · 2020: 31 · 2019: 17 · 2018: 20 · 2017: 13 · 2016: 18 · 2015: 10 · 2014: 7 · 2013: 10 · 2012: 1
- FWCI
- 2.45
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Qualitative Health Research (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- study exclusion, critical appraisal, quality assessment, reporting quality, qualitative evidence synthesis, sensitivity analysis
- MeSH terms
- Review Literature as Topic, Sensitivity and Specificity, Reproducibility of Results, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
