The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
Should We Exclude Inadequately Reported Studies From Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews
Abstract
The role of critical appraisal of qualitative studies in systematic reviews remains an ongoing cause for debate. Key to such a debate is whether quality assessment can or should be used to exclude studies. In our study, we extended the use of existing criteria to assess the quality of reporting of studies included in two qualitative systematic reviews. We then excluded studies deemed to be inadequately reported from the subsequent analysis. We tested the impact of these exclusions on the overall findings of the synthesis and its depth or thickness. Exclusion of so-called inadequately reported studies had no meaningful effect on the synthesis. There was a correlation between quality of reporting of a study and its values as a source for the final synthesis. We propose that there is a possible case for excluding inadequately reported studies from qualitative evidence synthesis.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results of a study that are reported can be trustworthy (3, 22).”
From The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers (2018) · cites this paper as [22]
In the citing paper
“We used the Quality of Reporting Tool [36] to critically appraise the included sources of evidence.”
From Experiences of Using Digital Mindfulness-Based Interventions: Rapid Scoping Review and Thematic Synthesis (2023) · cites this paper as [36]
In the citing paper
“To account for the methodological variety of studies, the quality of reporting tool by Carroll et al [22] was used to assess quality.”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [22]
Citations
Cites
Qualitative evaluation and research methods
Ellis, D. J. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,943
Qualitative Data Analysis: A Sourcebook of New Methods
Lotto, L. S., Miles, M. B., & Huberman, A. M. (1986) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737008003329 · Cited by 10,357
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
(1990) · Cited by 8,425
Qualitative Research: Rigour and qualitative research
Mays, N., & Pope, C. (1995) · BMJ · doi:10.1136/bmj.311.6997.109 · Cited by 2,715
Checklists for improving rigour in qualitative research: a case of the tail wagging the dog?
Barbour, R. S. (2001) · BMJ · doi:10.1136/bmj.322.7294.1115 · Cited by 2,613
Methods for the synthesis of qualitative research: a critical review
Barnett-Page, E., & Thomas, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-59 · Cited by 2,386
Validity in qualitative research
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001) · Qualitative Health Research · doi:10.1177/104973201129119299 · Cited by 2,144
Synthesising qualitative and quantitative evidence: A review of possible methods
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005) · Journal of Health Services Research & Policy · doi:10.1177/135581960501000110 · Cited by 2,132
Pawson, R. (2006) · doi:10.4135/9781849209120 · Cited by 1,791
Emerging Criteria for Quality in Qualitative and Interpretive Research
Lincoln, Y. S. (1995) · Qualitative Inquiry · doi:10.1177/107780049500100301 · Cited by 1,568
Handbook for Synthesizing Qualitative Research
Sandelowski, M., & Barroso, J. V. (2006) · Medical Entomology and Zoology · Cited by 1,096
Popay, J., Rogers, A., & Williams, G. (1998) · Qualitative Health Research · doi:10.1177/104973239800800305 · Cited by 987
Haussler, S. C., & Cloutterbuck, J. (2008) · The Journal of Continuing Education in Nursing · doi:10.3928/00220124-20080101-07 · Cited by 941
Campbell, R., Pound, P., Pope, C., Britten, N., Pill, R., Morgan, M., & Donovan, J. (2003) · Social Science & Medicine · doi:10.1016/s0277-9536(02)00064-3 · Cited by 921
Grol, R., Dalhuijsen, J., Thomas, S., Veld, C. I. ‘., Rutten, G., & Mokkink, H. (1998) · BMJ · doi:10.1136/bmj.317.7162.858 · Cited by 877
How can systematic reviews incorporate qualitative research? A critical perspective
Dixon-Woods, M., Bonas, S., Booth, A., Jones, D. R., Miller, T., Sutton, A. J., Shaw, R. L., Smith, J. A., & Young, B. (2006) · Qualitative Research · doi:10.1177/1468794106058867 · Cited by 860
Self-citationMeta-Study of Qualitative Health Research
Paterson, B., Thorne, S., Canam, C., & Jillings, C. (2001) · doi:10.4135/9781412985017 · Cited by 664
The problem of appraising qualitative research
Dixon-Woods, M. (2004) · Quality and Safety in Health Care · doi:10.1136/qshc.2003.008714 · Cited by 653
Dixon-Woods, M., Sutton, A., Shaw, R., Miller, T., Smith, J., Young, B., Bonas, S., Booth, A., & Jones, D. (2007) · Journal of Health Services Research & Policy · doi:10.1258/135581907779497486 · Cited by 546
Self-citationUsing qualitative metasummary to synthesize qualitative and quantitative descriptive findings
Sandelowski, M., Barroso, J. V., & Voils, C. I. (2007) · Research in Nursing & Health · doi:10.1002/nur.20176 · Cited by 541
Hannes, K., Lockwood, C. S., & Pearson, A. (2010) · Qualitative Health Research · doi:10.1177/1049732310378656 · Cited by 469
Carroll, C., Booth, A., & Cooper, K. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-29 · Cited by 461
Self-citationA multidimensional conceptual framework for analysing public involvement in health services research
Oliver, S. R., Rees, R. W., Clarke‐Jones, L., Milne, R., Oakley, A., Gabbay, J., Stein, K., Buchanan, P., & Gyte, G. M. (2008) · Health Expectations · doi:10.1111/j.1369-7625.2007.00476.x · Cited by 360
Harden, A., Garcia, J., Oliver, S., Rees, R., Shepherd, J., Brunton, G., & Oakley, A. (2004) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2003.014829 · Cited by 328
Eakin, J. M., & Mykhalovskiy, E. (2003) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2003.00392.x · Cited by 296
Noyes, J., & Popay, J. (2007) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2006.04092.x · Cited by 269
Qualitative research in systematic reviews
Dixon‐Woods, M., & Fitzpatrick, R. M. (2001) · BMJ · doi:10.1136/bmj.323.7316.765 · Cited by 232
Toward Methodological Emancipation in Applied Health Research
Thorne, S. E. (2010) · Qualitative Health Research · doi:10.1177/1049732310392595 · Cited by 147
Carroll, C., Booth, A., Papaioannou, D. E., Sutton, A., & Wong, R. (2009) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.20041 · Cited by 112
Self-citationReasons for delay in contraceptive clinic utilization
Zabin, L. S., Stark, H. A., & Emerson, M. R. (1991) · Journal of Adolescent Health · doi:10.1016/0197-0070(91)90015-e · Cited by 93
Wilkinson, A., Forbes, A., Bloomfield, J. G., & Gee, C. F. (2004) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2003.11.001 · Cited by 89
How to Appraise the Studies: An Introduction to Assessing Study Quality
Petticrew, M. P., & Roberts, H. M. (2006) · Wiley eBooks · doi:10.1002/9780470754887.ch5 · Cited by 65
Metasynthesis: a guide to knitting smoke
Downe, S. M. (2008) · Cited by 56
Carroll, C., Lloyd-Jones, M., Cooke, J., & Owen, J. (2011) · Journal of Public Health · doi:10.1093/pubmed/fdr103 · Cited by 44
Self-citationAddressing the issue of e‐learning and online genetics for health professionals
Gresty, K. A., Skirton, H., & Evenden, A. J. (2007) · Nursing and Health Sciences · doi:10.1111/j.1442-2018.2007.00296.x · Cited by 33
Students' Acquisition and Use of School Condoms in a High School Condom Availability Program
Schuster, M. A., Bell, R. M., Berry, S. H., & Kanouse, D. E. (1997) · PEDIATRICS · doi:10.1542/peds.100.4.689 · Cited by 32
Can the Internet widen participation in reproductive medicine education for professionals?
Whittington, K. (2004) · Human Reproduction · doi:10.1093/humrep/deh333 · Cited by 21
Guttmacher, S. J., Lieberman, L., Hc, W., Ward, D. M., Radosh, A., Rafferty, Y., & Freudenberg, N. (1995) · PubMed · Cited by 19
Delivering multiprofessional web-based psychosocial education: the lessons learnt
Kinghorn, S. (2005) · International Journal of Palliative Nursing · doi:10.12968/ijpn.2005.11.8.19613 · Cited by 14
CPD for GPs using the THOR-GP website
Thorley, K., Turner, S., Hussey, L. J., Zarin, N., & Agius, R. M. (2007) · Occupational Medicine · doi:10.1093/occmed/kqm116 · Cited by 13
Sex, Kids and Politics: Health Services in Schools
Emihovich, C. A., & Herrington, C. D. (1997) · Medical Entomology and Zoology · Cited by 11
Safer medicine administration through the use of e-learning.
Hare, C., Davies, C., & Shepherd, M. (2006) · PubMed · Cited by 11
Community Reactions to Reproductive Health Care at Three School‐Based Clinics in Louisiana
Zeanah, P. D., Morse, E. V., Simon, P. M., Stock, M. R., Pratt, J. L., & Sterne, S. (1996) · Journal of School Health · doi:10.1111/j.1746-1561.1996.tb06277.x · Cited by 9
An evaluation of Brook sexual health outreach in schools
Salmon, D., & Ingram, J. C. (2008) · Bristol Research (University of Bristol) · Cited by 8
Information Technology in Nursing
(2018) · Springer Publishing Company eBooks · doi:10.1891/9780826121820.0009 · Cited by 7
Hurst, J. (2005) · EDTNA-ERCA Journal · doi:10.1111/j.1755-6686.2005.tb00419.x · Cited by 4
Research works printed but not linked: 8
- Conole G., Hall M., Smith S. (2002). An evaluation of an online course for medical practitioners. Educational Technology & Society, 5, 66-75.
- 10.1258/1355819052801804 · DOI · Paper page
- 10.1177/1049732311402096 · DOI
- 10.2105/AJPH.89.2.18 · DOI
- Nelson M., Quinney D. (1997). Evaluating a school-based health clinic. Health Visitor, 70, 419-421.
- 10.1136/bmj.322.7294.1115 · DOI · Paper page
- Tanner K., Kirton A., Stone N., Ingham R. (2003). Evaluating the effectiveness of Time 4U services and associated educational interventions in Worcestershire: Summary of main findings. Worcester, UK: Worcester County Council.
- Washkansky G. (2008). Sexual health drop in service. London: London Borough of Hammersmith & Fulham.
Cited by
Sorted by citations
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737
“Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).”
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069
“To account for the methodological variety of studies, the quality of reporting tool by Carroll et al [22] was used to assess quality.”
MethodLong, H. A., French, D. P., & Brooks, J. M. (2020) · Research Methods in Medicine & Health Sciences · doi:10.1177/2632084320947559 · Cited by 1,112
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107
Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study
Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.008 · Cited by 1,082
“Compared to reporting quality criteria, methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the results that are reported can be trustworthy [43].”
Background
Metadata
- Authors
- Carroll, C., Booth, A., & Lloyd-Jones, M.
- Affiliations
- University of Sheffield (GB)
- Venue
- Qualitative Health Research
- Publisher
- SAGE Publications
- Published
- 2012-08-03
- Volume
- 22
- Issue
- 10
- Pages
- 1425-1434
- ISSN
- 1049-7323 (ISSN-L)
- 1552-7557
- OpenAlex
- W2161865743
- Cited by
- 248
- References
- 49
- Citations by year
- 2026: 16 · 2025: 22 · 2024: 26 · 2023: 14 · 2022: 21 · 2021: 22 · 2020: 31 · 2019: 17 · 2018: 20 · 2017: 13 · 2016: 18 · 2015: 10 · 2014: 7 · 2013: 10 · 2012: 1
- FWCI
- 2.45
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Qualitative Health Research (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Qualitative Research Methods and Applications
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- study exclusion, critical appraisal, quality assessment, reporting quality, qualitative evidence synthesis, sensitivity analysis
- MeSH terms
- Review Literature as Topic, Sensitivity and Specificity, Reproducibility of Results, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
