A Comparative Analysis of Three Online Appraisal Instruments’ Ability to Assess Validity in Qualitative Research
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The concept of validity has been a central component in critical appraisal exercises evaluating the methodological quality of quantitative studies. Reactions by qualitative researchers have been mixed in relation to whether or not validity should be applied to qualitative research and if so, what criteria should be used to distinguish high-quality articles from others. We compared three online critical appraisal instruments’ ability to facilitate an assessment of validity. Many reviewers have used the critical appraisal skills program (CASP) tool to complete their critical appraisal exercise; however, CASP appears to be less sensitive to aspects of validity than the evaluation tool for qualitative studies (ETQS) and the Joanna Briggs Institute (JBI) tool. The ETQS provides detailed instructions on how to interpret criteria; however, it is the JBI tool, with its focus on congruity, that appears to be the most coherent.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This tool is the most coherent and sensitive tool for validity, with its focus on congruity.”
From The impact of health literacy interventions on glycemic control and self‐management outcomes among type 2 diabetes mellitus: A systematic review (2023) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1984) · Cited by 81,983
Qualitative Data Analysis: An Expanded Sourcebook
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80231-2 · Cited by 48,420
Lincoln, Y. S., Guba, E. G., & Pilotta, J. J. (1985) · International Journal of Intercultural Relations · doi:10.1016/0147-1767(85)90062-8 · Cited by 45,787
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,535
Determining Validity in Qualitative Inquiry
Creswell, J. W., & Miller, D. L. (2000) · Theory Into Practice · doi:10.1207/s15430421tip3903_2 · Cited by 9,184 · cited 1×
“Creswell and Miller (2000) suggested prolonged engagement in the field—also referred to as persistent observation—to improve theoretical validity.”
Background(2014) · The SAGE Encyclopedia of Action Research · doi:10.4135/9781446294406.n148 · Cited by 7,250
Neufeld, A., & Ryan, A. G. (1988) · Canadian Journal of Program Evaluation · doi:10.3138/cjpe.3.006 · Cited by 5,913
Verification strategies for establishing reliability and validity in qualitative research
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100202 · Cited by 4,978
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Understanding and Validity in Qualitative Research
Maxwell, J. (1992) · Harvard Educational Review · doi:10.17763/haer.62.3.8323320856251826 · Cited by 2,951 · cited 5×
“The following paragraphs expand on this table and describe the five types of validity identified by Maxwell (1992).”
BackgroundMethodInfluentialThe problem of rigor in qualitative research
Sandelowski, M. (1986) · Advances in Nursing Science · doi:10.1097/00012272-198604000-00005 · Cited by 2,772 · cited 1×
“These translations have been criticized by some and welcomed by others (Sandelowski, 1986; 1Catholic University Leuven, Leuven, Belgium 2University of Adelaide, Adelaide, South Australia, Australia Corresponding Author: Karin Hannes, Catholic University Leuven, Centre for Methodology of Educational…”
BackgroundQualitative Research: Rigour and qualitative research
Mays, N., & Pope, C. (1995) · BMJ · doi:10.1136/bmj.311.6997.109 · Cited by 2,715
The Enlightened Eye: Qualitative Inquiry and the Enhancement of Educational Practice
Gray, J., & Eisner, E. W. (1994) · Studies in Art Education · doi:10.2307/1320828 · Cited by 2,469
Validity in qualitative research
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001) · Qualitative Health Research · doi:10.1177/104973201129119299 · Cited by 2,144
Quality in Qualitative Research
Seale, C. (1999) · Qualitative Inquiry · doi:10.1177/107780049900500402 · Cited by 1,743
(1997) · Cited by 1,249
III. “Degrees of Freedom” and the Case Study
Campbell, D. T. (1975) · Comparative Political Studies · doi:10.1177/001041407500800204 · Cited by 1,060
Giacomini, M. K., Cook, D. J., & for the Evidence-Based Medicine Working Group (2000) · JAMA · doi:10.1001/jama.284.3.357 · Cited by 931 · cited 1×
“The accuracy of reporting can be increased through the use of multiple observers recording and describing the participants’ behavior and context, which allows for cross checking of observations (Giacomini & Cook, 2000).”
BackgroundEvaluative Criteria for Qualitative Research in Health Care: Controversies and Recommendations
Cohen, D. J., & Crabtree, B. F. (2008) · The Annals of Family Medicine · doi:10.1370/afm.818 · Cited by 829
Toward an agenda for evaluation of qualitative research
Stige, B., Malterud, K., & Midtgarden, T. (2009) · Qualitative Health Research · doi:10.1177/1049732309348501 · Cited by 327
Synthesizing qualitative research: a review of published reports
Dixon-Woods, M., Booth, A., & Sutton, A. J. (2007) · Qualitative Research · doi:10.1177/1468794107078517 · Cited by 260
The Problem of Criteria for Judging Interpretive Inquiry
Smith, J. K. (1984) · Educational Evaluation and Policy Analysis · doi:10.2307/1163977 · Cited by 153
Rigour in qualitative research
Whiteley, A. M. (2002) · eSpace (Curtin University) · Cited by 99
Attree, P., & Milton, B. S. (2006) · Evidence & Policy · doi:10.1332/174426406775249688 · Cited by 40
Cochrane Qualitative Research Methods Group
Harden, A., Hannes, K., Lockwood, C. S., Noyes, J., Harris, J., & Booth, A. (2011) · Lirias · Cited by 29
Self-citation
Research works printed but not linked: 12
- Brody, H. ( 1992). Philosophic approaches. In B. Crabtree & W. Miller (Eds.), Doing qualitative research (pp. 174-185). Newbury Park, CA: Sage.
- Burke Johnson, R., 1998, Education, 118, 282
- Denzin, N.K., 1978, The research act: A theoretical orientation to sociological methods
- Hannes K., 2009, Chapter 6: Critical appraisal of qualitative research
- Hannes, K., 2010, Update on a review of published qualitative evidence syntheses: Moving toward more systematic and transparent approaches
- Harden, A., Paper presented at the Research Methods Festival of the National Centre for Research Methods
- Health Care Practice Research & Development Unit., 2009, Evaluation tool for qualitative research
- Joanna Briggs Institute., 2007, SUMARI: The Joanna Briggs Institute system for the unified management, assessment and review of information
- Public Health Resource Unit, 2009, Critical appraisal tool for qualitative studies
- 10.3102/01623737006004379 · doi:10.3102/01623737006004379 · Paper page
- Wilson, T.D., Paper presented at the Fourth International Conference on Conceptions of Library and Information Science, Seattle
- Winter, G.A., 2000, The Qualitative Report, 4
Cited by
Lockwood, C., Munn, Z., & Porritt, K. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000062 · Cited by 2,456
Self-citationPace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.002 · Cited by 1,331
The pillars of trustworthiness in qualitative research
Ahmed, S. K. (2024) · Journal of Medicine Surgery and Public Health · doi:10.1016/j.glmedi.2024.100051 · Cited by 1,283
Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0468-4 · Cited by 1,164
“Similarly, this type of review can explain and explore why an intervention is not adopted in spite of evidence of its effectiveness [4, 13, 25].”
BackgroundSelf-citationTondeur, J., van Braak, J., Ertmer, P. A., & Ottenbreit-Leftwich, A. (2017) · Educational Technology Research and Development · doi:10.1007/s11423-016-9481-2 · Cited by 1,131
Metadata
- Authors
- Hannes, K., Lockwood, C., & Pearson, A.
- Venue
- Qualitative Health Research
- Publisher
- SAGE Publications
- Published
- 2010-07-29
- Volume
- 20
- Issue
- 12
- Pages
- 1736-1743
- ISSN
- 1049-7323 (ISSN-L)
- 1552-7557
- OpenAlex
- W2116511599
- Cited by
- 409
- References
- 25
- FWCI
- 7.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.