A Review of the Quality Indicators of Rigor in Qualitative Research

In the paper

“Some authors have raised concerns about the use of such checklists and adherence to strict, universal criteria because they do not afford sufficient flexibility to accommodate the diverse approaches and multiple interpretive practices often represented in qualitative studies.[7][8][9][10][11]”

What the sources say

• Supported

  In “Qualitative Research Is Not a Unified Field,” the paper says that treating qualitative research as a unified field produces appraisal criteria that fit some methodologies but not others. Its introduction also reports concerns that rigid criteria can stifle interpretive and creative aspects of qualitative research. (LLM, conf=0.93)

  cites [7] 7. Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. BMJ Quality & Safety. 2004;13:223-225 source
• ?Not checked — no full text of the sourcecites [8] 8. Stige B, Malterud K, Midtgarden T. Toward an agenda for evaluation of qualitative research. Qual Health Res. 2009;19(10):1504-16. source
• ?Not checked — no full text of the sourcecites [9] 9. Wasserman JA, Clair JM, Wilson KL. Problematics of grounded theory: Innovations for developing an increasingly rigorous qualitative method. Qual Res. 2009;9(3):355-381. source

In the paper

“In recent years, multiple publications have synthesized quality criteria and recommendations for use by researchers and peer reviewers alike, often in the form of checklists.[2][3][4][5][6]”

What the sources say

• Supported

  The “Checklist for Qualitative Papers” section says the paper establishes criteria for judging qualitative research and provides a checklist to help authors and reviewers. This supports citing it as one example of the publications described; the citation need not establish the number of other publications. (LLM, conf=0.93)

  cites [2] 2. Anderson C. Presenting and evaluating qualitative research. Am J Pharm Educ. 2010; 74(8):Article 141. source
• ?Not checked — no full text of the sourcecites [3] 3. Santiago-Delefosse M, Gavin A, Bruchez C, Roux P, Stephen SL. Quality of qualitative research in the health sciences: analysis of the common criteria present in 58 assessment guidelines by expert users. Soc Sci Med. 2016;148:142-51. source
• ?Not checked — no full text of the sourcecites [4] 4. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245-1251. source

In the paper

“Developing a FINER research question is critical to study rigor and quality and should not be rushed, as all other aspects of research design depend on the focus and clarity of the research question(s) guiding the study.15”

What the sources say

• Partly supported

  The Abstract says poorly constructed questions can affect all subsequent stages of a study, and the Conclusion says good qualitative questions shape study design and analysis. The paper supports careful development of research questions, but these passages do not establish the specific FINER framework named in the claim. (LLM, conf=0.88)

  cites [15] 15. Agee J. Developing qualitative research questions: A reflective process. Int J Qual Stud Educ. 2009;22(4):431-447. sourceExperimental — accuracy not yet measured

In the paper

“Developing and refining a primary research question focused on both the phenomena of interest and the context in which it is situated is essential to research rigor and quality.”

What the sources say

• Supported

  In “Beginning the process” (page not provided), Agee says qualitative questions should inquire about phenomena and attend to the particular situation and context. In “Reflexivity and ethical considerations in developing questions” (page not provided), she says questions should reflect that particularity; she also links developing and refining questions to more rigorous inquiry. The Abstract connects question quality to subsequent stages of a study. (LLM, conf=0.88)

  cites [15] 15. Agee J. Developing qualitative research questions: A reflective process. Int J Qual Stud Educ. 2009;22(4):431-447. source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [16] 16. Creswell J. Qualitative, quantitative, and mixed methods approaches. Res Des. 2013:1-26.

In the paper

“The goal of rigor in qualitative research can be described as ensuring that the research design, method, and conclusions are explicit, public, replicable, open to critique, and free of bias.41”

What the sources say

• Partly supported

  The 'Overall Criteria for Quality in Qualitative Research' section supports making the design, methods, and conclusions transparent and systematic; the paper also discusses appraisal of their quality. But the 'Reliability' section says exact replicability is counter-intuitive for qualitative research, and 'Nature of Qualitative Research versus Quantitative Research' treats subjectivity as an essential element rather than something to eliminate as bias. Page numbers are unavailable in our copy. (LLM, conf=0.88)

  cites [41] 41. Leung L. Validity, reliability, and generalizability in qualitative research. J Fam Med Prim Care. 2015;4(3):324. sourceExperimental — accuracy not yet measured

In the paper

“Although researchers can employ great flexibility in the selection of study methods, inclusion of best practice methods for assuring the rigor and trustworthiness of results is critical to study design.”

In the paper

“Given the unique and often highly sensitive nature of data being collected by the researcher, trustworthiness is an essential component of the researcher-participant relationship.”

In the paper

“The conceptual framework contributes directly to determining sample definitions, size, and recruitment of participants.”

In the paper

“In addition to defining the sample and selecting participants, other considerations related to sampling bias may impact the quantity and quality of data generated and therefore the quality of the study result.”

In the paper

“Ringsted, Hodges, and Sherpbier describe three essential parts to an effective conceptual framework: theories and/ or concepts and principles relevant to the phenomenon of interest; what is known and unknown from prior work, observations, and examples; and the researcher's observations, ideas, and suppositions regarding the research problem statement and question.21”

What the sources say

• ?Not checked — no full text of the sourcecites [21] 21. Ringsted C, Hodges B, Scherpbier A. "The research compass": an introduction to research in medical education: AMEE Guide No. 56. Med Teach. 2011;33(9):695-709. source

In the paper

“Glassick, Huber, and Maeroff identified six criteria applicable to assessing the quality of scholarship.18,19”

What the sources say

• ?Not checked — no full text of the sourcecites [18] 18. Glassick CE, Huber MT, Maeroff GI, Boyer EL. Scholarship Assessed: Evaluation of the Professoriate. San Francisco, CA: Jossey-Bass; 1997.
• ?Not checked — no full text of the sourcecites [19] 19. Glassick CE. Boyer's expanded definitions of scholarship, the standards for assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000;75(9):877-880. source

In the paper

“By being transparent in the process and reporting of results when saturation may not have been possible, the resulting data may still contribute to the field and to further inquiry.”

In the paper

“The 2015 second edition of the Review Criteria for Research Manuscripts 14 and work by Ravitch and colleagues 23,24 provide specific guidance for applying the conceptual framework to each stage of the research process to enhance rigor and quality.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [14] 14. Durning SJ, Carline J, eds. Review criteria for research manuscripts, 2 nd ed. Washington, DC: Association of American Medical Colleges; 2015.
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [23] 23. Ravitch S, Carl N. Qualitative Research: Bridging the Conceptual, Theoretical, and Methodological. Thousand Oaks, CA: Sage; 2016.
• ?Not checked — no full text of the sourcecites [24] 24. Ravitch SM, Riggan M. Reason & Rigor: How Conceptual Frameworks Guide Research. Los Angeles, CA: Sage; 2012. source

In the paper

“Any of these factors could potentially change the nature of the relationship between the researcher and the study participants and influence the trustworthiness of data collected or the study result.”

In the paper

“Interview methods range from highly structured to increase dependability or completely open-ended to allow for interviewers to clarify a participant's response for increased credibility and confirmability.”

In the paper

“Member checking, a common standard of rigor, is a practice to increase study credibility and confirmability that involves asking a research subject to verify the transcription of an interview.1,16,28”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [1] 1. Denzin, Norman, Lincoln YS. The Sage Handbook of Qualitative Research. 4th ed. Los Angeles, CA: Sage Publications; 2011.
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [16] 16. Creswell J. Qualitative, quantitative, and mixed methods approaches. Res Des. 2013:1-26.
• ?Not checked — no full text of the sourcecites [28] 28. Whittemore R, Chase SK, Mandle CL. Validity in qualitative research. Qual Health Res. 2001;11(4):522-537. source

In the paper

“Though data saturation is probably the most common terminology used to describe the achievement of sufficient sample size, it does not apply to all study designs.”

In the paper

“Therefore, sampling design in qualitative research is not random but defined purposively to include the most appropriate participants in the most appropriate context for answering the research question.”

In the paper

“Lincoln and Guba outline four criteria for establishing the overall trustworthiness of qualitative research results: credibility, the researcher ensures and imparts to the reader supporting evidence that the results accurately represent what was studied; transferability, the researcher provides detailed contextual information such that readers can determine whether the results are applicable to”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [27] 27. Guba E, Lincoln Y. Sage Handbook of Qualitative Research. Thousand Oaks, CA: Sage; 1994.

In the paper

“Lingard describes four types of unknowns to pursue during literature review: what no one knows; what is not yet well understood; what controversy or conflicting results, understandings, or perspectives exist; and what are unproven assumptions.22”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [22] 22. Lingard L. Writing an effective literature review. Perspect Med Educ. 2018;7(1):47-49.

In the paper

“Regardless, interview protocols and structure are often modified or refined, based on concurrent data collection and analysis processes to support or refute preliminary interpretations and refine focus and continuing inquiry.”

In the paper

“In other instances, attrition and factors related to time and resources may contribute to not reaching data saturation within the limits of the study.”

In the paper

“Developing a well-integrated conceptual framework is essential to establishing a research topic based upon a thorough and integrated review of relevant literature (addressing Glassick criteria #1 and #2: clear purpose and adequate preparation).”

In the paper

“Conversely, the number of participants available for inclusion in a sample may be small and some risk of not reaching data saturation may be unavoidable.”

In the paper

“Quality criteria for assessing a study's problem statement, conceptual framework, and research question include the following: introduction builds a logical case and provides context for the problem statement; problem statement is clear and well-articulated; conceptual framework is explicit and justified; research purpose and/or question is clearly stated; and constructs being investigated are”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [14] 14. Durning SJ, Carline J, eds. Review criteria for research manuscripts, 2 nd ed. Washington, DC: Association of American Medical Colleges; 2015.
• ?Not checked — no full text of the sourcecites [24] 24. Ravitch SM, Riggan M. Reason & Rigor: How Conceptual Frameworks Guide Research. Los Angeles, CA: Sage; 2012. source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [25] 25. Miles MB, Huberman M. Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed. Thousand Oaks, CA: Sage; 1994.

In the paper

“Recent contributions from Ravitch and colleagues present best practices in developing frameworks for conceptual and methodological coherence within a study design, regardless of the research approach.23,24”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [23] 23. Ravitch S, Carl N. Qualitative Research: Bridging the Conceptual, Theoretical, and Methodological. Thousand Oaks, CA: Sage; 2016.
• ?Not checked — no full text of the sourcecites [24] 24. Ravitch SM, Riggan M. Reason & Rigor: How Conceptual Frameworks Guide Research. Los Angeles, CA: Sage; 2012. source

In the paper

“The development of a strong conceptual framework facilitates selection of appropriate study methods to minimize the bias inherent in qualitative studies and help readers to trust the research and the researcher (see 23 Glassick criteria #3 in Table 1).”

In the paper

“Convenience sampling, using the most accessible research participants, is the least rigorous approach to defining a study sample and may result in low accuracy, poor representativeness, low credibility, and lack of transferability of study results.”

What the sources say

• ?Not checked — no full text of the sourcecites [29] 29. Marshall MN. Sampling for qualitative research. Fam Pract. 1996;13(6):522-525. source

In the paper

“The first two criteria, clear purpose and adequate preparation, are directly related to formulating effective research questions and a strong conceptual framework.”

In the paper

“Reflexivity, the idea that a researcher's preconceptions and biases can influence decisions and actions throughout qualitative research activities, is a critical aspect of rigor even at the earliest stages of the study.”

In the paper

“As in any research paradigm, the goal of quality and rigor in qualitative research is to minimize the risk of bias and maximize the accuracy and credibility of research results.”

In the paper

“This framework provides detailed explanations of what should be reported in each of 21 sections of a qualitative research manuscript.”

What the sources say

• ?Not checked — no full text of the sourcecites [4] 4. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245-1251. source

In the paper

“Beginning the study with a well-developed conceptual framework and active use of both researcher reflexivity and rigorous peer review during study implementation can drive both study rigor and quality.”

In the paper

“Should any competing or controversial theories emerge during data collection or analysis, it is vital to the credibility and trustworthiness of the study that the author disclose and explore those negative cases.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [16] 16. Creswell J. Qualitative, quantitative, and mixed methods approaches. Res Des. 2013:1-26.

In the paper

“Data triangulation is used to identify convergence of data obtained through multiple data sources and methods (eg, observation field notes and interview transcripts) to avoid or minimize error or bias and optimize accuracy in data collection and analysis processes.33,35,36”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [33] 33. Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res. 1999; 34(5 Pt 2):1189-1208.
• ?Not checked — no full text of the sourcecites [35] 35. Creswell J, Miller D. Determining validity in qualitative inquiry. Theory Pract. 2000;39(3):124-130. source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [36] 36. Johnson RB. Examining the validity structure of qualitative research. Education. 1997;118(2):282.

In the paper

“The peer reviewer offers a critique of the study methods and validation of the conclusions drawn by the author as a thorough check on researcher bias.”

In the paper

“A study's conceptual framework is critical to achieving rigor and quality and provides guidance in developing predetermined notions or plans for what to observe, how to record, and how to minimize the influence of potential bias.34”

What the sources say

• ?Not checked — no full text of the sourcecites [34] 34. Mulhall A. In the field: Notes on observation in qualitative research. J Adv Nurs. 2003;41(3):306-313. source

In the paper

“To ensure dependability, the research method must be reported in detail such that the reader can determine proper research practices have been followed and that future researchers can repeat the study.40”

What the sources say

• ?Not checked — no full text of the sourcecites [40] 40. Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. Educ Inf. 2004;22(2):63-75. source

In the paper

“The use of best practices applying to data collection and data analysis facilitates the full examination of data relative to the study purpose and research question and helps to prevent premature closure of the study.”

In the paper

“Rigor is best achieved through thoughtful and deliberate planning, diligent and ongoing application of researcher reflexivity, and honest communication between the researcher and the audience regarding the study and its results.”

In the paper

“Regardless of the researcher's degree of visibility to the study participants, persistent observation of the targeted sample is critical to the confirmability standard and to achieving data saturation.”

What the sources say

• ?Not checked — no full text of the sourcecites [28] 28. Whittemore R, Chase SK, Mandle CL. Validity in qualitative research. Qual Health Res. 2001;11(4):522-537. source

In the paper

“That is, study conclusions must be clearly grounded in persistent phenomena witnessed during the study, rather than on a fluke event.28”

What the sources say

• ?Not checked — no full text of the sourcecites [28] 28. Whittemore R, Chase SK, Mandle CL. Validity in qualitative research. Qual Health Res. 2001;11(4):522-537. source

In the paper

“Thus, due to the creative interpretative and contextually bound nature of qualitative studies, the application of standards of rigor and adherence to systematic processes welldocumented in an audit trail are essential.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [33] 33. Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res. 1999; 34(5 Pt 2):1189-1208.

In the paper

“Thick and rich descriptions, where robust descriptive language is used to provide sufficient contextual information, enable the reader to determine credibility, transferability, dependability, and confirmability.”

In the paper

“The process of demonstrating the credibility of research is rooted in honest and transparent reporting of how biases and other possible confounders were identified and addressed throughout study processes.”

In the paper

“Rigor in the research process and results are achieved when each element of study methodology is systematic and transparent through complete, methodical, and accurate reporting.33”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [33] 33. Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res. 1999; 34(5 Pt 2):1189-1208.