Checklists for improving rigour in qualitative research: a case of the tail wagging the dog?

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Details of sampling are often dealt with in the methods section of papers and are disregarded in the analysis section, which often consists of little more than a description of undifferentiated themes that emerged during data analysis.

“When they are provided at all, details of sampling are often dealt with in the methods section of papers and are disregarded in the analysis section, which often consists of little more than a description of undifferentiated themes that emerged during data analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Sometimes researchers choose to disregard their own interpretations and to accept those of respondents at face value.

“Sometimes researchers choose to disregard their own interpretations and to accept those of respondents at face value.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Few published papers yield the surprises likely to be a feature of analyses driven entirely by respondents' concerns, and the terminology and theories to which papers appeal generally bear an uncanny resemblance to current disciplinary concerns and debates.

“However, few published papers yield the surprises likely to be a feature of analyses driven entirely by respondents' concerns, and the terminology and theories to which papers appeal generally bear an uncanny resemblance to current disciplinary concerns and debates.”

Probably not a substantial claim: may not pass S2 (checkable).

All too often, the tension between new themes from data and themes that could have been anticipated from the outset is not explored in the presentation of findings.

“All too often, however, the tension between these two different sorts of insight—and its potential to illuminate the topic being studied—is not explored in the presentation of findings.”

Probably not a substantial claim: may not pass S4 (based).

Several researchers have informed the author that they must comply with various procedures (such as respondent validation, multiple coding, etc) to satisfy the requirements of specific journals where they hope to publish.

“Over the past two years, several researchers have informed me that they must comply with various procedures (such as respondent validation, multiple coding, etc) in order to satisfy the requirements of specific journals where they hope to publish their work.”

Classified as probably not a substantial claim.

The five technical fixes currently enjoying the greatest popularity are purposive sampling, grounded theory, multiple coding, triangulation, and respondent validation.

“From reading recent journals and my experience of reviewing journal articles and grant submissions, I find that the five technical fixes currently enjoying the greatest popularity are purposive sampling, grounded theory, multiple coding, triangulation, and respondent validation (table).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Purposive sampling allows researchers to deliberately include "outliers" conventionally discounted in quantitative approaches.

“With purposive sampling, researchers deliberately seek to include “outliers” conventionally discounted in quantitative approaches.9”

• cites [9] Barbour, 1999, The case for combining qualitative and quantitative approaches in health services research., J Health Serv Res Policy sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

It can be useful to have another person cast an eye over segments of data or emergent coding frameworks, and this is a core activity of supervision sessions and research team meetings.

“It can be useful to have another person cast an eye over segments of data or emergent coding frameworks, and this is a core activity of supervision sessions and research team meetings.15”

• cites [15] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Such exercises encourage thoroughness, both in interrogating the data at hand and in providing an account of how an analysis was developed.

“Such exercises encourage thoroughness, both in interrogating the data at hand and in providing an account of how an analysis was developed.”

May combine two claims.

The production of similar findings from different methods merely provides corroboration or reassurance; the absence of similar findings does not provide grounds for refutation.

“The production of similar findings from different methods merely provides corroboration or reassurance; the absence of similar findings does not, however, provide grounds for refutation.”

Probably not a substantial claim: may not pass S2 (checkable).

Otherwise we risk compromising the unique contribution that systematic qualitative research can make to health services research.

“Otherwise we risk compromising the unique contribution that systematic qualitative research can make to health services research”

• Also stated in: § Conclusion · plain

  “Otherwise we run the risk of compromising the unique contribution that systematic and thoughtfully carried out qualitative research can make to health services research.”

• AssumptionRests on an assumption: Systematic qualitative research has a unique contribution to make to health services research.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Richardson suggests that it is more helpful to conceive of complementary rather than competing perspectives and offers the term "crystallisation" as an alternative to triangulation.

“Richardson suggests that it is more helpful to conceive of complementary rather than competing perspectives and offers the term “crystallisation” as an alternative to triangulation.19”

• cites [19] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In the absence of an attempt to systematically analyse the commonalities and contradictions reflected in the data, many researchers produce an artificially neat and tidy account that is descriptive rather than analytical and which militates against formulating in-depth analyses.

“In the absence of an attempt to systematically analyse the commonalities and contradictions reflected in the data, many researchers produce an artificially neat and tidy account that is descriptive rather than analytical and which militates against formulating in-depth analyses.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Most health services research involves a one-off data collection exercise, in which respondent validation may be more trouble than it is worth.

“Most health services research, however, involves a one-off data collection exercise, in which respondent validation may be more trouble than it is worth.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Whether multiple coding is carried out by a conscientious lone researcher, by a team, or by involving independent experts is immaterial; what matters is that a systematic process is followed and that this is rendered transparent in the written research project.

“Whether this is carried out by a conscientious lone researcher, by a team, or by involving independent experts is immaterial: what matters is that a systematic process is followed and that this is rendered transparent in the written research project.”

Probably not a substantial claim: may not pass S2 (checkable).

Checklists can be counterproductive if used prescriptively.

“Although checklists have undoubtedly contributed to the wider acceptance of such methods, these can be counterproductive if used prescriptively.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

A sleight of hand produces a list of "themes," and readers are invited to take it on trust that theory somehow emerges from the data without being offered a step by step explanation of how theoretical insights have been built up.

“A sleight of hand produces a list of “themes,” and we are invited to take it on trust that theory somehow emerges from the data without being offered a step by step explanation of how theoretical insights have been built up.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

As Mays and Pope point out, researchers seek to provide an overview whereas respondents have individual concerns, and this can result in apparently discrepant accounts.

“As Mays and Pope point out, researchers seek to provide an overview whereas respondents have individual concerns, and this can result in apparently discrepant accounts.20”

• cites [20] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Checklists have contributed to the wider acceptance of qualitative research methods and to convincing potential sceptics of its thoroughness.

“Checklists have played an important role in conferring respectability on qualitative research and in convincing potential sceptics of its thoroughness.1–3”

• cites [1] sourceNot checked yet

  Not checked in the free run.

• cites [2] sourceNot checked yet

  Not checked in the free run.

• cites [3] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In practice, researchers are unlikely to obtain research funding without having carried out a thorough literature review or having formulated some idea of the content of the data they are likely to collect.

“In practice, however, you are unlikely to obtain research funding without having carried out a thorough literature review or having formulated some idea of the content of the data you are likely to collect.”

May combine two claims.

Respondents' reactions to emerging findings can help refine explanations, as can key informants'.

“Respondents' reactions to emerging findings can certainly help refine explanations—as can key informants'—but several commentators have questioned whether it is always appropriate.20,21”

• cites [20] sourceNot checked yet

  Not checked in the free run.

• cites [21] Bloor M .Techniques of validation in qualitative research: a critical commentary. In: Miller G Dingwall R , eds. Context and method in qualitative research. London:Sage,1997:37–50. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multiple coding concerns the same issue as the quantitative equivalent "inter-rater reliability" and is a response to the charge of subjectivity sometimes levelled at the process of qualitative data analysis.

“Multiple coding concerns the same issue as the quantitative equivalent “inter-rater reliability” and is a response to the charge of subjectivity sometimes levelled at the process of qualitative data analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Purposive sampling offers researchers a degree of control rather than being at the mercy of any selection bias inherent in pre-existing groups.

“Purposive (or theoretical8) sampling, however, offers researchers a degree of control rather than being at the mercy of any selection bias inherent in pre-existing groups (such as clinic populations).”

• cites [8] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Uncritical adoption of grounded theory can result in explanations tinged with the "near mysticism" that Melia derides in the original text on grounded theory.

“Uncritical adoption of grounded theory can result in explanations tinged with the “near mysticism” that Melia derides in the original text on grounded theory.12”

• cites [12] Glaser B Strauss A .The discovery of grounded theory. Chicago:Aldine,1967.Not checked yet

  Not checked in the free run.

May combine two claims.

Respondent validation exercises, such as reading of drafts, make considerable demands on participants' time and, depending on the research topic and content of transcripts, can even be exploitative or distressing.

“Respondent validation exercises, such as reading of drafts, make considerable demands on participants' time and, depending on the research topic and content of transcripts, can even be exploitative or distressing.23”

• cites [23] Barbour RS .Engagement, representation and presentation in research practice. In: Barbour RS Huby G , eds. Meddling with mythology: AIDS and the social construction of knowledge. London:Routledge,1998:183–200.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Respondent validation can be particularly valuable in action research projects, where researchers work with participants on an ongoing basis to facilitate change.

“Respondent validation can be particularly valuable in action research projects, where researchers work with participants on an ongoing basis to facilitate change.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Some element of multiple coding can be a valuable strategy.

“While I would caution against multiple coding of entire datasets (on the grounds of economy in both cost and effort), some element of multiple coding can be a valuable strategy.”

Probably not a substantial claim: may not pass S2 (checkable).

The greatest potential of multiple coding lies in its capacity to furnish alternative interpretations and thereby to act as the "devil's advocate" in alerting researchers to all potentially competing explanations.

“The greatest potential of multiple coding lies in its capacity to furnish alternative interpretations and thereby to act as the “devil's advocate” implied in many of the checklists1–3 in alerting researchers to all potentially competing explanations.”

• cites [1] sourceNot checked yet

  Not checked in the free run.

• cites [2] sourceNot checked yet

  Not checked in the free run.

• cites [3] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Accepting respondents' accounts at face value can be cosy but may lead to collusion; Atkinson has warned of the dangers of "romanticising" respondents' accounts.

“This can be cosy but may lead to collusion: Atkinson has warned of the dangers of “romanticising” respondents' accounts.22”

• cites [22] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

The degree of concordance between researchers is not really important; what is ultimately of value is the content of disagreements and the insights that discussion can provide for refining coding frames.

“However, the degree of concordance between researchers is not really important; what is ultimately of value is the content of disagreements and the insights that discussion can provide for refining coding frames.”

Probably not a substantial claim: may not pass S2 (checkable).

Using qualitative datasets to full advantage involves applying the constant comparative method to continuously compare the views and experiences of respondents selected purposively to illuminate subtle but potentially important differences.

“That would involve applying the constant comparative method11 to continuously compare the views and experiences of respondents who have been selected precisely—indeed, purposively—in order to illuminate subtle but potentially important differences.”

• cites [11] Green, 1998, Commentary: grounded theory and the constant comparative method., BMJNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Some strategies claimed as examples of purposive sampling in effect involve hybrids that retain elements of random or convenience sampling and are unlikely to yield the spread of respondents required.

“Some strategies claimed as examples of purposive sampling in effect involve hybrids, which retain elements of random or convenience sampling and which are unlikely to yield the spread of respondents required.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In its purest form, grounded theory alleges that all explanations or theories are derived from the dataset itself rather than from a researcher's prior theoretical viewpoint.

“In its purest form the grounded theory approach to data analysis alleges that all explanations or theories are derived from the dataset itself rather than from a researcher's prior theoretical viewpoint.12”

• cites [12] Glaser B Strauss A .The discovery of grounded theory. Chicago:Aldine,1967.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Triangulation addresses the issue of internal validity by using more than one method of data collection to answer a research question.

“Triangulation addresses the issue of internal validity by using more than one method of data collection to answer a research question.”

Probably not a substantial claim: may not pass S2 (checkable).

Such approaches do not use qualitative datasets to full advantage.

“Such approaches do not use qualitative datasets to full advantage.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Triangulation relies on the notion of a fixed point, or superior explanation, against which other interpretations can be measured.

“Triangulation relies on the notion of a fixed point, or superior explanation, against which other interpretations can be measured.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Reducing qualitative research to a list of technical procedures is overly prescriptive and results in "the tail wagging the dog."

“Reducing qualitative research to a list of technical procedures (such as purposive sampling, grounded theory, multiple coding, triangulation, and respondent validation) is overly prescriptive and results in “the tail wagging the dog””

• Also stated in: § Conclusion · plain

  “Reducing qualitative research to a list of technical procedures, however extensive, is overly prescriptive and results in “the tail wagging the dog.””

Probably not a substantial claim: may not pass S3 (load-bearing).

As Mays and Pope conclude, comprehensiveness may be a more realistic goal for qualitative research than is internal validity.

“As Mays and Pope conclude, comprehensiveness may be a more realistic goal for qualitative research than is internal validity.20”

• cites [20] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Qualitative research, with its distinctive approach to harnessing the analytical potential of exceptions, allows a research question to be examined from various angles.

“Qualitative research, with its distinctive approach to harnessing the analytical potential of exceptions, allows a research question to be examined from various angles.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Purposive sampling allows deviant cases to illuminate, by juxtaposition, processes and relations that routinely come into play, enabling "the exception to prove the rule."

“It allows for such deviant cases to illuminate, by juxtaposition, those processes and relations that routinely come into play, thereby enabling “the exception to prove the rule.”9,10”

• cites [9] Barbour, 1999, The case for combining qualitative and quantitative approaches in health services research., J Health Serv Res Policy sourceNot checked yet

  Not checked in the free run.

• cites [10] Frankland J Bloor M .Some issues arising in the systematic analysis of focus group materials. In: Barbour RS Kitzinger J , eds. Developing focus group research: politics, theory and practice. London:Sage,1999:144–155. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Several commentators have questioned whether respondent validation is always appropriate.

“Respondents' reactions to emerging findings can certainly help refine explanations—as can key informants'—but several commentators have questioned whether it is always appropriate.20,21”

• cites [20] sourceNot checked yet

  Not checked in the free run.

• cites [21] Bloor M .Techniques of validation in qualitative research: a critical commentary. In: Miller G Dingwall R , eds. Context and method in qualitative research. London:Sage,1997:37–50. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Technical fixes can strengthen the rigour of qualitative research only if they are embedded in a broader understanding of qualitative research design and data analysis.

“Technical fixes will achieve little unless they are embedded in a broader understanding of the rationale and assumptions behind qualitative research.”

• Also stated in: § Conclusion · plain

  “They can strengthen the rigour of qualitative research only if they are embedded in a broad understanding of qualitative research design and data analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

According to this approach, apparent contradictions (or exceptions) do not pose a threat to researchers' explanations; they merely provide further scope for refining theories.

“According to this approach, apparent contradictions (or exceptions) do not pose a threat to researchers' explanations; they merely provide further scope for refining theories.”

Classified as probably not a substantial claim.

Each of the technical fixes discussed has limitations.

“Although some of the technical fixes discussed here may seem appealing in the face of the dual imperatives of securing grant funding and publication, each has limitations.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Grounded theory addresses original theorising and has realistic potential of developing existing theory or, occasionally, new theories.

“Grounded theory Original theorising Developing existing theory or, occasionally, new theories”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Respondent validation addresses confirmation or refutation of interpretations and has realistic potential of corroborating or, more often, refining findings.

“Respondent validation Confirmation or refutation of interpretations Corroborating or, more often, refining findings”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Triangulation addresses confirmation or refutation of internal validity and has realistic potential of corroborating or, more often, refining findings.

“Triangulation Confirmation or refutation of internal validity Corroborating or, more often, refining findings”

May combine two claims.

Purposive sampling addresses bias and has realistic potential of enhancing sample coverage and providing a framework for analysis.

“Purposive sampling Bias Enhancing sample coverage and providing a framework for analysis”

May combine two claims.

Multiple coding addresses inter-rater reliability and has realistic potential of refining interpretations or coding frameworks.

“Multiple coding Inter-rater reliability Refining interpretations or coding frameworks”