Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Open accessThe DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index.
Abstract
Trisha Greenhalgh and Jill Russell discuss the relative merits of “scientific” and “social practice” approaches to evaluation and argue that eHealth evaluation is in need of a paradigm shift.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from a review of eHealth innovations [12] indicates that centralised diffusion systems such as those considered here were more effective in disseminating authoritative evidence and preserving the ‘fidelity’ [20] of interventions, but less successful in terms of adopter engagement, partly because of strict limits on the extent to which implementation could be modified to address the”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Too often the focus is limited to defining an appropriate technical package – reporting formats, equipment, software, training, etc. – which long experience in OECD countries suggests is a necessary but by no means sufficient requirement [10]. Failure may just as easily result from institutional or social constraints, and stakeholder analysis may be as important as traditional information systems”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While some implementation frameworks were designed around a rigid (and apparently systematic) logic model [51], we have previously argued that such an approach is counterproductive because eHealth technologies are typically introduced into a complex system in a turbulent and contested policy context [52].”
From Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies (2017) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, choice of appropriate methods for such evaluations remains a contested issue [29,30].”
The cited source says what the claim attributes to it.
In the citing paper
“Rather, in this work we adopted the interpretive position that trades off notions of sampling, significance and generalizability in favor of depth of enquiry, contextualization, abstraction, dialogue and critical reflection [30,60,70].”
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 1×
“, people perceive issues in different ways and assign different values and significance to facts)—hence, reality is never objectively or unproblematically knowable—and that the identity and values of the researcher are inevitably implicated in the research process [10].”
BackgroundMaking Social Science Matter: Why Social Inquiry Fails and How it Can Succeed Again
Flyvbjerg, B. (2001) · doi:10.1017/cbo9780511810503 · Cited by 4,681
Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory
DeSanctis, G., & Poole, M. S. (1994) · Organization Science · doi:10.1287/orsc.5.2.121 · Cited by 3,879 · cited 1×
“Controlled, feature-at-a-time comparisons are vulnerable to repeated decomposition: there are features within features, contingencies within contingencies, and tasks within tasks [25].”
BackgroundStudying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (1991) · Information Systems Research · doi:10.1287/isre.2.1.1 · Cited by 3,349 · cited 2×
“The approach to evaluation presented in the previous PLoS Medicine series rests on a set of assumptions that philosophers of science call ‘‘positivist’’ [9]: that there is an external reality that can be objectively measured; that phenomena such as ‘‘project goals’’, ‘‘outcomes’’, and ‘‘formative feedback’’ can be pre…”
BackgroundMethodDevelopmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
Patton, M. Q. (2010) · Cited by 1,740
Knowledge for Theory and Practice
Van De Ven, A. H., & Johnson, P. E. (2006) · Academy of Management Review · doi:10.5465/amr.2006.22527385 · Cited by 1,663
Method and Measurement in Sociology.
Churchill, L., & Cicourel, A. V. (1966) · American Sociological Review · doi:10.2307/2091316 · Cited by 877
Technology as equivoque: Sensemaking in new technologies.
Weick, K. E. (1990) · Cited by 532
Studying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (2002) · Qualitative Research in Information Systems · doi:10.4135/9781849209687.n4 · Cited by 515
CONTANDRIOPOULOS, D., LEMIRE, M., DENIS, J.-L., & TREMBLAY, É. (2010) · Milbank Quarterly · doi:10.1111/j.1468-0009.2010.00608.x · Cited by 445 · cited 1×
“Some eHealth initiatives will lend themselves to scientific evaluation based mainly or even entirely on positivist assumptions, but others, particularly those that are largescale, complex, politically driven, and differently framed by different stakeholders, may require evaluators to reject these assumptions and apply…”
BackgroundTheorising big IT programmes in healthcare: Strong structuration theory meets actor-network theory
Greenhalgh, T., & Stones, R. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2009.12.034 · Cited by 371 · cited 2×
“Fifth, consider the dynamic macro-level context (economic, political, demographic, technological) in which the eHealth innovation is being introduced [28].”
BackgroundMethodSelf-citationMathison, S. (2005) · Sage eBooks · doi:10.4135/9781412950558 · Cited by 352
Greenhalgh, T., Hinder, S., Stramer, K., Bratan, T., & Russell, J. (2010) · BMJ · doi:10.1136/bmj.c5814 · Cited by 343
Self-citationEvaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307 · cited 2×
“One recommended systematically addressing each part of a ‘‘chain of reasoning’’, at the centre of which was the program’s goals [6].”
BackgroundGreenhalgh, T., Stramer, K., Bratan, T., Byrne, E., Russell, J., & Potts, H. W. W. (2010) · BMJ · doi:10.1136/bmj.c3111 · Cited by 275
Self-citationRobertson, A., Cresswell, K., Takian, A., Petrakaki, D., Crowe, S., Cornford, T., Barber, N., Avery, A., Fernando, B., Jacklin, A., et al. (2010) · BMJ · doi:10.1136/bmj.c4564 · Cited by 230
Understanding policy practices: action, dialectic and deliberation in policy analysis
Wagenaar, H., & Cook, S. D. N. (2003) · Deliberative Policy Analysis · doi:10.1017/cbo9780511490934.007 · Cited by 177
Pentland, B. T., Goodman, P. S., & Sproull, L. S. (1992) · Administrative Science Quarterly · doi:10.2307/2393456 · Cited by 170
Method and Measurement in Sociology
Wilkins, L. T., & Cicourel, A. V. (1965) · The British Journal of Sociology · doi:10.2307/588568 · Cited by 159
Doomed to Failure: Narratives of Inevitability and Conspiracy in a Failed IS Project
Brown, A. D., & Jones, M. R. (1998) · Organization Studies · doi:10.1177/017084069801900104 · Cited by 150 · cited 1×
“Much has been written about why electronic health (eHealth) initiatives fail [1–4].”
BackgroundA critical approach to evaluation
Klecuń, E., & Cornford, T. (2005) · European Journal of Information Systems · doi:10.1057/palgrave.ejis.3000540 · Cited by 139 · cited 1×
“N ‘‘critical’’ approaches assume that critical questioning can generate insights about power relationships and interests and that one purpose of evaluation is to ask such questions on behalf of less powerful and potentially vulnerable groups (such as patients) [11].”
BackgroundWhy Health Care Information Systems Succeed or FailPreprint
Heeks, R., Mundy, D., & Salazar, A. J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3540062 · Cited by 136
Kushner, S. (2000) · SAGE Publications Ltd eBooks · doi:10.4135/9781849209557 · Cited by 112
`Back to the Rough Ground!' Beyond Theory to Practice in Evaluation
Schwandt, T. A. (2003) · Evaluation · doi:10.1177/13563890030093008 · Cited by 104 · cited 1×
“—consciously thinking about issues such as values, perspectives, relationships, and trust. These traditions reject the assumption that a rigorous evaluation can be exclusively scientific. Rather, they hold that as well as the scientific agenda of factors, variables, and causal relationships, the evaluation must also e…”
BackgroundGreenhalgh, T., Stramer, K., Bratan, T., Byrne, E., Russell, J., Hinder, S., & POTTS, H. W. W. (2010) · UCL Discovery (University College London) · Cited by 101
Self-citationEvaluating eHealth: How to Make Evaluation More Methodologically Robust
Lilford, R. J., Foster, J., & Pringle, M. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000186 · Cited by 79 · cited 1×
“has published three papers offering a “robust” and “scientific” approach to eHealth evaluation”
BackgroundManagement Implications in Information Systems Research: The Untold Story
Portland State University, Ramiller, N., Pentland, B., & Michigan State University (2009) · Journal of the Association for Information Systems · doi:10.17705/1jais.00202 · Cited by 77 · cited 2×
“Expressing findings as statistical relationships between variables may draw attention away from people taking action [20].”
BackgroundDialogue in Evaluation: A Hermeneutic Perspective
Widdershoven, G. (2001) · Evaluation · doi:10.1177/13563890122209676 · Cited by 72 · cited 1×
“Learning happens more through the processes of evaluation than from the final product of an evaluation report [31].”
BackgroundThe Biggest Computer Programme in the World Ever! How's it Going?
Brennan, S. (2007) · Journal of Information Technology · doi:10.1057/palgrave.jit.2000104 · Cited by 60 · cited 1×
“tional Health Service [23], viewed by”
House, E. R. (2006) · Evaluation · doi:10.1177/1356389006064196 · Cited by 47 · cited 1×
“eHealth initiatives often occur in a complex and fast-moving socio-political arena. The tasks of generating, authorising, and disseminating evidence on the success of these initiatives do not occur in a separate asocial and apolitical bubble. They are often produced by, and in turn feed back into, the political proces…”
BackgroundEvaluating eHealth: Undertaking Robust International Cross-Cultural eHealth Research
Bates, D. W., & Wright, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000105 · Cited by 45 · cited 2×
“Using this taxonomy, the approach endorsed by the previous PLoS Medicine series [5–7] represents a welcome shift from a bureaucratic model (in which management consultants were commissioned to produce evaluations that directly served political ends) to an autocratic model (in which academic experts use systematic meth…”
MethodIS Success and Failure—The Problem of Scale
KREPS, D., & RICHARDSON, H. (2007) · The Political Quarterly · doi:10.1111/j.1467-923x.2007.00871.x · Cited by 39 · cited 2×
“Much has been written about why electronic health (eHealth) initiatives fail [1–4].”
BackgroundGetting To Know Schools In A Democracy
Simons, H. (2013) · Routledge eBooks · doi:10.4324/9780203064528 · Cited by 34
The object of one's passion: Engagement and community in democratic evaluation
Kushner, S. (2002) · Evaluation Journal of Australasia · doi:10.1177/1035719x0200200206 · Cited by 13 · cited 1×
“Kushner observes that “The [positivist evaluation] model is elegant in its simplicity, appealing for its rationality, reasonable in asking little more than that people do what they say they will do, and efficient in its economical definition of what data count” (”
BackgroundImplementing an Electronic Medical Record System: Successes, Failures, Lessons
Scott, T., Rundall, T. G., Vogt, T., Hsu, J. T.-A., & Aarts, J. (2007) · Cited by 7
(2005) · Encyclopedia of Evaluation · doi:10.4135/9781412950558.n141 · Cited by 5
Implementing an Electronic Medical Record System: Successes, Failures, Lessons
Taylor, P. (2008) · JAMA · doi:10.1001/jama.300.13.1594 · Cited by 3 · cited 1×
“Much has been written about why electronic health (eHealth) initiatives fail [1–4].”
Background
Research works printed but not linked: 1
- Patton MQ 1997 Utilization-focused evaluation: The new century. Third edition London Sage
Cited by
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887
Self-citationSocial Media: A Review and Tutorial of Applications in Medicine and Health Care
Grajales III, F. J., Sheps, S., Ho, K., Novak-Lauscher, H., & Eysenbach, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2912 · Cited by 737
“Although literature reviews in medicine have traditionally followed positivistic epistemologies, we drew upon a different approach, the critical-interpretivist theory [18], to conduct this review.”
BackgroundHealth information technology and digital innovation for national learning health and care systems
Sheikh, A., Anderson, M., Albala, S., Casadei, B., Franklin, B. D., Richards, M., Taylor, D., Tibble, H., & Mossialos, E. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00005-4 · Cited by 389
“8 Instead, we discuss many practical aspects of HIT,”
BackgroundVirtual online consultations: advantages and limitations (VOCAL) study
Greenhalgh, T., Vijayaraghavan, S., Wherton, J., Shaw, S., Byrne, E., Campbell-Richards, D., Bhattacharya, S., Hanson, P., Ramoutar, S., Gutteridge, C., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-009388 · Cited by 333
Self-citationKaplan, B. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104239 · Cited by 303
“Instead, the long-standing emphasis on technology using positivist approaches comes at the expense of the range and depth of ethical, legal, and social issues, and contributes to triumphalist accounts of telemedicine trials even though research and evaluation results may be less positive [15,27,52].”
Metadata
- Authors
- Greenhalgh, T., & Russell, J.
- Affiliations
- Queen Mary University of London (GB)
- University College London (GB)
- Corresponding author
- Greenhalgh, Trisha
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-11-02
- Volume
- 7
- Issue
- 11
- Pages
- e1000360
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2970573
- OpenAlex
- W2064799587
- Cited by
- 273
- References
- 40
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 10 · 2023: 12 · 2022: 7 · 2021: 15 · 2020: 25 · 2019: 17 · 2018: 15 · 2017: 17 · 2016: 15 · 2015: 11 · 2014: 14 · 2013: 84 · 2012: 15
- FWCI
- 46.05
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Funders
- Medical Research Council
- Topics
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- evaluation paradigms, science assessment, evaluation principles, paradigm shift, health technology implementation
- MeSH terms
- Program Evaluation, Health Care Reform, Universal Health Insurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.