Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Open accessThe DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index.
Abstract
Trisha Greenhalgh and Jill Russell discuss the relative merits of “scientific” and “social practice” approaches to evaluation and argue that eHealth evaluation is in need of a paradigm shift.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from a review of eHealth innovations [12] indicates that centralised diffusion systems such as those considered here were more effective in disseminating authoritative evidence and preserving the ‘fidelity’ [20] of interventions, but less successful in terms of adopter engagement, partly because of strict limits on the extent to which implementation could be modified to address the”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Too often the focus is limited to defining an appropriate technical package – reporting formats, equipment, software, training, etc. – which long experience in OECD countries suggests is a necessary but by no means sufficient requirement [10]. Failure may just as easily result from institutional or social constraints, and stakeholder analysis may be as important as traditional information systems”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While some implementation frameworks were designed around a rigid (and apparently systematic) logic model [51], we have previously argued that such an approach is counterproductive because eHealth technologies are typically introduced into a complex system in a turbulent and contested policy context [52].”
From Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies (2017) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, choice of appropriate methods for such evaluations remains a contested issue [29,30].”
The cited source says what the claim attributes to it.
In the citing paper
“Rather, in this work we adopted the interpretive position that trades off notions of sampling, significance and generalizability in favor of depth of enquiry, contextualization, abstraction, dialogue and critical reflection [30,60,70].”
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 1×
“, people perceive issues in different ways and assign different values and significance to facts)—hence, reality is never objectively or unproblematically knowable—and that the identity and values of the researcher are inevitably implicated in the research process [10].”
BackgroundMaking Social Science Matter: Why Social Inquiry Fails and How it Can Succeed Again
Flyvbjerg, B. (2001) · doi:10.1017/cbo9780511810503 · Cited by 4,681
Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory
DeSanctis, G., & Poole, M. S. (1994) · Organization Science · doi:10.1287/orsc.5.2.121 · Cited by 3,879 · cited 1×
“Controlled, feature-at-a-time comparisons are vulnerable to repeated decomposition: there are features within features, contingencies within contingencies, and tasks within tasks [25].”
BackgroundStudying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (1991) · Information Systems Research · doi:10.1287/isre.2.1.1 · Cited by 3,349 · cited 2×
“The approach to evaluation presented in the previous PLoS Medicine series rests on a set of assumptions that philosophers of science call ‘‘positivist’’ [9]: that there is an external reality that can be objectively measured; that phenomena such as ‘‘project goals’’, ‘‘outcomes’’, and ‘‘formative feedback’’ can be pre…”
BackgroundMethodDevelopmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
Patton, M. Q. (2010) · Cited by 1,740
Knowledge for Theory and Practice
Van De Ven, A. H., & Johnson, P. E. (2006) · Academy of Management Review · doi:10.5465/amr.2006.22527385 · Cited by 1,663
Method and Measurement in Sociology.
Churchill, L., & Cicourel, A. V. (1966) · American Sociological Review · doi:10.2307/2091316 · Cited by 877
Technology as equivoque: Sensemaking in new technologies.
Weick, K. E. (1990) · Cited by 532
Studying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (2002) · Qualitative Research in Information Systems · doi:10.4135/9781849209687.n4 · Cited by 515
CONTANDRIOPOULOS, D., LEMIRE, M., DENIS, J.-L., & TREMBLAY, É. (2010) · Milbank Quarterly · doi:10.1111/j.1468-0009.2010.00608.x · Cited by 445 · cited 1×
“Some eHealth initiatives will lend themselves to scientific evaluation based mainly or even entirely on positivist assumptions, but others, particularly those that are largescale, complex, politically driven, and differently framed by different stakeholders, may require evaluators to reject these assumptions and apply…”
Background
Research works printed but not linked: 1
- MQ Patton, 1997, Utilization-focused evaluation: The new century. Third edition
Cited by
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887
Self-citationSocial Media: A Review and Tutorial of Applications in Medicine and Health Care
Grajales III, F. J., Sheps, S., Ho, K., Novak-Lauscher, H., & Eysenbach, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2912 · Cited by 737
“Although literature reviews in medicine have traditionally followed positivistic epistemologies, we drew upon a different approach, the critical-interpretivist theory [18], to conduct this review.”
BackgroundHealth information technology and digital innovation for national learning health and care systems
Sheikh, A., Anderson, M., Albala, S., Casadei, B., Franklin, B. D., Richards, M., Taylor, D., Tibble, H., & Mossialos, E. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00005-4 · Cited by 389
“8 Instead, we discuss many practical aspects of HIT,”
BackgroundVirtual online consultations: advantages and limitations (VOCAL) study
Greenhalgh, T., Vijayaraghavan, S., Wherton, J., Shaw, S., Byrne, E., Campbell-Richards, D., Bhattacharya, S., Hanson, P., Ramoutar, S., Gutteridge, C., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-009388 · Cited by 333
Self-citationKaplan, B. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104239 · Cited by 303
“Instead, the long-standing emphasis on technology using positivist approaches comes at the expense of the range and depth of ethical, legal, and social issues, and contributes to triumphalist accounts of telemedicine trials even though research and evaluation results may be less positive [15,27,52].”
Metadata
- Authors
- Greenhalgh, T., & Russell, J.
- Affiliations
- Queen Mary University of London (GB)
- University College London (GB)
- Corresponding author
- Greenhalgh, Trisha
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-11-02
- Volume
- 7
- Issue
- 11
- Pages
- e1000360
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2970573
- OpenAlex
- W2064799587
- Cited by
- 273
- References
- 40
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 10 · 2023: 12 · 2022: 7 · 2021: 15 · 2020: 25 · 2019: 17 · 2018: 15 · 2017: 17 · 2016: 15 · 2015: 11 · 2014: 14 · 2013: 84 · 2012: 15
- FWCI
- 46.05
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Funders
- Medical Research Council
- Topics
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- evaluation paradigms, science assessment, evaluation principles, paradigm shift, health technology implementation
- MeSH terms
- Program Evaluation, Health Care Reform, Universal Health Insurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles. https://app.cochat.ai/papers/10.1371/journal.pmed.1000360
- The paper
- Greenhalgh, T., & Russell, J. (2010). Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles. PLoS Medicine. https://doi.org/10.1371/journal.pmed.1000360
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.