Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Open accessThe DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index.
Abstract
Trisha Greenhalgh and Jill Russell discuss the relative merits of “scientific” and “social practice” approaches to evaluation and argue that eHealth evaluation is in need of a paradigm shift.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from a review of eHealth innovations [12] indicates that centralised diffusion systems such as those considered here were more effective in disseminating authoritative evidence and preserving the ‘fidelity’ [20] of interventions, but less successful in terms of adopter engagement, partly because of strict limits on the extent to which implementation could be modified to address the”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Too often the focus is limited to defining an appropriate technical package – reporting formats, equipment, software, training, etc. – which long experience in OECD countries suggests is a necessary but by no means sufficient requirement [10]. Failure may just as easily result from institutional or social constraints, and stakeholder analysis may be as important as traditional information systems”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While some implementation frameworks were designed around a rigid (and apparently systematic) logic model [51], we have previously argued that such an approach is counterproductive because eHealth technologies are typically introduced into a complex system in a turbulent and contested policy context [52].”
From Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies (2017) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, choice of appropriate methods for such evaluations remains a contested issue [29,30].”
The cited source says what the claim attributes to it.
In the citing paper
“Rather, in this work we adopted the interpretive position that trades off notions of sampling, significance and generalizability in favor of depth of enquiry, contextualization, abstraction, dialogue and critical reflection [30,60,70].”
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 1×
“, people perceive issues in different ways and assign different values and significance to facts)—hence, reality is never objectively or unproblematically knowable—and that the identity and values of the researcher are inevitably implicated in the research process [10].”
BackgroundMaking Social Science Matter: Why Social Inquiry Fails and How it Can Succeed Again
Flyvbjerg, B. (2001) · doi:10.1017/cbo9780511810503 · Cited by 4,681
Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory
DeSanctis, G., & Poole, M. S. (1994) · Organization Science · doi:10.1287/orsc.5.2.121 · Cited by 3,879 · cited 1×
“Controlled, feature-at-a-time comparisons are vulnerable to repeated decomposition: there are features within features, contingencies within contingencies, and tasks within tasks [25].”
BackgroundStudying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (1991) · Information Systems Research · doi:10.1287/isre.2.1.1 · Cited by 3,349 · cited 2×
“The approach to evaluation presented in the previous PLoS Medicine series rests on a set of assumptions that philosophers of science call ‘‘positivist’’ [9]: that there is an external reality that can be objectively measured; that phenomena such as ‘‘project goals’’, ‘‘outcomes’’, and ‘‘formative feedback’’ can be pre…”
BackgroundMethodDevelopmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
Patton, M. Q. (2010) · Cited by 1,740
Knowledge for Theory and Practice
Van De Ven, A. H., & Johnson, P. E. (2006) · Academy of Management Review · doi:10.5465/amr.2006.22527385 · Cited by 1,663
Method and Measurement in Sociology.
Churchill, L., & Cicourel, A. V. (1966) · American Sociological Review · doi:10.2307/2091316 · Cited by 877
Technology as equivoque: Sensemaking in new technologies.
Weick, K. E. (1990) · Cited by 532
Studying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (2002) · Qualitative Research in Information Systems · doi:10.4135/9781849209687.n4 · Cited by 515
CONTANDRIOPOULOS, D., LEMIRE, M., DENIS, J.-L., & TREMBLAY, É. (2010) · Milbank Quarterly · doi:10.1111/j.1468-0009.2010.00608.x · Cited by 445 · cited 1×
“Some eHealth initiatives will lend themselves to scientific evaluation based mainly or even entirely on positivist assumptions, but others, particularly those that are largescale, complex, politically driven, and differently framed by different stakeholders, may require evaluators to reject these assumptions and apply…”
Background
Research works printed but not linked: 1
- Patton MQ 1997 Utilization-focused evaluation: The new century. Third edition London Sage
Cited by
Items 201–273 of 273
Kpobi, L. N. A. (2015) · SUNScholar (Stellenbosch University) · Cited by 1
Phạm, Q. (2019) · TSpace (University of Toronto) · Cited by 3
Wright, J., & Cave, B. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0004 · Cited by 1
(2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.002.0006 · Cited by 1
Greenhalgh, T., Darbyshire, J. L., Lee, C., Ladds, E. C., & Ceolta‐Smith, J. (2023) · Research Square · doi:10.21203/rs.3.rs-3718323/v1 · Cited by 1
Self-citationEpidemiological approach and design
Ricciardi, W., & Boccia, S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0010
Learning from the implementation of inter-organisational web-based care planning and coordination
Walker, R., Blacker, V., Pandita, L., Close, J., Mason, W., & Watson, J. (2013) · Australian Journal of Primary Health · doi:10.1071/py13023
Sharma, U. (2011) · Brunel University Research Archive (BURA) (Brunel University London) · Cited by 1
Developing healthy public policy
Nutbeam, D. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780198800125.003.0027
Engaging Patients and Lowering Costs: Technology to the Rescue
Zordich, T., & Menichetti, J. (2016) · Advances in medical diagnosis, treatment, and care (AMDTC) book series · doi:10.4018/978-1-5225-0663-8.ch005
Effective public health action
Jones, C. S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0060
Improving your professional practice
Grainger, C. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0050
Inference, causality, and interpretation
Lang, I. (2020) · Oxford University Press eBooks · doi:10.1093/med/9780198800125.003.0012
Minhas, R., Feder, G. S., & Griffiths, C. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0038
Richards, R. G. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0036
Diment, K. (2015) · Research Online (University of Wollongong)
Georgiou, A. (2014) · Advances in healthcare information systems and administration book series · doi:10.4018/978-1-4666-6320-6.ch003
Evaluating health care technologies
Milne, R., & Stevens, A. J. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0040 · Cited by 2
Rashbass, J., & Newton, J. N. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0016
Janssen, R., Hettinga, M., Prins, H., Visser, S., Menko, R., Krediet, I., Haaker, T., & Bodenstaff, L. (2013)
Assessing Electronic Health Records: Are Basic Assumptions in “Health Technology Assessment” Useful?
Ekeland, A. G. (2016) · International Conference on eHealth, Telemedicine, and Social Medicine
Engaging Patients and Lowering Costs: Technology to the Rescue
Zordich, T., & Menichetti, J. (2017) · IGI Global eBooks · doi:10.4018/978-1-5225-0978-3.ch042
Adaptive behavior in complex healthcare interventions: assessment using computer simulation
Chiêm, J., Durme, T. V., Vandendorpe, F., Schmitz, O., Speybroeck, N., Cès, S., & Macq, J. (2012)
Team-based electronic communication in the care of patients with complex conditions
Voruganti, R. T. (2017) · TSpace (University of Toronto)
Nabukenya, J., & Ashaba, J. (2020) · Research Square · doi:10.21203/rs.2.12339/v2 · Cited by 4
Influencing international policy
Lang, T., & Caraher, M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0031 · Cited by 1
Lawrence, D. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0034
Saunders, P. J., Kibble, A. J., & Burls, A. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0015
Jessop, E. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0046
Health, sustainability, and climate change
Pencheon, D. C., Roschnik, S., & Cosford, P. A. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0058 · Cited by 2
Media advocacy for policy influence
Chapman, S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0030 · Cited by 1
Stewart, A., & Burton, H. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0023
Fruchtman, C. S., Mbuyita, S., Mwanyika‐Sando, M., Braun, M., de Savigny, D., & Muñoz, D. C. (2020) · doi:10.21203/rs.3.rs-55573/v1
“In the early days of digital health, this was a common shortcoming where many digital health solutions were developed in isolation and focussed on a single challenge (25, 28, 29).”
BackgroundJessop, E. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0045
Rice, T., & Lang, I. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0037
Greaves, F., & Guest, C. S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0059
Gray, C. S., Gravesande, J., Hans, P. K., Nie, J. X., Sharpe, S., Loganathan, M., Lyons, R., & Cott, C. A. (2018) · doi:10.2196/preprints.11950
Finding and appraising evidence
Brice, A., Burls, A., & Hill, A. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0013
Brambleby, P. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0006
Steel, N., Melzer, D. R., & Lang, I. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0042
Health care process and patient experience
Delnoij, D. M. J. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0039
Protecting and promoting health in the workplace
Aw, T., Whitaker, S., & Harrington, M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0019
Gogarty, M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0055
Translating goals, indicators, and targets into public health action
Jenkin, R. A., Jorm, C. M., & Frommer, M. S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0029
Pearson, V. A. H. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0053 · Cited by 1
Gray, J. A. M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0052
Gostin, L. O. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0033
Public health practice in primary care
Gillam, S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0025
Translating policy into indicators and targets
Battersby, J. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0028
Maryon‐Davis, A. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0047
Danaher, B. G., Crowley, R. N., Brendryen, H., Seeley, J. R., Tyler, M. S., & Severson, H. H. (2018) · doi:10.2196/preprints.10882
Consultancy in a national strategy
Guest, C. S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0049
Gray, J. A. M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0051
Programme planning and project management
Fien, J. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0054
Kumaran, K., & Lang, I. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0011
Georgiou, A. (2015) · IGI Global eBooks · doi:10.4018/978-1-4666-8756-1.ch004
(2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.002.0013
(2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.002.0005
(2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.002.0004
(2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.002.0008
Foreword to the second edition
(2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.002.0007
(2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.002.0012
Husain, L., Greenhalgh, T., Hughes, G., Finlay, T. M. D., & Wherton, J. P. (2022) · doi:10.2196/preprints.42358
Self-citationMussi, C. C., Luz, R., Damázio, D. D. R., Parma, G. C., & Porto, B. S. D. S. (2023) · CONTECSI - International Conference on Information Systems and Technology Management · doi:10.5748/19contecsi/pse/hea/7066
PPH 4.0: a privacy-preserving health 4.0 framework with machine learning and cellular automata
Mitra, A., & Pal, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-24028-7.00002-7 · Cited by 3
Berntsen, G., Knarvik, U., Rumpsfeld, M., & Gullslett, M. K. (2025) · doi:10.2196/preprints.74385
Graever, L., Mafra, P. C., Figueira, V. K., Miler, V. N., Sobreiro, J. D. S. L., da Silva, G. P. D. C., Issa, A. F. C., Savassi, L. C. M., Dias, M. B., Melo, M. M., et al. (2025) · JMIR Cardio · doi:10.2196/64438 · Cited by 4
“Other reviews and editorials by Robert et al [41], Greenhalgh et al [42,43], and Greenhalgh and Russell [85] refer to some hardships that we also found in our study.”
Evaluating for learning and sustainability (ELS) framework: a realist synthesis
Bird, M., MacPhee, M., Shaw, J. C., Wodchis, W. P., Jeffs, L. P., Austin, T., Bruno, F., Panesar, B., Côté-Boileau, É., Reid, R. J., et al. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12743-4 · Cited by 3
“…inclusion of staff input (C) there is an increased likelihood that evaluation data will be used in decision-making (O) due to staff’s integral role in the evaluation process, their sense of ownership, and the increased value (M) they attribute to evaluation [14, 18, 26, 39, 42, 45, 51, 56–66].”
Gu, Y., Humphrey, G., Warren, J., & Wilson, M. (2014) · Journal of the Association for Information Systems
Bird, M., Côté-Boileau, É., Wodchis, W. P., Jeffs, L. P., MacPhee, M., Shaw, J. C., Austin, T., Bruno, F., Bhalla, M., & Gray, C. S. (2024) · Figshare · doi:10.6084/m9.figshare.26623669
Bird, M., Côté-Boileau, É., Wodchis, W. P., Jeffs, L. P., MacPhee, M., Shaw, J. C., Austin, T., Bruno, F., Bhalla, M., & Gray, C. S. (2024) · Figshare · doi:10.6084/m9.figshare.24264759
Bird, M., Côté-Boileau, É., Wodchis, W. P., Jeffs, L. P., MacPhee, M., Shaw, J. C., Austin, T., Bruno, F., Bhalla, M., & Gray, C. S. (2023) · Figshare · doi:10.6084/m9.figshare.24264684
Metadata
- Authors
- Greenhalgh, T., & Russell, J.
- Affiliations
- Queen Mary University of London (GB)
- University College London (GB)
- Corresponding author
- Greenhalgh, Trisha
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-11-02
- Volume
- 7
- Issue
- 11
- Pages
- e1000360
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2970573
- OpenAlex
- W2064799587
- Cited by
- 273
- References
- 40
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 10 · 2023: 12 · 2022: 7 · 2021: 15 · 2020: 25 · 2019: 17 · 2018: 15 · 2017: 17 · 2016: 15 · 2015: 11 · 2014: 14 · 2013: 84 · 2012: 15
- FWCI
- 46.05
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Funders
- Medical Research Council
- Topics
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- evaluation paradigms, science assessment, evaluation principles, paradigm shift, health technology implementation
- MeSH terms
- Program Evaluation, Health Care Reform, Universal Health Insurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.