Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Open accessThe DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index.
Abstract
Trisha Greenhalgh and Jill Russell discuss the relative merits of “scientific” and “social practice” approaches to evaluation and argue that eHealth evaluation is in need of a paradigm shift.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from a review of eHealth innovations [12] indicates that centralised diffusion systems such as those considered here were more effective in disseminating authoritative evidence and preserving the ‘fidelity’ [20] of interventions, but less successful in terms of adopter engagement, partly because of strict limits on the extent to which implementation could be modified to address the”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Too often the focus is limited to defining an appropriate technical package – reporting formats, equipment, software, training, etc. – which long experience in OECD countries suggests is a necessary but by no means sufficient requirement [10]. Failure may just as easily result from institutional or social constraints, and stakeholder analysis may be as important as traditional information systems”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While some implementation frameworks were designed around a rigid (and apparently systematic) logic model [51], we have previously argued that such an approach is counterproductive because eHealth technologies are typically introduced into a complex system in a turbulent and contested policy context [52].”
From Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies (2017) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, choice of appropriate methods for such evaluations remains a contested issue [29,30].”
The cited source says what the claim attributes to it.
In the citing paper
“Rather, in this work we adopted the interpretive position that trades off notions of sampling, significance and generalizability in favor of depth of enquiry, contextualization, abstraction, dialogue and critical reflection [30,60,70].”
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 1×
“, people perceive issues in different ways and assign different values and significance to facts)—hence, reality is never objectively or unproblematically knowable—and that the identity and values of the researcher are inevitably implicated in the research process [10].”
BackgroundMaking Social Science Matter: Why Social Inquiry Fails and How it Can Succeed Again
Flyvbjerg, B. (2001) · doi:10.1017/cbo9780511810503 · Cited by 4,681
Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory
DeSanctis, G., & Poole, M. S. (1994) · Organization Science · doi:10.1287/orsc.5.2.121 · Cited by 3,879 · cited 1×
“Controlled, feature-at-a-time comparisons are vulnerable to repeated decomposition: there are features within features, contingencies within contingencies, and tasks within tasks [25].”
BackgroundStudying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (1991) · Information Systems Research · doi:10.1287/isre.2.1.1 · Cited by 3,349 · cited 2×
“The approach to evaluation presented in the previous PLoS Medicine series rests on a set of assumptions that philosophers of science call ‘‘positivist’’ [9]: that there is an external reality that can be objectively measured; that phenomena such as ‘‘project goals’’, ‘‘outcomes’’, and ‘‘formative feedback’’ can be pre…”
BackgroundMethodDevelopmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
Patton, M. Q. (2010) · Cited by 1,740
Knowledge for Theory and Practice
Van De Ven, A. H., & Johnson, P. E. (2006) · Academy of Management Review · doi:10.5465/amr.2006.22527385 · Cited by 1,663
Method and Measurement in Sociology.
Churchill, L., & Cicourel, A. V. (1966) · American Sociological Review · doi:10.2307/2091316 · Cited by 877
Technology as equivoque: Sensemaking in new technologies.
Weick, K. E. (1990) · Cited by 532
Studying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (2002) · Qualitative Research in Information Systems · doi:10.4135/9781849209687.n4 · Cited by 515
CONTANDRIOPOULOS, D., LEMIRE, M., DENIS, J.-L., & TREMBLAY, É. (2010) · Milbank Quarterly · doi:10.1111/j.1468-0009.2010.00608.x · Cited by 445 · cited 1×
“Some eHealth initiatives will lend themselves to scientific evaluation based mainly or even entirely on positivist assumptions, but others, particularly those that are largescale, complex, politically driven, and differently framed by different stakeholders, may require evaluators to reject these assumptions and apply…”
Background
Research works printed but not linked: 1
- Patton MQ 1997 Utilization-focused evaluation: The new century. Third edition London Sage
Cited by
Items 101–200 of 273
Scott, P. (2015) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v22i1.139 · Cited by 16
“2 The digital maturity index (page 30), 2 a ‘census of digital progress’, seems to be the consider critical-interpretive social perspectives 42 to understand the human factors in technology deployment and should seek a theoretically informed insight into the mechanisms of change.”
Bonnici, T., Gerry, S., Wong, D., Knight, J., & Watkinson, P. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0257-8 · Cited by 16
“[38] have highlighted that the effectiveness of a system cannot be divorced from the organisational and cultural setting in which it is implemented.”
BackgroundCoorey, G., Peiris, D., Scaria, A., Mulley, J., Neubeck, L., Hafiz, N., & Redfern, J. (2021) · Journal of Medical Internet Research · doi:10.2196/25333 · Cited by 17
“A more recent framework calls the desirability of the technology to the user its value proposition, and more complex interventions may be less likely to reach this threshold for adoption [43].”
BackgroundSalvemini, A. V., Piza, E. L., Carter, J. G., Grommon, E., & Merritt, N. (2015) · Evaluation Review · doi:10.1177/0193841x15583404 · Cited by 14
“For example, Greenhalgh and Russell (2010) argue that rigid experimentation does not reflect basic realities of eHealth programs in the medical field, which often have multiple and sometimes competing goals and that program outcomes typically ‘‘erode and change over time and across contexts’’ (p.…”
BackgroundNetwork-based approaches for evaluating ambient assisted living (AAL) technologies
Gomersall, T., Nygård, L., Mihailidis, A., Sixsmith, A. J., Hwang, A. S., Hedman, A., & Astell, A. J. (2017) · Evaluation · doi:10.1177/1356389017697615 · Cited by 15
Patient and Family Engagement Approaches for Digital Health Initiatives: Protocol for a Case Study
Shen, N., Jankowicz, D., & Strudwick, G. (2021) · JMIR Research Protocols · doi:10.2196/24274 · Cited by 16
“…failed initiatives suggest that the active involvement of patients in the various aspects of developing and implementing digital health initiatives (or patient engagement) is critical to its success, especially those targeted for use specifically by patients and family members themselves [7-9].”
Fernando, M., Abell, B., McPhail, S., Tyack, Z. F., Tariq, A., & Naicker, S. (2024) · JMIR Medical Informatics · doi:10.2196/60402 · Cited by 20
“Contextually informed implementation strategies tailored to influence clinician behavior could have a greater influence on CDSS adoption than technological design and content features [15,24-26”
Morrison, C. P., Jones, M., Jones, R., & Vuylsteke, A. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-103 · Cited by 15
“The method of data analysis followed the principles articulated in [15,16] for interpretive evaluations of clinical information systems using ethnographic methods.”
MethodCassidy-Seyoum, S. A., Chheng, K., Chanpheakdey, P., Meershoek, A., Hsiang, M. S., von Seidlein, L., Tripura, R., Adhikari, B., Ley, B., Price, R. N., et al. (2024) · PLOS Global Public Health · doi:10.1371/journal.pgph.0003476 · Cited by 12
“The applied framework mainly utilizes Proctor et al .’s ‘ implementation outcomes’ (acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, and sustainability) [35] and also includes expectations of the intervention from Greenhalgh and Russel’s framework [36].”
Is there and do we need evidence on eHealth interventions?
Al-Shorbaji, N. (2013) · IRBM · doi:10.1016/j.irbm.2012.11.001 · Cited by 17
Complexity science education for clinical nurse researchers
Bird, M., & Strachan, P. H. (2019) · Journal of Professional Nursing · doi:10.1016/j.profnurs.2019.07.007 · Cited by 14
MacLure, K., Stewart, D. C., & Strath, A. (2013) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2013-000337 · Cited by 13
Fruchtman, C. S., Mbuyita, S., Mwanyika‐Sando, M., Braun, M., de Savigny, D., & Muñoz, D. C. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06285-8 · Cited by 13
“In the early days of digital health, this was a common shortcoming where many digital health solutions were developed in isolation and focussed on a single challenge [30, 33, 34].”
BackgroundAn analysis of electronic document management in oncology care
Poulter, T., Gannon, B., & Bath, P. A. (2012) · Health Informatics Journal · doi:10.1177/1460458211435514 · Cited by 12
Students’ perceptions on their use of an EHR: pilot questionnaire study
Ellis, B., Quayle, S., Bailey, I., Tishkovskaya, S., Spencer, J., & Richardson, R. (2020) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100163 · Cited by 11
Evaluating clinical decision support software (CDSS): challenges for robust evidence generation
Laka, M., Carter, D., & Merlin, T. (2024) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462324000059 · Cited by 15
“The relationship between digital health systems and their environment is usually non-linear (25).”
BackgroundWears, R. L. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003889 · Cited by 10
Evaluating Health Information Systems Using Ontologies
Eivazzadeh, S., Anderberg, P., Larsson, T. C., Fricker, S. A., & Berglund, J. S. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5185 · Cited by 13
“In one aspect at least, the evaluation of health information systems matches well with their implementation: they both fail very often [1,2,3].”
Cadre stratégique pour soutenir l’évaluation des projets complexes et innovants en santé numérique
Alami, H., Fortin, J., Gagnon, M., Lamothe, L., Ghandour, E. K., Ahmed, M. A. A., & Roy, D. A. (2020) · Santé Publique · doi:10.3917/spub.202.0221 · Cited by 15
Phạm, Q., Shaw, J., Morita, P. P., Seto, E., Stinson, J., & Cafazzo, J. A. (2019) · Journal of Medical Internet Research · doi:10.2196/14849 · Cited by 12
“The emergence of methodological slack as a distinct system antecedent for digital health research innovation is significant in its recognition of the tangible constraints imposed by positivist research paradigms and traditional models of scientific inquiry [58].”
BackgroundGDPR and Systems for Health Behavior Change: A Systematic Review
Agyei, E. E. Y. F., & B, H. O.-K. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45712-9_18 · Cited by 14
Elston, J. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0056 · Cited by 9
Baysari, M. T., Dort, B. A. V., Stanceski, K., Hargreaves, A., Zheng, W. Y., Moran, M., Day, R. O., Li, L., Westbrook, J. I., & Hilmer, S. N. (2022) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzad004 · Cited by 13
“Technologies, including CDS, are typically embedded in complex, fast-paced sociotechnical systems, the result being that the outcomes of evaluation are likely to be context-dependent and unstable and may not be replicable [16, 22, 23].”
BackgroundMethodChallenges in evaluating telehealth through RCT-the problem of randomization
Urvashi, S., Reed, J. E., Cathal, D., & Bell, D. (2012) · Studies in health technology and informatics · doi:10.3233/978-1-61499-101-4-323 · Cited by 10
“methods that recognize evaluation strategy as social-practice, and methods that enable consideration of contextual issues [12,13,16,18-21] which may offer greater value findings to those responsible for commissioning and delivering care.”
BackgroundFrom benefits idealisation to value optimisation: application in the digital health context
Smith, N., Burton‐Jones, A., & Sullivan, C. M. (2020) · Australian Health Review · doi:10.1071/ah19255 · Cited by 10
Validity and Reliability of the eHealth Analysis and Steering Instrument
Henkemans, O. B., Dusseldorp, E., Keijsers, J. F. E. M., Kessens, J. M., Neerincx, M. A., & Otten, W. (2013) · Medicine 2 0 · doi:10.2196/med20.2571 · Cited by 7
“Greenhalgh and Russell [5] point out that “assumptions, methods, and study designs of experimental science, whilst useful in many contexts, may be ill-suited to the particular challenges of evaluating eHealth programs” (p.”
BackgroundInfluentialConte, K. P., Marks, L., Loblay, V., Grøn, S., Green, A. S., Innes‐Hughes, C., Milat, A., Persson, L., Williams, M., Thackway, S. V., et al. (2020) · BMC Public Health · doi:10.1186/s12889-020-08644-2 · Cited by 8
“In clinical contexts, IT systems have a long history of design, use, and often, abandonment [5, 6].”
BackgroundMixed Methods: A Paradigm for Holistic Evaluation of Health IT.
Scott, P. (2016) · PubMed · Cited by 9
Co-design with affect stories and applied ethics for health technologies
Davat, A., Martin-Juchat, F., & Ménissier, T. (2024) · Frontiers in Communication · doi:10.3389/fcomm.2024.1327711 · Cited by 7
Expanding the boundaries of clinical informatics for interdisciplinary systems research
King, N. E. (2014) · Health Systems · doi:10.1057/hs.2013.18 · Cited by 7
A Flexible Toolkit for Evaluating Person-Centred Digital Health and Wellness at Scale
Lennon, M., Bouamrane, M.-M., Grieve, E., O’Donnell, C. A., O’Connor, S., Agbakoba, R., Devlin, A. M., Barry, S., Bikker, A., Finch, T. L., et al. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-41652-6_11 · Cited by 9
Waterson, P. E. (2019) · Applied Ergonomics · doi:10.1016/j.apergo.2019.103011 · Cited by 9
Huynh, T., Kroh, J., & Schultz, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0293550 · Cited by 8
Factors Influencing Procurement of Digital Health Care: A Case Study in Dutch District Nursing
Holterman, S., Hettinga, M., Buskens, E., & Lahr, M. M. H. (2021) · International Journal of Health Policy and Management · doi:10.34172/ijhpm.2021.115 · Cited by 12
“57 To have digital health evaluated towards implementation, Mathews et al provide a useful digital health scorecard with technological, clinical, usability and costs components including an assessment whether stakeholders’ requirements are satisfied.”
Bolton, P. A., & Burkle, F. M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0021 · Cited by 6
Gatekeeping's Influence on Equitable Evaluation Practice
Moore, T. R., Valmadrid, L. C., Baragwanath, R., Haack, N., & Bakken, L. L. (2023) · American Journal of Evaluation · doi:10.1177/10982140231154503 · Cited by 6
“Traditionally, evaluation has been driven by the postpositivist paradigm that places empirical method and rigor over a concern for the population studied (Greenhalgh & Russell, 2010; Parker, 2004).”
MethodGaddi, A. V., & Capello, F. (2013) · Springer eBooks · doi:10.1007/978-88-470-5253-6_1 · Cited by 8
Vaezipour, A., Aldridge, D., Wall, K. J., Koenig, S. T., & Theodoros, D. (2019) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct.2019.00043 · Cited by 6
“These results support previous research into why so many technology programs fail in health care [33], as well as the importance of the ecosystem in which the technology will be implemented [34].”
ResultExploring variation in the performance of planned birth: A mixed method study
Coates, D., Henry, A., Chambers, G. M., Repon, P., Makris, A., Clerke, T., & Donnolley, N. R. (2021) · Midwifery · doi:10.1016/j.midw.2021.102988 · Cited by 8
Aminoff, H., Meijer, S., Arnelo, U., & Groth, K. (2021) · JMIR Formative Research · doi:10.2196/26505 · Cited by 8
Taylor, R., & Guest, C. S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0018 · Cited by 5
Rabbi, M., Philyaw-Kotov, M. L., Li, J., Li, K. T., Rothman, B., Giragosian, L., Reyes, M., Gadway, H., Cunningham, R. M., Bonar, E. E., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13545 · Cited by 8
“While it is extremely useful for clarifying the causal pathway through which an intervention is supposed to operate, logic model has on occasion been criticized for being too stringent or rigid [44].”
BackgroundCase Study as Antidote to the Literal
Kushner, S. (2015) · Emerald Group Publishing Limited eBooks · doi:10.1108/s1474-786320140000015003 · Cited by 4
Sosin, D. M., & Hopkins, R. S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0014 · Cited by 4
Walzer, S., Armbruster, C., Kunze, C., & Farin‐Glattacker, E. (2023) · Springer eBooks · doi:10.1007/978-3-658-39302-1_2 · Cited by 4
Mixed Methods: A Paradigm for Holistic Evaluation of Health IT
Scott Philip, J. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-635-4-102 · Cited by 6
“A common argument used against the mixed method paradigm, still raised [5, 29] (and implied in [30, 31]), is that qualitative and quantitative methods and data derive from contrasting philosophical worldviews (a set of beliefs and values giving rise to a particular outlook upon life and reality, typically either inter…”
BackgroundChaniaud, N., Loup‐Escande, É., & Mégalakaki, O. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-96098-2_72 · Cited by 5
Tintorer, D. L., Manresa‐Domínguez, J. M., Jiménez‐Zarco, A. I., Rodríguez-Blanco, T., Beneyto, S. F., Torán‐Monserrat, P., Mundet, X., & Saigí-Rubió, F. (2020) · BMC Family Practice · doi:10.1186/s12875-020-1081-x · Cited by 5
Understanding Innovation Deployment and Evaluation in Healthcare: The Triality Framework
Sharma, U., Barnett, J. C., & Clarke, M. (2012) · InTech eBooks · doi:10.5772/35671 · Cited by 6
“In particular, ethical issues arise due to evaluation strategies such as RCT as it governs the provision and allocation of equipment to patients (Finch et al., 2003, 2006; Heaven et al., 2006; Lankshear & Mason, 2001; Greenhalgh & Russell, 2010; McDonald et al., 2006; Ammenwerth et al., 2003).”
BackgroundJones, E. B., Swain, M. J., Patel, V., & Furukawa, M. F. (2014) · Healthcare · doi:10.1016/j.hjdsi.2013.12.015 · Cited by 6
Evaluation of eHealth Implementations in Uganda: Investigating Practices, Challenges and InsightsPreprint
Nabukenya, J., & Ashaba, J. (2019) · Research Square · doi:10.21203/rs.2.12339/v1 · Cited by 5
“Secondly, the evaluation takes place in a complex healthcare setting that involves multiple stakeholder categories (such as patients, clinicians, administrators, IT specialists, funders) on top of legislation, social, political and economic environments [47].”
BackgroundDigital health interventions in HIV projects in Kenya
Mudogo, C. M., Mulwa, A. S., & Kyalo, D. N. (2025) · South African journal of information management · doi:10.4102/sajim.v27i1.1973 · Cited by 3
Grajales, F. J., III (2012) · Open Collections · doi:10.14288/1.0072915 · Cited by 4
Janssen, R., Hettinga, M., Visser, S., Menko, R., Prins, H., Krediet, I., Haaker, T., & Bodenstaff, L. (2013) · Cited by 3
Jurkeviciute, M., Enam, A., Torres-Bonilla, J., & Eriksson, H. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01399-9 · Cited by 4
Digital Transformation in Healthcare: Are patients ready to adopt eHealth in Primary Health Care
Rachmani, E., Haikal, Mubarokah, K., Anjani, S., Handayani, S. S., Setyowati, M., Nurjanah, N., & Yanuar, A. M. F. (2021) · International Seminar on Application for Technology of Information and Communication (iSemantic) · doi:10.1109/isemantic52711.2021.9573228 · Cited by 6
Stakes, positions and logics: An institutional field analysis of cross-border health IT policy
Currie, W. L., & Seddon, J. J. M. (2021) · Journal of Information Technology · doi:10.1177/02683962211040513 · Cited by 8
“The legacy of large-scale national HIT programmes, however, has repeatedly ended in failure to deliver bene fi ts to patients and health professionals (Greenhalgh & Russell, 2010; Greenhalgh & Stones, 2010).”
BackgroundFlowers, J. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0003 · Cited by 3
Communicable disease epidemics
O’Brien, S. J. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0017 · Cited by 4
Giebel, G. D., & Giebel, G. D. (2022) · Springer eBooks · doi:10.1007/978-3-658-35691-0_3 · Cited by 3
Chiêm, J., Durme, T. V., Vandendorpe, F., Schmitz, O., Speybroeck, N., Cès, S., & Macq, J. (2013) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12101 · Cited by 7
“Indeed, qualitative studies build on perceptions and interpretations to find a contextual meaning and propose a causation mechanism [6].”
BackgroundScott-Samuel, A., Ardern, K., & Birley, M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0005 · Cited by 3
Priorities and ethics in health care
Griffiths, S. M., Martin, R., & Sinclair, D. D. A. D. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0002 · Cited by 6
Jurkeviciute, M. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0975-9 · Cited by 3
Mesoergonomics, the missing part of the macroergonomics and microergonomics approach
Panjaitan, N., Ab-Samat, H., & Ali, A. (2020) · Journal of Engineering Design and Technology · doi:10.1108/jedt-08-2020-0333 · Cited by 4
Engaging communities in participatory research and action
Minkler, M., & Chang, C. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0020 · Cited by 3
Understanding data, information, and knowledge
Tennison, B. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0007 · Cited by 4
Scoping public health problems
Bammer, G. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0001 · Cited by 3
Phippard, T. M. (2012) · Lirias · Cited by 3
How to make an app-based program work and show how it works
Janssen, R., Engel, N., Krumeich, A. J. S. M., Esmail, A., Dheda, K. U. J., Thomas, R., & Pai, N. P. (2025) · PLOS Global Public Health · doi:10.1371/journal.pgph.0005034 · Cited by 4
“Furthermore, we identify different moments in the study which illustrate the messiness and unpredictability of the technology in practice [8].”
InfluentialNewhouse, N., Ulytė, A., Marciniak-Nuqui, Z., van Dael, J., Marjanovic, S., Brennan-Olsen, S. L., & Shaw, S. E. (2025) · Health and Social Care Delivery Research · doi:10.3310/gjss1422 · Cited by 2
Litchfield, I. J., Turner, A. M., Filho, J. B. F., Lee, M. G., & Weber, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105381 · Cited by 3
Bras, A. L., Zarca, K., Mimouni, M., Durand‐Zaleski, I., & ARS Ile de France Telemedicine Group (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043031 · Cited by 3
“To analyze the difficulties encountered in the implementation of the evaluations, we used the framework provided by T Greenhalgh and J Russel and examined the baseline hypotheses about the effects that underlie the sample size calculation, the interacting variables, the study design, and the political environment [14].”
BackgroundMethodResultInfluentialBird, M., Côté-Boileau, É., Wodchis, W. P., Jeffs, L. P., MacPhee, M., Shaw, J. C., Austin, T., Bruno, F., Bhalla, M., & Gray, C. S. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02348-5 · Cited by 2
“Additionally, Greenhalgh and Russell postulate that other evaluation factors traditionally aligning with the positiv-ist paradigm, such as evaluator objectivity and distance from the study phenomenon/innovation, do not actually constitute “good research” as traditionally thought [16].”
BackgroundInfluentialBooth, N., McConnell, T., Tully, M. A., Hamill, R., & Best, P. (2024) · Mental Health and Digital Technologies · doi:10.1108/mhdt-11-2023-0002 · Cited by 4
Community patient follow-up as a part of P. vivax case management in Cambodia: a mixed methods study
Cassidy-Seyoum, S. A., Chheng, K., Chanpheakdey, P., Meershoek, A., Hsiang, M. S., von Seidlein, L., Adhikari, B., Tripura, R., Ley, B., Price, R. N., et al. (2026) · BMC Health Services Research · doi:10.1186/s12913-026-15274-8 · Cited by 3
Bozan, K., & Datta, P. M. (2016) · Healthcare delivery in the information age · doi:10.1007/978-3-319-25973-4_7 · Cited by 2
Schmitz, O., Chiêm, J., Durme, T. V., Cès, S., & Macq, J. (2014) · Presses universitaires Saint-Louis Bruxelles eBooks · doi:10.4000/books.pusl.4947 · Cited by 2
The Need for Balanced Health Policies to Avoid Path Dependent Medicine
Takian, A. (2013) · International Journal of Health Policy and Management · doi:10.15171/ijhpm.2013.02 · Cited by 2
“Moreover, we have over-emphasized evidence-based policy making, which is mostly based on randomized control trials and heavily positivist views of the world (13,14) at the price of misallocation of many resources and abandoning many priorities.”
BackgroundEvaluating health care systems
McKee, M., Khoshaba, B., & Karanikolos, M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0043 · Cited by 2
Almacen, A. M. B., & Cabaluna, A. Y. (2021) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2021.3.2.8 · Cited by 4
Friel, S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0041 · Cited by 4
Mallinson, S., Popay, J., & Williams, G. H. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0009 · Cited by 2
Trust and Provenance in Communication to eHealth Consumers
Cumming, G., French, T., Hogg, J., McKendrick, D., Gilstad, H., Molik, D. C., & Luciano, J. (2017) · Palgrave Macmillan UK eBooks · doi:10.1057/978-1-349-95173-4_11 · Cited by 5
Mansel, C. O., Sarnataro, R., & Liu, P. J. (2021) · British Journal of Healthcare Management · doi:10.12968/bjhc.2020.0064 · Cited by 1
Fernando, M., Naicker, S., Abell, B., McPhail, S., & Tyack, Z. F. (2026) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000561 · Cited by 1
Going beyond healthcare IT inter-operability in chronic disease management
King, N. E., & Jayaraman, R. (2016) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon.2016.7490527 · Cited by 1
“The synthesis of secondary sources is an accepted method in healthcare systems research [25], especially if the intention is to evaluate more than just the technology or clinical interventions, but also the social practices [24].”
BackgroundMethodPublic health in poorer countries
Banatvala, N., & Heymann, E. P. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0032 · Cited by 1
Funding and delivering health care
Dixon, A. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0035 · Cited by 1
Raffle, A. E., Barratt, A., & Gray, J. A. M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0022 · Cited by 1
Information technology and informatics
Detmer, D. E. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0008 · Cited by 1
Translating evidence to policy
Smith, L. A., An, J., & Kawachi, I. I. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0027 · Cited by 1
Inference, causality, and interpretation
Lang, I. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0012 · Cited by 1
Sim, F. M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0044 · Cited by 1
Ward, J., Grimshaw, J., & Eccles, M. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0057 · Cited by 1
Early Detection of Inpatient Deterioration Using Wearable Monitors
Bonnici, T. (2019) · doi:10.13140/rg.2.2.26843.92967 · Cited by 2
Viswanath, K. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0024 · Cited by 1
Steel, N., & Guest, C. S. (2013) · Oxford University Press eBooks · doi:10.1093/med/9780199586301.003.0048 · Cited by 2
Evaluation of Online Public Sexual Health Care in the Netherlands: A Scenario-Based User Evaluation
Roskam, R. V. (2014) · Cited by 2
Metadata
- Authors
- Greenhalgh, T., & Russell, J.
- Affiliations
- Queen Mary University of London (GB)
- University College London (GB)
- Corresponding author
- Greenhalgh, Trisha
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-11-02
- Volume
- 7
- Issue
- 11
- Pages
- e1000360
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2970573
- OpenAlex
- W2064799587
- Cited by
- 273
- References
- 40
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 10 · 2023: 12 · 2022: 7 · 2021: 15 · 2020: 25 · 2019: 17 · 2018: 15 · 2017: 17 · 2016: 15 · 2015: 11 · 2014: 14 · 2013: 84 · 2012: 15
- FWCI
- 46.05
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Funders
- Medical Research Council
- Topics
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- evaluation paradigms, science assessment, evaluation principles, paradigm shift, health technology implementation
- MeSH terms
- Program Evaluation, Health Care Reform, Universal Health Insurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.