Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Open accessThe DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index.
Abstract
Trisha Greenhalgh and Jill Russell discuss the relative merits of “scientific” and “social practice” approaches to evaluation and argue that eHealth evaluation is in need of a paradigm shift.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence from a review of eHealth innovations [12] indicates that centralised diffusion systems such as those considered here were more effective in disseminating authoritative evidence and preserving the ‘fidelity’ [20] of interventions, but less successful in terms of adopter engagement, partly because of strict limits on the extent to which implementation could be modified to address the”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Too often the focus is limited to defining an appropriate technical package – reporting formats, equipment, software, training, etc. – which long experience in OECD countries suggests is a necessary but by no means sufficient requirement [10]. Failure may just as easily result from institutional or social constraints, and stakeholder analysis may be as important as traditional information systems”
From Beyond pilotitis: taking digital health interventions to the national level in China and Uganda (2017) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While some implementation frameworks were designed around a rigid (and apparently systematic) logic model [51], we have previously argued that such an approach is counterproductive because eHealth technologies are typically introduced into a complex system in a turbulent and contested policy context [52].”
From Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies (2017) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, choice of appropriate methods for such evaluations remains a contested issue [29,30].”
The cited source says what the claim attributes to it.
In the citing paper
“Rather, in this work we adopted the interpretive position that trades off notions of sampling, significance and generalizability in favor of depth of enquiry, contextualization, abstraction, dialogue and critical reflection [30,60,70].”
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 1×
“, people perceive issues in different ways and assign different values and significance to facts)—hence, reality is never objectively or unproblematically knowable—and that the identity and values of the researcher are inevitably implicated in the research process [10].”
BackgroundMaking Social Science Matter: Why Social Inquiry Fails and How it Can Succeed Again
Flyvbjerg, B. (2001) · doi:10.1017/cbo9780511810503 · Cited by 4,681
Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory
DeSanctis, G., & Poole, M. S. (1994) · Organization Science · doi:10.1287/orsc.5.2.121 · Cited by 3,879 · cited 1×
“Controlled, feature-at-a-time comparisons are vulnerable to repeated decomposition: there are features within features, contingencies within contingencies, and tasks within tasks [25].”
BackgroundStudying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (1991) · Information Systems Research · doi:10.1287/isre.2.1.1 · Cited by 3,349 · cited 2×
“The approach to evaluation presented in the previous PLoS Medicine series rests on a set of assumptions that philosophers of science call ‘‘positivist’’ [9]: that there is an external reality that can be objectively measured; that phenomena such as ‘‘project goals’’, ‘‘outcomes’’, and ‘‘formative feedback’’ can be pre…”
BackgroundMethodDevelopmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
Patton, M. Q. (2010) · Cited by 1,740
Knowledge for Theory and Practice
Van De Ven, A. H., & Johnson, P. E. (2006) · Academy of Management Review · doi:10.5465/amr.2006.22527385 · Cited by 1,663
Method and Measurement in Sociology.
Churchill, L., & Cicourel, A. V. (1966) · American Sociological Review · doi:10.2307/2091316 · Cited by 877
Technology as equivoque: Sensemaking in new technologies.
Weick, K. E. (1990) · Cited by 532
Studying Information Technology in Organizations: Research Approaches and Assumptions
Orlikowski, W. J., & Baroudi, J. J. (2002) · Qualitative Research in Information Systems · doi:10.4135/9781849209687.n4 · Cited by 515
CONTANDRIOPOULOS, D., LEMIRE, M., DENIS, J.-L., & TREMBLAY, É. (2010) · Milbank Quarterly · doi:10.1111/j.1468-0009.2010.00608.x · Cited by 445 · cited 1×
“Some eHealth initiatives will lend themselves to scientific evaluation based mainly or even entirely on positivist assumptions, but others, particularly those that are largescale, complex, politically driven, and differently framed by different stakeholders, may require evaluators to reject these assumptions and apply…”
Background
Research works printed but not linked: 1
- Patton MQ 1997 Utilization-focused evaluation: The new century. Third edition London Sage
Cited by
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887
Self-citationSocial Media: A Review and Tutorial of Applications in Medicine and Health Care
Grajales III, F. J., Sheps, S., Ho, K., Novak-Lauscher, H., & Eysenbach, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2912 · Cited by 737
“Although literature reviews in medicine have traditionally followed positivistic epistemologies, we drew upon a different approach, the critical-interpretivist theory [18], to conduct this review.”
BackgroundHealth information technology and digital innovation for national learning health and care systems
Sheikh, A., Anderson, M., Albala, S., Casadei, B., Franklin, B. D., Richards, M., Taylor, D., Tibble, H., & Mossialos, E. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00005-4 · Cited by 389
“8 Instead, we discuss many practical aspects of HIT,”
BackgroundVirtual online consultations: advantages and limitations (VOCAL) study
Greenhalgh, T., Vijayaraghavan, S., Wherton, J., Shaw, S., Byrne, E., Campbell-Richards, D., Bhattacharya, S., Hanson, P., Ramoutar, S., Gutteridge, C., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-009388 · Cited by 333
Self-citationKaplan, B. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104239 · Cited by 303
“Instead, the long-standing emphasis on technology using positivist approaches comes at the expense of the range and depth of ethical, legal, and social issues, and contributes to triumphalist accounts of telemedicine trials even though research and evaluation results may be less positive [15,27,52].”
A systematic review of the research evidence for the benefits of teledentistry
Estai, M., Kanagasingam, Y., Tennant, M., & Bunt, S. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x16689433 · Cited by 247
Westbrook, J. I., Reckmann, M., Li, L., Runciman, W. B., Burke, R., Lo, C., Baysari, M. T., Braithwaite, J., & Day, R. O. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001164 · Cited by 216
“However, as other studies have demonstrated [40,43,44], success in achieving this outcome is dependent upon many contextual and organisational factors and multimethod studies are of great value in order to understand the mechanisms by which e-prescribing systems impact upon prescribing behaviours [12].”
ResultBeyond pilotitis: taking digital health interventions to the national level in China and Uganda
Huang, F., Blaschke, S., & Lucas, H. (2017) · Globalization and Health · doi:10.1186/s12992-017-0275-z · Cited by 182
“Evidence from a review of eHealth innovations [12] indicates that centralised diffusion systems such as those con-”
BackgroundInfluentialThe importance of values in evidence-based medicine
Kelly, M. P., Heath, I., Howick, J., & Greenhalgh, T. (2015) · BMC Medical Ethics · doi:10.1186/s12910-015-0063-3 · Cited by 209
“An alternative view is that scientific findings are meaningless unless these societal, cultural and political perspectives are taken into account – since such values-based influences necessarily frame the problem and shape the research questions and the interpretation of findings [30].”
BackgroundSelf-citationChang, L. W., Kagaayi, J., Arem, H., Nakigozi, G., Ssempijja, V., Serwadda, D., Quinn, T. C., Gray, R. H., Bollinger, R. C., & Reynolds, S. J. (2011) · AIDS and Behavior · doi:10.1007/s10461-011-9995-x · Cited by 184
eHealth for improving quality of life in breast cancer patients: A systematic review
Triberti, S., Savioni, L., Sebri, V., & Pravettoni, G. (2019) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2019.01.003 · Cited by 184
People-centred science: strengthening the practice of health policy and systems research
Sheikh, K., George, A., & Gilson, L. (2014) · Health Research Policy and Systems · doi:10.1186/1478-4505-12-19 · Cited by 183
“The dangers of HPSR that purports to support context-free and widely generalizable knowledge is increasingly evident [14], as also shown by the emerging social science-informed perspectives on paying for performance [15,16].”
BackgroundStudying technology use as social practice: the untapped potential of ethnography
Greenhalgh, T., & Swinglehurst, D. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-45 · Cited by 163
“Whereas controlled experiments produce learning in terms of quantitative, predictive statements about the relationship between predefined variables, ethnographic studies produce a different kind of learning in terms of interpretive insights about actions and events placed in context [31].”
BackgroundSelf-citationBeyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods
Pham, Q., Wiljer, D., & Cafazzo, J. A. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5720 · Cited by 162
Greenhalgh, T., Maylor, H., Shaw, S., Wherton, J., Papoutsi, C., Betton, V., Nelissen, N., Gremyr, A., Rushforth, A., Koshkouei, M., et al. (2020) · JMIR Research Protocols · doi:10.2196/16861 · Cited by 152
“This is probably because the introduction of technology-supported change in health care and social care invariably involves not only technical implementation but also the ongoing judicious management of interacting subprojects characterized by competing values, goals, stakeholder interests, and local and national poli…”
BackgroundSelf-citationBurton‐Jones, A., Akhlaghpour, S., Ayre, S. J., Barde, P., Staib, A. N., & Sullivan, C. M. (2019) · Information and Organization · doi:10.1016/j.infoandorg.2019.100255 · Cited by 198
Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review
Enam, A., Torres-Bonilla, J., & Eriksson, H. (2018) · Journal of Medical Internet Research · doi:10.2196/10971 · Cited by 151
“Evaluation of eHealth interventions is complex because of several reasons (eg, the need for multidisciplinary collaboration [8], context dependency [9], and differences in epistemological beliefs considering the interventions in clinical studies or including social aspects as well [10-14]).”
BackgroundCrossing levels in systems ergonomics: A framework to support ‘mesoergonomic’ inquiry
Karsh, B.-T., Waterson, P., & Holden, R. J. (2014) · Applied Ergonomics · doi:10.1016/j.apergo.2013.04.021 · Cited by 149
“By contrast, Greenhalgh and Russell (2010) in their discussion of the high failure rates involved in implementing eHealth programs describe macro-level factors in regard to economic or political factors, meso-level factors as the influence of specific organisational or professional group influences…”
BackgroundShaw, S., Wherton, J., Vijayaraghavan, S., Morris, J., Bhattacharya, S., Hanson, P., Campbell-Richards, D., Ramoutar, S., Collard, A., Hodkinson, I., et al. (2018) · Health Services and Delivery Research · doi:10.3310/hsdr06210 · Cited by 146
Self-citationPutting the social back into sociotechnical: Case studies of co-design in digital health
Papoutsi, C., Wherton, J., Shaw, S., Morrison, C., & Greenhalgh, T. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa197 · Cited by 126
“Our approach to theory-driven research goes beyond prescriptive and compartmentalized application of theoretical concepts; instead, we were interested in interrogating the assumptions by which programs were developed and employed our theoretical orientation in a way that surfaced the nuances of different contexts.(25,…”
MethodSelf-citationGREENHALGH, T., RUSSELL, J., ASHCROFT, R. E., & PARSONS, W. (2011) · Milbank Quarterly · doi:10.1111/j.1468-0009.2011.00642.x · Cited by 119
Self-citationO’Donnell, A., Kaner, E., Shaw, C., & Haighton, C. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0703-x · Cited by 108
Abimbola, S., Patel, B., Peiris, D., Patel, A., Harris, M., Usherwood, T., & Greenhalgh, T. (2019) · BMC Medicine · doi:10.1186/s12916-019-1463-x · Cited by 102
Self-citationGates, P., Hardie, R., Raban, M. Z., Li, L., & Westbrook, J. I. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa230 · Cited by 123
“We adapted these tools to best reflect the complexity of conducting these “realworld” intervention studies and removed the criterion requiring control groups as—while highly desirable—opportunities for obtaining control groups may be very limited for real-world studies of this type of organization-wide IT intervention…”
MethodWaterson, P. E. (2013) · Applied Ergonomics · doi:10.1016/j.apergo.2013.07.004 · Cited by 120
Greenhalgh, T., Shaw, S., Wherton, J., Hughes, G., Lynch, J., A'Court, C., Hinder, S., Fahy, N., Byrne, E., Finlayson, A., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010208 · Cited by 85
“In particular, when randomised trial designs are used to ‘control for’ the multiple organisational, social, cultural and political influences on which this embedding depends, the external validity of any effect size becomes questionable.(29) Technical descriptions and trials of ‘technology on’ versus ‘technology off’…”
BackgroundSelf-citationHusain, L., Greenhalgh, T., Hughes, G., Finlay, T. M. D., & Wherton, J. P. (2022) · Journal of Medical Internet Research · doi:10.2196/42358 · Cited by 105
“…by social scientists for oversimplifying context, removing key content (eg, unmeasured variables), overlooking historical path dependencies, and failing to explore how different variables combine and unfold over time to produce complex and sometimes unpredictable outcomes for individuals [72,73].”
Self-citationDyadic Interviews as a Tool for Qualitative Evaluation
Morgan, D. L., Eliot, S., Lowe, R. A., & Gorman, P. (2015) · American Journal of Evaluation · doi:10.1177/1098214015611244 · Cited by 138
“Unlike most other evaluations of electronic health initiatives that have used quantitative methods, we used a social practice approach (Greenhalgh & Russell, 2010) that allowed us to gather qualitative insights about how physicians framed and enacted their individual implementation experience using…”
MethodGreenhalgh, T., Morris, L., Wyatt, J. C., Thomas, G., & Gunning, K. (2013) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.01.002 · Cited by 79
“[14] T.”
BackgroundInfluentialSelf-citationHas evidence‐based medicine ever been modern? A Latour‐inspired understanding of a changing EBM
Wieringa, S., Engebretsen, E., Heggen, K. M., & Greenhalgh, T. (2017) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12752 · Cited by 98
“Thus, EBM began with explicit aspirations to modernism that still persevere today.(10) For instance, in a paper on evaluating e‐health interventions, the authors write:”
BackgroundSelf-citationTechnological Frames and Care Robots in Eldercare
Frennert, S., Aminoff, H., & Östlund, B. (2021) · International Journal of Social Robotics · doi:10.1007/s12369-020-00641-0 · Cited by 75
“care robots cannot be treated as a purely technical issue [14, 15].”
BackgroundRecommendations for implementing gamification for mental health and wellbeing
Cheng, V. W. S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.586379 · Cited by 74
“However, much like how digital health technologies operate within a wider psychosocial context (Ritterband et al., 2009; Greenhalgh and Russell, 2010), so do games operate within a system that players find engaging precisely because all components of the game (not just its individual elements) work together with the p…”
BackgroundInfluentialGreenhalgh, T., Swinglehurst, D., & Stones, R. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02390 · Cited by 71
“This iterative process drew particularly on the principles of interpretivist information systems research, especially the notion of the hermeneutic circle (constantly comparing each new item of data or theoretical insight with an emerging picture of the whole).(133,134) Using a small subsample of three or four detaile…”
MethodSelf-citationBender, J. L., Jimenez, M. C., Ferris, L. E., Katz, J. D., & Jadad, A. R. (2012) · Supportive Care in Cancer · doi:10.1007/s00520-012-1655-9 · Cited by 75
“These findings indicate a need for more exploratory research designs, as suggested by Greenhalgh and Russell [34], which take into account the complex array of personal, social, political, and ideological factors that determine the success of eHealth interventions.”
BackgroundPrice, M., & Lau, F. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-43 · Cited by 89
“There is a need to consider HISs as complex systems [9,49] and the CAMM can support the assessment by providing a time based model against which one can assess complex interventions.”
BackgroundDistinguishing features in the assessment of mHealth apps
Tarricone, R., Petracca, F., Ciani, O., & Cucciniello, M. (2021) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2021.1891883 · Cited by 62
“be performed considering the specific organizational and cultural backgrounds [32].”
BackgroundGreenhalgh, T. (2012) · Journal of Primary Health Care · doi:10.1071/hc12092 · Cited by 91
Self-citationGrafting: Balancing Control and Cultivation in Information Infrastructure Innovation
Sanner, T. A., Manda, T. D., & Nielsen, P. (2014) · Journal of the Association for Information Systems · doi:10.17705/1jais.00356 · Cited by 81
Speerin, R., Needs, C., Chua, J., Woodhouse, L. J., Nordin, M. C., McGlasson, R. A., & Briggs, A. M. (2020) · Best Practice & Research Clinical Rheumatology · doi:10.1016/j.berh.2020.101548 · Cited by 70
Digital health interventions for chronic diseases: a scoping review of evaluation frameworks
Bashi, N., Fatehi, F., Mosadeghi-Nik, M., Askari, M. S., & Karunanithi, M. (2020) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100066 · Cited by 67
Truitt, E., Thompson, R. W., Blazey-Martin, D., Nisai, D., & Salem, D. N. (2016) · Hospital Pharmacy · doi:10.1310/hpj5106-474 · Cited by 76
Patient Work and Their Contexts: Scoping Review
Yin, K., Jung, J., Coiera, E. W., Laranjo, L., BLANDFORD, A. E., Khoja, A. A., Tai, W., Phillips, D. P., & Lau, A. Y. S. (2020) · Journal of Medical Internet Research · doi:10.2196/16656 · Cited by 85
“Mesolevel contextual factors were the influences of people socially close to the patient or influences exerted by the immediate physical surroundings or social circumstances (eg, finances) of the patient [35].”
BackgroundDiabetes Care in the Digital Era: a Synoptic Overview
Fatehi, F., Menon, A., & Bird, D. (2018) · Current Diabetes Reports · doi:10.1007/s11892-018-1013-5 · Cited by 69
Berntsen, G., Strisland, F., Malm-Nicolaisen, K., Smaradottir, B. F., Fensli, R. W., & Røhne, M. (2019) · Journal of Medical Internet Research · doi:10.2196/12517 · Cited by 69
“There is much literature on promising digital solutions, but the large-scale adoption of these tools is slow [37], and there is disillusionment with large-scale electronic health (eHealth) impact [38].”
Ford, T., Edwards, V., Sharkey, S., Ukoumunne, O. C., Byford, S., Norwich, B., & Logan, S. (2012) · BMC Public Health · doi:10.1186/1471-2458-12-719 · Cited by 54
“Analysis will be guided by a realist perspective, to identify experiences as the lived ‘reality’ of participants, but we are also interested in the ways in which participants account for their experiences within the context of the trial and their own schools based experiences [58].”
BackgroundNewman, L., Bidargaddi, N., & Schrader, G. (2016) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2016.05.006 · Cited by 53
Takian, A., Petrakaki, D., Cornford, T., Sheikh, A., & Barber, N. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-105 · Cited by 51
“Under the modified design, individual researchers became deeply immersed in the research sites [30,60].”
BackgroundPhilibert, I., & Barach, P. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001598 · Cited by 52
“in clinical microsystem groups that develop norms that govern their mission, primary tasks, manifest and latent functions, and consensus on the criteria used to measure results.(33) Social norms used in the medical and nursing professions, and organisa-”
BackgroundHind, D., Parkin, J., Whitworth, V., Rex, S., Young, T. A., Hampson, L. V., Sheehan, J., Maguire, C., Cantrill, H., Scott, E., et al. (2017) · Health Technology Assessment · doi:10.3310/hta21270 · Cited by 66
“Reporting Before the main results, we provide an exploration of context, which critical interpretivists define as more than the physical environment,(178,179) factors, variables and causal relationships,(180) but as something sociorelational, which incorporates settings, ‘roles, interactions and relationships’;(178) ‘…”
BackgroundShen, N., Sequeira, L., Silver, M. P., Carter-Langford, A., Strauss, J., & Wiljer, D. (2019) · JMIR Mental Health · doi:10.2196/13306 · Cited by 67
“In addition to public education, patient engagement is essential to its success [78-80].”
Ethnographic process evaluation in primary care: explaining the complexity of implementation
Bunce, A. E., Gold, R., Davis, J. V., McMullen, C. K., Jaworski, V., Mercer, M., & Nelson, C. (2014) · BMC Health Services Research · doi:10.1186/s12913-014-0607-0 · Cited by 49
“This emphasis can lead researchers to overlook the myriad, seemingly mundane details that may affect an intervention’s ultimate success, such as: who did what when; how people felt and talked about the intervention; how relationships, hierarchies, and workflows changed; and the resistance, compromises, and workarounds…”
BackgroundGeraedts, M., Drösler, S. E., Döbler, K., Eberlein-Gonska, M., Heller, G., Kuske, S., Manser, T., Sens, B., Stausberg, J., & Schrappe, M. (2017) · Das Gesundheitswesen · doi:10.1055/s-0043-112431 · Cited by 55
“Typische Beispiele für komplexe Qualitätsverbesserungs-Initiativen auf Systemebene sind die Einführung von eHealth-Instrumenten [153] oder die qualitätsorientierte Vergütung (P4P).”
BackgroundGreenhalgh, T., Fahy, N., & Shaw, S. (2017) · International Journal of Health Policy and Management · doi:10.15171/ijhpm.2017.65 · Cited by 44
Self-citationAckerman, S., Sarkar, U., Tieu, L., Handley, M. A., Schillinger, D., Hahn, K., Hoskote, M., Gourley, G., & Lyles, C. R. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx015 · Cited by 46
Morley, J., Morton, C. E., Karpathakis, K., Taddeo, M., & Floridi, L. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3826358 · Cited by 50
Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review
Santis, K. K. D., Jahnel, T., Matthias, K., Mergenthal, L., Khayyal, H. A., & Zeeb, H. (2022) · JMIR Public Health and Surveillance · doi:10.2196/37820 · Cited by 47
Jeffries, M., Phipps, D. L., Howard, R. L., Avery, A. J., Rodgers, S., & Ashcroft, D. M. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014810 · Cited by 48
“Similarly, in previous literature, information systems have been associated with enabling managers to capture information, place local clinicians under surveillance and make their actions calculable.(39) Furthermore, an effect of such surveillance is for individuals to adapt their own behaviour to ensure they act legi…”
BackgroundGani, I., Litchfield, I. J., Shukla, D., Delanerolle, G., Cockburn, N., & Pathmanathan, A. (2025) · Journal of Medical Internet Research · doi:10.2196/62763 · Cited by 60
“Sociocultural factors including public perceptions of the technology [49].”
Information Architecture of Web-Based Interventions to Improve Health Outcomes: Systematic Review
Pugatch, J., Grenen, E., Surla, S., Schwarz, M., & Cole-Lewis, H. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.7867 · Cited by 34
“Most digital health intervention research describes the studies but fails to address the actual features of the Web tools being used, such as their IA [11-13].”
Technology to support aging in place: key messages for policymakers and funders
Genge, C., McNeil, H., Debergue, P., & Freeman, S. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1287486 · Cited by 47
“There is a need to challenge the existing hierarchy of research evidence that venerates clinical trials as the “gold standard (Greenhalgh and Russell, 2010).”
BackgroundEvaluation of the Impacts of Innovation in the Health Care Sector: A comparative analysis
Cucciniello, M., & Nasi, G. (2013) · Public Management Review · doi:10.1080/14719037.2013.798026 · Cited by 47
“In turn, this might result in the redefinition of innovation boundaries and the actual results, including the failure of the innovation adoption (Greenhalgh and Russell 2010).”
BackgroundO’Donovan, J., Kabali, K., Taylor, C., Chukhina, M., Kading, J. C., Fuld, J. P., & O’Neil, E. (2018) · Human Resources for Health · doi:10.1186/s12960-018-0315-7 · Cited by 35
Application of Mobile Health Technologies Aimed at Salt Reduction: Systematic Review
Ali, S. H., Luo, R., Li, Y., Liu, X., Tang, C., & Zhang, P. (2019) · JMIR mhealth and uhealth · doi:10.2196/13250 · Cited by 40
“Despite the strong advocacy of mHealth as an innovative tool for improving both quality and access of health care [25-27], there remains a great need of continued systematic academic evaluations of its direct impact on key health outcomes [28-33].”
Jeffries, M., Phipps, D. L., Howard, R. L., Avery, A. J., Rodgers, S., & Ashcroft, D. M. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2131-5 · Cited by 42
“This creates complexity in the way that healthcare organisations operate [15, 32].”
BackgroundChan, R. J., Howell, D., Lustberg, M. B., Mustian, K. M., Koczwara, B., Ng, C. C., Kim, Y., Nápoles, A. M., Dixit, N., Klemanski, D., et al. (2020) · Supportive Care in Cancer · doi:10.1007/s00520-020-05513-x · Cited by 38
Mayo, F. J. D., Escalona, M. J., Mejías, M., Aragón, G., García-García, J. A., Valderrama, J. T., & Enríquez, J. G. (2015) · The Scientific World JOURNAL · doi:10.1155/2015/863591 · Cited by 32
“By the same token, Greenhalgh and Russell [6] offer an alternative set of guiding principles for e-Health evaluation based on traditions that view evaluation as a social practice rather than a scientific testing and illustrate it using England’s controversial Summary Care Record program as an…”
BackgroundOxford Handbook of Public Health Practice
(2013) · doi:10.1093/med/9780199586301.001.0001 · Cited by 35
George, A. S., LeFevre, A. E., Jacobs, T., Kinney, M. V., Buse, K., Chopra, M., Daelmans, B. M. E. G., Haakenstad, A., Huicho, L., Khosla, R., et al. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2018-001316 · Cited by 35
A systematic review of eHealth technologies for breast cancer supportive care
Gyawali, B., Bowman, M., Sharpe, I., Jalink, M., Srivastava, S., & Wijeratne, D. T. (2023) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2023.102519 · Cited by 28
Mitchell, B., Cristancho, S., Nyhof, B. B., & Lingard, L. (2017) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-006218 · Cited by 36
Evaluation of Digitalisation in Healthcare and the Quantification of the “Unmeasurable”
Cresswell, K. M., Anderson, S., Montgomery, C. M., Weir, C. J., Atter, M., & Williams, R. (2023) · Journal of General Internal Medicine · doi:10.1007/s11606-023-08405-y · Cited by 41
Ackerman, S., Tebb, K., Stein, J. C., Frazee, B. W., Hendey, G. W., Schmidt, L. A., & Gonzales, R. (2012) · Social Science & Medicine · doi:10.1016/j.socscimed.2012.09.013 · Cited by 34
“…helpful in explaining implementation dif fi culties, non-use or unintended outcomes of technology projects (Cresswell, Worth, & Sheikh, 2010; Greenhalgh & Russell, 2010), and are increasingly recognized as a valuable approach to evaluating and building health IT projects and health care…”
BackgroundFeather, J. S., Howson, M., Ritchie, L., Carter, P. D., Parry, D. T., & Koziol-McLain, J. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5455 · Cited by 28
“However, all of these evaluation approaches must bear in mind the “emergent” nature of eHealth interventions and the degree to which the objectives and nature of the evaluation of the intervention may differ as prototypes are developed and evaluators gain experience [57].”
The Crush the Crave Quit Smoking App and Young Adult Smokers: Qualitative Case Study of Affordances
Struik, L. L., Bottorff, J. L., Baskerville, N. B., & Oliffe, J. L. (2018) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9489 · Cited by 28
“Given that these interventions are situated in the daily lives of smokers, understanding end-users’ perspectives and the mechanisms by which these apps may promote behavior change is critical for adjusting and leveraging future apps focused on quitting smoking [14,15].”
BackgroundZeleke, A. A., Naziyok, T. P., Fritz, F., Christianson, L., & Röhrig, R. (2020) · Journal of Medical Internet Research · doi:10.2196/21382 · Cited by 41
“Further debate and consensus among academics and researchers in biomedical informatics should continue to determine how and when health information technology evaluation is rigorous and produces good quality data [57].”
BackgroundEvaluating health information technologies: A systematic review of framework recommendations
Neame, M. T., Sefton, G., Roberts, M., Harkness, D., Sinha, I. P., & Hawcutt, D. B. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104247 · Cited by 27
Spithoff, S., Mathieson, S., Sullivan, F., Guan, Q., Sud, A., Hum, S., & O'Brien, M. A. (2020) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2020.04.190199 · Cited by 25
“resources on an expensive trial and perhaps causing harm.(35,37,101) Conversely, if the evaluation takes too”
BackgroundRoig, F., & Saigí-Rubió, F. (2011) · Gaceta Sanitaria · doi:10.1016/j.gaceta.2011.01.004 · Cited by 29
Cresswell, K. M., Coleman, J. J., Slee, A., Morrison, Z., & Sheikh, A. (2013) · Journal of the Royal Society of Medicine · doi:10.1177/0141076813502955 · Cited by 25
“User participation is vital, although hard to achieve as it can be resource-intensive – only a system that is used will deliver the desired benefits.(14,15)”
BackgroundAgile Application of Video Telemedicine During the COVID-19 Pandemic
Dhahri, A. A., Iqbal, M. R., & Pardoe, H. (2020) · Cureus · doi:10.7759/cureus.11320 · Cited by 25
“During the current COVID-19 pandemic crisis, it was a necessity of the time to bring the change through a model to provide continuity in patient care in such a way to avoid transmission of the deadly virus [2,18].”
BackgroundGreenhalgh, T., Darbyshire, J. L., Lee, C., Ladds, E. C., & Ceolta‐Smith, J. (2024) · BMC Medicine · doi:10.1186/s12916-024-03371-6 · Cited by 42
“…meaning-making, on the grounds that reducing the complexities of a system-wide quality improvement effort to a set of abstract and generic “success factors” will miss unique aspects of the case such as historical path dependencies, personalities, framing and meaning-making and micropolitics [59].”
BackgroundSelf-citationRussell, J., Berney, L., Stansfeld, S. A., Lanz, D., Kerry, S., Chandola, T., & Bhui, K. S. (2016) · Trials · doi:10.1186/s13063-016-1497-8 · Cited by 36
“Specifically, we explore how the qualitative data questioned assumptions embedded in the trial about the nature of the learning process; exposed the ‘slippery’ and contested nature of abstracted variables on which an RCT depends [8]; challenged aspects of the trial’s logic model, and provided a rich understanding of t…”
BackgroundSelf-citationTieman, J., & Bradley, S. L. (2013) · Australian Journal of Primary Health · doi:10.1071/py13030 · Cited by 24
“These challenges include different users having different requirements and perspectives on 87 what is a successful outcome (Pawson et al 2011, Greenhalgh & Russell 2010).”
BackgroundBieg, T., Gerdenitsch, C., Schwaninger, I., Kern, B., & Frauenberger, C. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107249 · Cited by 26
Clinical Videoconferencing as eHealth: A Critical-Realist Review and Qualitative Meta-Synthesis
Ekeland, A. G., Hansen, A. H., & Bergmo, T. S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8497 · Cited by 22
Chang, L. W., Mwanika, A., Kaye, D. K., Muhwezi, W. W., Nabirye, R. C., Mbalinda, S. N., Okullo, I., Kennedy, C. E., Groves, S. L., Sisson, S. D., et al. (2011) · Informatics for Health and Social Care · doi:10.3109/17538157.2010.542530 · Cited by 24
Georgiou, A., Westbrook, J. I., & Braithwaite, J. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-68 · Cited by 28
“Although evaluations are generally conducted to identify areas for improvement or provide an overall assessment, they (like any evaluand) are susceptible to failure [2].”
BackgroundGoderis, G., Colman, E., Irusta, L. A., Hecke, A. V., Pétré, B., Devroey, D., Deun, E. V., Faes, K., Charlier, N., Verhaeghe, N., et al. (2020) · International Journal of Integrated Care · doi:10.5334/ijic.5435 · Cited by 28
Gray, C. S., Gravesande, J., Hans, P. K., Nie, J. X., Sharpe, S., Loganathan, M., Lyons, R., & Cott, C. A. (2019) · JMIR Formative Research · doi:10.2196/11950 · Cited by 19
“Narrative and ethnographic methods are being increasingly adopted in the evaluation of complex eHealth interventions [29,40-43], and they can help capture contextual variables as a means to illuminate complexities and tensions [44].”
Lai, D. J., Liu, Z., Johnston, E., Dikomitis, L., D’Oliveira, T. C., & Shergill, S. S. (2024) · Critical Care · doi:10.1186/s13054-024-05089-6 · Cited by 20
Constructive eHealth evaluation: lessons from evaluation of EHR development in 4 Danish hospitals
Høstgaard, A. M. B., Bertelsen, P., & Nøhr, C. G. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0444-2 · Cited by 27
“Greenhalgh and Russell have offered a different approach based on ten alternative principles for eHealth evaluation [78].”
BackgroundAlami, H., Lehoux, P., Attieh, R., Fortin, J., Fleet, R. P., Niang, M., Offredo, K., Rouquet, R., Ahmed, M. A. A., & Ly, B. A. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.721898 · Cited by 24
Tintorer, D. L., Manresa‐Domínguez, J. M., Ribera, E. P., Beneyto, S. F., Mundet, X., & Saigí-Rubió, F. (2018) · BMC Family Practice · doi:10.1186/s12875-018-0739-0 · Cited by 24
Cresswell, K. M., Ali, M., Avery, A. J., Barber, N., Cornford, T., Crowe, S., Fernando, B., Jacklin, A., Jani, Y. H., Klecuń, E., et al. (2011) · Figshare · Cited by 29
Klecuń, E., Lichtner, V., Cornford, T., & Petrakaki, D. (2014) · Journal of the Association for Information Systems · doi:10.17705/1jais.00354 · Cited by 23
“Recently, realist evaluation has gained influence in the health informatics community and in associated funding bodies because they seek to commission evaluations that will provide findings that are transferrable to other settings (Greenhalgh & Russell, 2010a; Oroviogoicoechea & Watson, 2009).”
BackgroundVoruganti, T., Husain, A. F., Grunfeld, E., & Webster, F. (2018) · Supportive Care in Cancer · doi:10.1007/s00520-018-4103-7 · Cited by 18
Neijenhuijs, K. I., van der Hout, A., Veldhuijzen, E., Scholten-Peeters, G. G. M., van Uden‐Kraan, C. F., Cuijpers, P. J., & Leeuw, I. M. V. (2019) · Journal of Medical Internet Research · doi:10.2196/13408 · Cited by 17
Whole System Demonstrator trial: policy, politics, and publication ethics
Greenhalgh, T. (2012) · BMJ · doi:10.1136/bmj.e5280 · Cited by 17
Self-citationFernando, M., Abell, B., Tyack, Z. F., Donovan, T., McPhail, S., & Naicker, S. (2023) · Journal of Medical Internet Research · doi:10.2196/45163 · Cited by 27
“Consequently, CDSS innovations and associated implementation strategies may not be contextually tailored to local health system settings [24,27].”
BackgroundPractice-centred evaluation and the privileging of care in health information technology evaluation
Darking, M., Anson, R., Bravo, F., Davis, J., Flowers, S., Gillingham, E., Goldberg, L., Helliwell, P., Henwood, F., Hudson, C., et al. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-243 · Cited by 18
Metadata
- Authors
- Greenhalgh, T., & Russell, J.
- Affiliations
- Queen Mary University of London (GB)
- University College London (GB)
- Corresponding author
- Greenhalgh, Trisha
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-11-02
- Volume
- 7
- Issue
- 11
- Pages
- e1000360
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2970573
- OpenAlex
- W2064799587
- Cited by
- 273
- References
- 40
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 10 · 2023: 12 · 2022: 7 · 2021: 15 · 2020: 25 · 2019: 17 · 2018: 15 · 2017: 17 · 2016: 15 · 2015: 11 · 2014: 14 · 2013: 84 · 2012: 15
- FWCI
- 46.05
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Funders
- Medical Research Council
- Topics
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- evaluation paradigms, science assessment, evaluation principles, paradigm shift, health technology implementation
- MeSH terms
- Program Evaluation, Health Care Reform, Universal Health Insurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.