Evaluating policy and service interventions: framework to guide selection and interpretation of study end points
Abstract
The effect of many cost effective policy and service interventions cannot be detected at the level of the patient. This new framework could help improve the design (especially choice of primary end point) and interpretation of evaluative studies
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Choice of appropriate methods for evaluating national EHR implementations remains a contested issue.
“Indeed, choice of appropriate methods for such evaluations remains a contested issue [29,30].”
Cites
Developing and evaluating complex interventions: the new Medical Research Council guidance
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008) · BMJ · doi:10.1136/bmj.a1655 · Cited by 10,995 · see citing works
Incidence of Adverse Events and Negligence in Hospitalized Patients
Brennan, T. A., Leape, L. L., Laird, N. M., Hebert, L. E., Localio, A. R., Lawthers, A. G., Newhouse, J. P., Weiler, P. C., & Hiatt, H. H. (1991) · New England Journal of Medicine · doi:10.1056/nejm199102073240604 · Cited by 4,631 · see citing works
An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU
Pronovost, P., Needham, D., Berenholtz, S., Sinopoli, D., Chu, H., Cosgrove, S., Sexton, B., Hyzy, R., Welsh, R., Roth, G., et al. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa061115 · Cited by 4,387 · see citing works
Effect of Reducing Interns' Work Hours on Serious Medical Errors in Intensive Care Units
Landrigan, C. P., Rothschild, J. M., Cronin, J. W., Kaushal, R., Burdick, E., Katz, J. T., Lilly, C. M., Stone, P. H., Lockley, S. W., Bates, D. W., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejmoa041406 · Cited by 1,760 · cited 1× · see citing works
“However, as the signal (change in outcome due to intervention) diminishes in relation to the noise (changes in outcome due to uncontrolled sources of variation), a study based on process measurement will become more cost effective. Such was the case in Landrigan’s study of the effects of fatigue on the quality of care…”
BackgroundA look into the nature and causes of human errors in the intensive care unit
Donchin, Y., Gopher, D., Olin, M., Badihi, Y., Bieski, M., Sprung, C. L., Pizov, R., & Cotev, S. (1995) · Critical Care Medicine · doi:10.1097/00003246-199502000-00015 · Cited by 838 · see citing works
Sazawal, S., & Black, R. (2003) · Lancet. Infectious Diseases (Print) · doi:10.1016/s1473-3099(03)00737-0 · Cited by 485 · see citing works
Publication Bias: The Problem That Won't Go Away
DICKERSIN, K., & MIN, Y. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26343.x · Cited by 427 · see citing works
Campbell, M. K., Fayers, P. M., & Grimshaw, J. M. (2005) · Clinical Trials · doi:10.1191/1740774505cn071oa · Cited by 275 · see citing works
Ethical issues in the design and conduct of cluster randomised controlled trials
Edwards, S., Braunholtz, D., Lilford, R. J., & Stevens, A. J. (1999) · BMJ · doi:10.1136/bmj.318.7195.1407 · Cited by 253 · see citing works
Self-citationMethods to reduce the impact of intraclass correlation in group-randomized trials.
Murray, D. M., & Blitstein, J. L. (2003) · Evaluation Review · doi:10.1177/0193841x02239019 · Cited by 221 · see citing works
North of England evidence based guidelines development project: methods of guideline development
Eccles, M., Clapp, Z., Grimshaw, J., Adams, P. C., Higgins, B. G., Purves, I., & Russell, I. T. (1996) · BMJ · doi:10.1136/bmj.312.7033.760 · Cited by 215 · see citing works
Brown, C., Hofer, T., Johal, A., Thomson, R., Nicholl, J., Franklin, B. D., & Lilford, R. J. (2008) · Quality and Safety in Health Care · doi:10.1136/qshc.2007.023648 · Cited by 186 · see citing works
Self-citationBarach, P. R., Johnson, J. K., Ahmad, A., Galvan, C., Bognár, A., DUNCAN, R. C., Starr, J. P., & BACHA, E. A. (2008) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2008.03.071 · Cited by 147 · see citing works
Self-citationThe measurement of active errors: methodological issues.
Lilford, R. J. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.suppl_2.ii8 · Cited by 101 · see citing works
Self-citationBrown, C., Hofer, T., Johal, A., Thomson, R., Nicholl, J., Franklin, B. D., & Lilford, R. J. (2008) · Quality and Safety in Health Care · doi:10.1136/qshc.2007.023655 · Cited by 100 · see citing works
Self-citationBrown, C., Hofer, T. P., Johal, A., Thomson, R., Nicholl, J. P., Franklin, B. D., & Lilford, R. J. (2008) · BMJ Quality & Safety · doi:10.1136/qshc.2007.023630 · Cited by 99 · see citing works
Self-citationInvesting in new medical technologies: A decision framework
Cosh, E., Girling, A. J., Lilford, R. J., McAteer, H., & Young, T. P. (2007) · Journal of Commercial Biotechnology · doi:10.1057/palgrave.jcb.3050062 · Cited by 86 · see citing works
Self-citationInter-rater reliability of case-note audit: a systematic review
Lilford, R. J., Edwards, A., Girling, A. J., Hofer, T. P., Tanna, G. L. D., Petty, J. L., & Nicholl, J. P. (2007) · Journal of Health Services Research & Policy · doi:10.1258/135581907781543012 · Cited by 74 · see citing works
Self-citationEvaluating service delivery interventions to enhance patient safety
Brown, C., & Lilford, R. J. (2008) · BMJ · doi:10.1136/bmj.a2764 · Cited by 62 · see citing works
Self-citationExperience with a structure, process and outcome framework for evaluating an information system
Cornford, T., Doukidis, G., & Forster, D. (1994) · Omega · doi:10.1016/0305-0483(94)90030-2 · Cited by 60 · see citing works
Housestaff and Medical Student Attitudes Toward Medical Errors and Adverse Events
Vohra, P. D., Johnson, J. K., Daugherty, C. K., Wen, M., & Barach, P. R. (2007) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(07)33053-5 · Cited by 55 · see citing works
Self-citationDimensions of Design Space: A Decision-Theoretic Approach to Optimal Research Design
Conti, S., & Claxton, K. (2009) · Medical Decision Making · doi:10.1177/0272989x09336142 · Cited by 40 · see citing works
Improving performance through knowledge translation in the Veterans Health Administration
Francis, J., & Perlin, J. B. (2006) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.52 · Cited by 36 · see citing works
Girling, A. J., Freeman, G. H., Gordon, J. P., Poole-Wilson, P. A., Scott, D. A., & Lilford, R. J. (2007) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462307070365 · Cited by 35 · see citing works
Self-citationQualitative versus quantitative research -- balancing cost, yield and feasibility
Runciman, W. B. (2002) · BMJ Quality & Safety · doi:10.1136/qhc.11.2.146 · Cited by 34 · see citing works
Management for Doctors: Decision analysis for medical managers
Thornton, J., & Lilford, R. J. (1995) · BMJ · doi:10.1136/bmj.310.6982.791 · Cited by 33 · see citing works
Self-citationReconciling the Quantitative and Qualitative Traditions—The Bayesian Approach
Lilford, R. J., & Braunholtz, D. (2003) · Public Money & Management · doi:10.1111/1467-9302.00369 · Cited by 22 · see citing works
Self-citationChanges to the University of Sydney medical curriculum
Goulston, K., & Oates, R. K. (2008) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2008.tb01716.x · Cited by 16 · see citing works
Should the UK government's deep cleaning of hospitals programme have been evaluated?
Brown, C., & Lilford, R. J. (2009) · Journal of Infection Prevention · doi:10.1177/1757177409106227 · Cited by 8 · see citing works
Self-citationGirling, A. J., Lilford, R. J., Braunholtz, D., & Gillett, W. R. (2007) · Clinical Trials · doi:10.1177/1740774506075872 · Cited by 5 · see citing works
Self-citationThe Safer Patients Initiative: the UK experience of attempting to improve safe clinical care
Shirley, P. J. (2008) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2008.tb02101.x · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Improving the health and welfare of people who live in slums
Lilford, R. J., Oyebode, O., Satterthwaite, D., Melendez-Torres, G. J., Chen, Y.-F., Mberu, B., Watson, S. I., Sartori, J., Ndugwa, R., Caiaffa, W., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(16)31848-7 · Cited by 305 · see citing works
Self-citationBenning, A., Ghaleb, M. A., Suokas, A. K., Dixon‐Woods, M., Dawson, J., Barber, N., Franklin, B. D., Girling, A. J., Hemming, K., Carmalt, M., et al. (2011) · BMJ · doi:10.1136/bmj.d195 · Cited by 227 · see citing works
Self-citationEvaluating the impact of prediction models: lessons learned, challenges, and recommendations
Kappen, T. H., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., Vergouwe, Y., & Moons, K. G. M. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0033-6 · Cited by 215 · see citing works
“Accordingly, preceding decision analytic studies help in separating the chaff from the wheat [45].”
BackgroundRaine, R., Fitzpatrick, R. M., Barratt, H., Bevan, G., Black, N. A., Boaden, R., Bower, P., Campbell, M., Denis, J., Devers, K. J., et al. (2016) · Health Services and Delivery Research · doi:10.3310/hsdr04160 · Cited by 187 · see citing works
Self-citationBaines, R. J., Langelaan, M., de Bruijne, M. C., Asscheman, H., Spreeuwenberg, P. M. M., van de Steeg, L., Siemerink, K. M., van Rosse, F., Broekens, M. A., & Wagner, C. (2013) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001126 · Cited by 171 · see citing works
Benning, A., Dixon‐Woods, M., Nwulu, U., Ghaleb, M. A., Dawson, J., Barber, N., Franklin, B. D., Girling, A. J., Hemming, K., Carmalt, M., et al. (2011) · BMJ · doi:10.1136/bmj.d199 · Cited by 166 · see citing works
Self-citationvan Sluisveld, N., Hesselink, G., van der Hoeven, J. G., Westert, G., Wollersheim, H., & Zegers, M. (2015) · Intensive Care Medicine · doi:10.1007/s00134-015-3666-8 · Cited by 130 · see citing works
“These reasons reflect the difficulty of demonstrating the effectiveness of complex quality improvement interventions, as has been mentioned in several publications [60–62].”
ResultRecommendations for Evaluation of Health Care Improvement Initiatives
Parry, G. J., Carson-Stevens, A., Luff, D. F., McPherson, M. E., & Goldmann, D. A. (2013) · Academic Pediatrics · doi:10.1016/j.acap.2013.04.007 · Cited by 124 · see citing works
Ayorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y. (2020) · PLoS ONE · doi:10.1371/journal.pone.0227580 · Cited by 111 · see citing works
Self-citationSchraagen, J. M., Schouten, T. J., Smit, M., Haas, F., van der Beek, D., van de Ven, J., & Barach, P. R. (2011) · BMJ Quality & Safety · doi:10.1136/bmjqs.2010.048983 · Cited by 104 · see citing works
Self-citationSecular trends and evaluation of complex interventions: the rising tide phenomenon
Chen, Y., Hemming, K., Stevens, A. J., & Lilford, R. J. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004372 · Cited by 102 · see citing works
Self-citationGolden, T. L., & Wendel, M. L. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00131 · Cited by 110 · see citing works
Cresswell, K., Callaghan, M., Khan, S., Sheikh, Z., Mozaffar, H., & Sheikh, A. (2020) · Health Informatics Journal · doi:10.1177/1460458219900452 · Cited by 87 · see citing works
“21 These may include qualitative evaluations, either as stand-alone studies or embedded with RCTs to investigate mechanisms of action.”
BackgroundMorrison, Z., Robertson, A., Cresswell, K., Crowe, S., & Sheikh, A. (2011) · Journal of Healthcare Engineering · doi:10.1260/2040-2295.2.1.25 · Cited by 81 · see citing works
“There is also a clear need for work that focuses on building an evidence base for the benefits of national EHR implementations [58, 59].”
BackgroundGroene, O., Botje, D., Suñol, R., Gómez, M. A. L., & Wagner, C. (2013) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzt058 · Cited by 76 · see citing works
Hoddinott, P., Craig, L., Maclennan, G., Boyers, D., & Vale, L. (2012) · BMJ Open · doi:10.1136/bmjopen-2011-000652 · Cited by 66 · see citing works
High Reliability Organizations and Surgical Microsystems: Re-engineering Surgical Care
Sánchez, J. A., & Barach, P. R. (2012) · Surgical Clinics of North America · doi:10.1016/j.suc.2011.12.005 · Cited by 64 · see citing works
Self-citationWhy and how to use mixed methods in primary health care research
Vedel, I., Kaur, N., Hong, Q. N., Sherif, R. E., Khanassov, V., Godard‐Sebillotte, C., Sourial, N., Yang, X. Q., & Pluye, P. (2018) · Family Practice · doi:10.1093/fampra/cmy127 · Cited by 58 · see citing works
Svendsen, M. L., Ehlers, L. H., Ingeman, A., & Johnsen, S. P. (2012) · Stroke · doi:10.1161/strokeaha.111.645184 · Cited by 54 · see citing works
Hunter, R. F., Adlakha, D., Cardwell, C. R., Cupples, M., Donnelly, M., Ellis, G., Gough, A., Hutchinson, G., Kearney, T., Longo, A., et al. (2021) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-021-01213-9 · Cited by 51 · see citing works
“These results pose several scientific and real-world implementation challenges that are too infrequently exposed in public heath intervention trials [60], including how to balance positive and negative results when primary and secondary outcomes are discordant [61].”
BackgroundTakian, A., Petrakaki, D., Cornford, T., Sheikh, A., & Barber, N. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-105 · Cited by 51 · see citing works
“Indeed, choice of appropriate methods for such evaluations remains a contested issue [29,30].”
BackgroundPhilibert, I., & Barach, P. R. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001598 · Cited by 51 · see citing works
“interventions that are aimed at the population of organisations in the field.(25) 26 They emphasise that efforts”
BackgroundSelf-citationConducting a multicentre and multinational qualitative study on patient transitions
Johnson, J. K., Barach, P. R., & Vernooij-Dassen, M. J. F. J. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001197 · Cited by 49 · see citing works
“We also used a series of quantitative methods to assess the cost effectiveness of handover training.(4) From the initial funding of the HANDOVER Project, the Project’s leadership agreed that, in addition to the research activities to meet the investigational aims of the project, there would be a separate set of activi…”
MethodSelf-citationWatson, S., Sahota, H., Taylor, C. A., Chen, Y., & Lilford, R. J. (2018) · Global Health Research and Policy · doi:10.1186/s41256-018-0073-z · Cited by 49 · see citing works
“A clinical intervention is applied directly to a patient whereas a service delivery intervention is applied to the organisation of a clinical intervention and its delivery [5].”
BackgroundInfluentialSelf-citationHEADROOM APPROACH TO DEVICE DEVELOPMENT: CURRENT AND FUTURE DIRECTIONS
Girling, A. J., Lilford, R. J., Cole, A., & Young, T. P. (2015) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462315000501 · Cited by 48 · see citing works
Self-citationSystems of service: reflections on the moral foundations of improvement
Davidoff, F. (2011) · BMJ Quality & Safety · doi:10.1136/bmjqs.2010.046177 · Cited by 46 · see citing works
“Crucial differences among the kind of measurements needed for improvement, accountability and research purposes have been well described,(32) as has the importance of using the appropriate primary outcome measurement in evaluating policy and service interventions.(12) Sophisticated frameworks for understanding context…”
BackgroundAssessing health system interventions: key points when considerting the value of randomizatioin
English, M., Schellenberg, J., & Todd, J. (2011) · Bulletin of the World Health Organization · doi:10.2471/blt.11.089524 · Cited by 45 · see citing works
Lay support for pregnant women with social risk: a randomised controlled trial
Kenyon, S. L., Jolly, K. B., Hemming, K., Hope, L., Blissett, J. M., Dann, S.-A., Lilford, R. J., & MacArthur, C. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-009203 · Cited by 46 · see citing works
“Identified benefit problem 154 (23) 160 (24)”
Self-citationAyorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01010-1 · Cited by 45 · see citing works
“ables and outcome variables) along a long service delivery causal chain [9].”
BackgroundSelf-citationRandomised health services studies
Hakama, M., Malila, N. K., & Dillner, J. (2012) · International Journal of Cancer · doi:10.1002/ijc.27561 · Cited by 43 · see citing works
Göbel, B., Zwart, D. L. M., Hesselink, G., Pijnenborg, L., Barach, P. R., Kalkman, C. J., & Johnson, J. K. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001192 · Cited by 44 · see citing works
Self-citationDyson, J. K., Rajasekhar, P. T., Wetten, A., Hamad, A. H., Ng, S. C., Paremal, S., Baqai, M. F., Lamb, C. A., Masson, S., Hudson, M., et al. (2016) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.13806 · Cited by 41 · see citing works
“This work describes the process of successfully introducing evidencebased care that reduces variability and it is logical to infer that if care is delivered at a consistently high level then a morbidity and mortality benefit will be observed if larger numbers were included.(15) Larger prospective controlled studies ma…”
Backgroundon behalf of the Physical Activity Loyalty (PAL) Study team, Hunter, R. F., Murray, J., Gough, A., Tang, J., Patterson, C., French, D., McIntosh, E., Xin, Y., & Kee, F. (2018) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-018-0758-1 · Cited by 40 · see citing works
Yao, G., Novielli, N., Manaseki‐Holland, S., Chen, Y., van der Klink, M. R., Barach, P. R., Chilton, P. J., & Lilford, R. J. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001210 · Cited by 39 · see citing works
Self-citationPressentin, K. B. V., Mash, R. J., Baldwin-Ragaven, L., Botha, R., Govender, I., & Steinberg, W. J. (2017) · South African Family Practice · doi:10.1080/20786190.2017.1348047 · Cited by 38 · see citing works
Measuring the Performance of Public Services: Principles and Practice
Pidd, M. (2012) · Cited by 38 · see citing works
Aujla, N., Chen, Y., Samarakoon, Y. M., Wilson, A. C., Grolmusova, N., Ayorinde, A., Hofer, T. P., Griffiths, F. E., Brown, C., Gill, P. S., et al. (2020) · Health Policy and Planning · doi:10.1093/heapol/czaa152 · Cited by 35 · see citing works
“In this paper, we concentrate on process, which can be broken down into processes carried out at the system level, such as use of audit and feedback or improving staff morale, and clinical processes impacting directly on patients, such as questions asked to make a diagnosis or prescribe a treatment (Lilford et al., 20…”
BackgroundSelf-citationLiu, C., Bartram, T., Casimir, G., & Leggat, S. G. (2014) · Public Management Review · doi:10.1080/14719037.2014.930507 · Cited by 34 · see citing works
“Continued research in this area is important as it will inform and influence Chinese health policy and the effective management of Chinese hospitals (Lilford et al. 2010).”
BackgroundWagner, C., Groene, O., Thompson, C., DerSarkissian, M., Klazinga, N., Arah, O., & Suñol, R. (2014) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzu020 · Cited by 33 · see citing works
“activities of professionals at the sharp end of care processes [4].”
BackgroundMajor, R., Brown, C., Shepherd, D. J., Rogers, S., Pickering, W. P., Warwick, G. L., Barber, S. R., Ashra, N., Morris, T., & Brunskill, N. J. (2019) · Journal of the American Society of Nephrology · doi:10.1681/asn.2018101042 · Cited by 34 · see citing works
Subbe, C. P., Tellier, G., & Barach, P. R. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047446 · Cited by 36 · see citing works
Self-citationHoddinott, P., Craig, L., MacLennan, G., Boyers, D., & Vale, L. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001039 · Cited by 32 · see citing works
Economic analysis of service and delivery interventions in health care
Sutton, M., Garfield-Birkbeck, S., Martin, G. P., Meacock, R., Stephen Morris, A., Sculpher, M. J., Street, A., Watson, S., & Lilford, R. J. (2018) · Health Services and Delivery Research · doi:10.3310/hsdr06050 · Cited by 31 · see citing works
Self-citationTraspaso de información en Medicina Intensiva
Sirgo, G., Fernández, M. C., Vidal, F. G., Arias, M. G., Peña, M. S. H., Ayerdi, B. A., Andrés, E. B., Sellés, A. F., García, P., García, M. G., et al. (2018) · Medicina Intensiva · doi:10.1016/j.medin.2017.12.002 · Cited by 31 · see citing works
Pressentin, K. B. V., Mash, R. J., Baldwin-Ragaven, L., Botha, R. P. G., Govender, I., Steinberg, W. J., & Esterhuizen, T. M. (2018) · The Annals of Family Medicine · doi:10.1370/afm.2133 · Cited by 31 · see citing works
Groene, O., Mora, N., Thompson, A. G. H., Saez, M., Casas, M., & Suñol, R. (2011) · BMC Health Services Research · doi:10.1186/1472-6963-11-344 · Cited by 28 · see citing works
“Attribution of quality improvement activity to outcomes might be supported by further breaking down assessment of processes into generic management processes, targeted management processes and clinical processes [41].”
BackgroundMeacock, R. (2018) · Health Economics Policy and Law · doi:10.1017/s1744133118000063 · Cited by 27 · see citing works
“The causal chain between intervention and effect may be more diluted (Lilford et al., 2010), and the effectiveness of a service intervention will be heavily dependent on the interplay of other areas of the system.”
BackgroundChallenges in Assessing the Process–Outcome Link in Practice
Parast, L. M., Doyle, B. J., Damberg, C. L., Shetty, K. D., Ganz, D. A., Wenger, N. S., & Shekelle, P. (2015) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3150-0 · Cited by 26 · see citing works
Economic impact of electronic prescribing in the hospital setting: A systematic review
Ahmed, Z., Barber, N., Jani, Y. H., Garfield, S., & Franklin, B. D. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.11.008 · Cited by 24 · see citing works
Bugnon, B., Geissbühler, A., Bischoff, T., Bonnabry, P., & von Plessen, C. (2020) · JMIR Formative Research · doi:10.2196/22319 · Cited by 22 · see citing works
“We based our model on Lilford et al’s [61] approach to mapping policy and service interventions with regard to structures, processes, outcomes, and intervening variables .”
Sirgo, G., Fernández, M. C., Vidal, F. G., Arias, M. G., Peña, M. S. H., Ayerdi, B. A., Andrés, E. B., Sellés, A. F., García, P., García, M. G., et al. (2018) · Medicina Intensiva (English Edition) · doi:10.1016/j.medine.2017.12.008 · Cited by 21 · see citing works
Ariss, S., Enderby, P., Smith, T., Nancarrow, S., Bradburn, M. J., Harrop, D., Parker, S. G., McDonnell, A., Dixon, S., Ryan, T., et al. (2015) · Health Services and Delivery Research · doi:10.3310/hsdr03010 · Cited by 21 · see citing works
Patient safety and rapid response systems
Hillman, K., Lilford, R. J., & Braithwaite, J. (2014) · The Medical Journal of Australia · doi:10.5694/mja14.01260 · Cited by 20 · see citing works
Self-citationSelf-care programmes for people living with leprosy: a scoping review
Ilozumba, O., & Lilford, R. J. (2021) · Leprosy Review · doi:10.47276/lr.92.4.317 · Cited by 19 · see citing works
Self-citationCresswell, K. M., Ali, M., Avery, A. J., Barber, N., Cornford, T., Crowe, S., Fernando, B., Jacklin, A., Jani, Y. H., Klecuń, E., et al. (2011) · Figshare · Cited by 18 · see citing works
Watson, S., Rego, R. T. T., Hofer, T. P., & Lilford, R. J. (2022) · BMJ Global Health · doi:10.1136/bmjgh-2022-008521 · Cited by 17 · see citing works
Self-citationBarach, P. R., & Kleinman, L. C. (2018) · Progress in Pediatric Cardiology · doi:10.1016/j.ppedcard.2018.02.010 · Cited by 16 · see citing works
Self-citationHunter, R. F., Cleland, C., Busby, J., Nightingale, G., Kee, F., Williams, A. J., Kelly, P., Kelly, M. P., Milton, K. E., Kokka, K., et al. (2022) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2022-219729 · Cited by 15 · see citing works
Evidence‐based inpatient handovers: a literature review and research agenda
Scott, P., Ross, P., & Prytherch, D. (2012) · Clinical Governance An International Journal · doi:10.1108/14777271211200710 · Cited by 15 · see citing works
Lilford, R. J., Daniels, B., McPake, B., Bhutta, Z. A., Mash, R. J., Griffiths, F. E., Omigbodun, A. O., Pinto, E. P., Jain, R., Asiki, G., et al. (2025) · The Lancet Global Health · doi:10.1016/s2214-109x(24)00536-9 · Cited by 14 · see citing works
Self-citationAssociation between Exercise and Metabolic Syndrome in Koreans
Lee, S. H., Lee, Y., Seo, J. H., & Kim, Y. A. (2018) · Journal of Obesity & Metabolic Syndrome · doi:10.7570/jomes.2018.27.2.117 · Cited by 14 · see citing works
Addressing barriers for change in clinical practice
Barach, P. R. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316218563.019 · Cited by 13 · see citing works
Self-citationStorm, M., Groene, O., Testad, I., Dyrstad, D. N., Heskestad, R. N., & Aase, K. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005962 · Cited by 13 · see citing works
Lilford, R. J., Girling, A. J., Sheikh, A., Coleman, J. J., Chilton, P. J., Burn, S. L., Jenkinson, D., Blake, L., & Hemming, K. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-314 · Cited by 13 · see citing works
“Conclusion We present a method to deal with the “inconvenient truth” [5] that occurs when complex generic service delivery interventions must be assessed for cost-effectiveness.”
Self-citationTowards a learning system for pediatric outcomes: Harvesting meaning from evidence
Kleinman, L. C., & Barach, P. R. (2018) · Progress in Pediatric Cardiology · doi:10.1016/j.ppedcard.2018.05.002 · Cited by 12 · see citing works
Self-citationLilford, R. J., Daniels, B., McPake, B., Bhutta, Z. A., Mash, R. J., Griffiths, F. E., Omigbodun, A. O., Pinto, E. P., Jain, R., Asiki, G., et al. (2025) · The Lancet Global Health · doi:10.1016/s2214-109x(24)00535-7 · Cited by 11 · see citing works
Self-citationJohnson, J. K., Arora, V. M., Barach, P. R., & Collaborative, O. B. O. T. E. H. R. (2013) · European Journal of General Practice · doi:10.3109/13814788.2013.819850 · Cited by 11 · see citing works
“We also used a series of quantitative methods to assess the cost eff ectiveness of handover training (11,12).”
MethodSelf-citationBarach, P. R., & Lipshultz, S. E. (2020) · Progress in Pediatric Cardiology · doi:10.1016/j.ppedcard.2020.101233 · Cited by 11 · see citing works
Self-citationBion, J., Aldridge, C., Beet, C., Boyal, A., Chen, Y., Clancy, M. J., Girling, A. J., Hofer, T. P., Lord, J. L., Mannion, R., et al. (2021) · Health Services and Delivery Research · doi:10.3310/hsdr09130 · Cited by 11 · see citing works
Self-citationRahimi, K., Patel, A. A., & MacMahon, S. W. (2012) · PubMed · Cited by 11 · see citing works
Cresswell, K. M., Coleman, J. J., Smith, P. A., Swainson, C. P., Slee, A., & Sheikh, A. (2016) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v23i2.178 · Cited by 12 · see citing works
Barach, P. R., & Lipshultz, S. E. (2016) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2016.07.027 · Cited by 10 · see citing works
Self-citationKudrna, L., Bird, P., Hemming, K., Quinn, L., Schmidtke, K. A., & Lilford, R. J. (2022) · BMJ Quality & Safety · doi:10.1136/bmjqs-2021-013514 · Cited by 10 · see citing works
Self-citationDoes Quality Improvement Face a Legitimacy Crisis? Poor Quality Studies, Small Effects
Groene, O. (2011) · Journal of Health Services Research & Policy · doi:10.1258/jhsrp.2011.011022 · Cited by 10 · see citing works
Toward a Learning System for ERAS: Embedding Implementation and Learning Evaluation
Ramaswamy, R., & Barach, P. R. (2020) · Springer eBooks · doi:10.1007/978-3-030-33443-7_39 · Cited by 9 · see citing works
Self-citationSheikh, A., Coleman, J. J., Chuter, A., Williams, R., Lilford, R. J., Slee, A., Morrison, Z., Cresswell, K. M., Robertson, A. R. R., Slight, S. P., et al. (2022) · Programme Grants for Applied Research · doi:10.3310/ksrs2009 · Cited by 11 · see citing works
Self-citationWhen is early septal myectomy in children with hypertrophic cardiomyopathy justified?
Bansal, N., Barach, P. R., Amdani, S. M., & Lipshultz, S. E. (2018) · Translational Pediatrics · doi:10.21037/tp.2018.09.08 · Cited by 9 · see citing works
Self-citationRattray, N. A., Ebright, P. R., Flanagan, M., Militello, L. G., Barach, P. R., Franks, Z., Rehman, S. U., Gordon, H. S., & Frankel, R. M. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1350-8 · Cited by 9 · see citing works
Self-citationJenniskens, K., Lagerweij, G. R., Naaktgeboren, C. A., Hooft, L., Moons, K. G. M., Poldervaart, J. M., Koffijberg, H., & Reitsma, J. B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.07.010 · Cited by 9 · see citing works
Zhang, Z., Zhang, R., Peng, Y., Zhai, S., Zhang, J., Jin, Q., Zhou, J., Li, H., & Chen, J. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-070130 · Cited by 8 · see citing works
Kumar, M. B., Madan, J. J., Mireku, M., Limato, R., Ndima, S. D., Kea, A. Z., Chikaphupha, K. R., Barasa, E., & Taegtmeyer, M. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2019-001390 · Cited by 8 · see citing works
“Measuring and attributing the downstream benefits of a service delivery intervention that are intrinsically valuable to a decision maker or population is challenging.(20) 68–72 Adding to the challenges of potential confounding, in ‘Step Six’ of the intervention (figure 1), QI teams have the freedom to design and test…”
BackgroundHow far does family physician supply correlate with district health system performance?
Dyers, R., Mash, R. J., & Naledi, T. (2015) · African Journal of Primary Health Care & Family Medicine · doi:10.4102/phcfm.v7i1.796 · Cited by 8 · see citing works
Singh, B., Lipunga, G., Thangavelu, P., Dhar, S., Cronemberger, L. F., Abhilash, K. P. P., Abraham, A. M., de Brito, C. A. A., Ferreira, M. L. B., Chandrashekar, N., et al. (2025) · The Lancet · doi:10.1016/s0140-6736(25)00263-6 · Cited by 7 · see citing works
“For the primary outcomes the analysis included a comparison of percentage of patients diagnosed before versus after the intervention, as well as interrupted time series analysis.”
Self-citationDesigning high-reliability healthcare teams
Barach, P. R. (2016) · International Conference on Information and Digital Technologies (IDT) · doi:10.1109/dt.2016.7557144 · Cited by 7 · see citing works
Self-citationChinhoyi, R. L., Zunza, M., & Pressentin, K. B. V. (2018) · African Journal of Primary Health Care & Family Medicine · doi:10.4102/phcfm.v10i1.1442 · Cited by 7 · see citing works
Implementation Science: Translating Research into Practice for Sustained Impact
Aarons, G. A., Sklar, M., & Sevdalis, N. (2017) · Springer eBooks · doi:10.1007/978-3-319-44010-1_18 · Cited by 6 · see citing works
An opportunistic evaluation of a routine service improvement project to reduce falls in hospital
Sheppard, D., Clarke, E., Hemming, K., Martín, J., & Lilford, R. J. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06073-4 · Cited by 6 · see citing works
Self-citationWatson, S., Lilford, R. J., Sun, J., & Bion, J. (2021) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/rssc.12501 · Cited by 5 · see citing works
Self-citationPressentin, K. B. V., Mash, R. J., Baldwin-Ragaven, L., Botha, R. P. G., Govender, I., & Steinberg, W. J. (2018) · South African Family Practice · doi:10.4102/safp.v60i1.4844 · Cited by 6 · see citing works
Failure to Rescue and Failure to Perceive Patients in Crisis
Subbe, C. P., & Barach, P. R. (2017) · Springer eBooks · doi:10.1007/978-3-319-44010-1_37 · Cited by 6 · see citing works
Self-citationEnterprise Risk Management in Healthcare
Levett, J. M., Fasone, J. M., Smith, A. L., Labovitz, S. S., Labovitz, J., Mellott, S., & Dotan, D. B. (2017) · Springer eBooks · doi:10.1007/978-3-319-44010-1_6 · Cited by 5 · see citing works
Discussion of “Evidence-based Health Informatics: How Do We Know What We Know?”
Al-Shorbaji, N., Hanmer, L. A., Hussein, R., Magrabi, F., Moen, A., Moura, L. A., Park, H., & Scott, P. (2015) · Methods of Information in Medicine · doi:10.3414/me14-02-0119 · Cited by 5 · see citing works
Operationalising perioperative safety: learning to walk the talk
Chejfec‐Ciociano, J. M., Barach, P. R., & Gelb, A. W. (2025) · British Journal of Anaesthesia · doi:10.1016/j.bja.2025.06.018 · Cited by 4 · see citing works
Self-citationSchmidtke, K. A., Kudrna, L., Quinn, L., Bird, P., Hemming, K., Venable, Z., & Lilford, R. J. (2024) · BMJ Quality & Safety · doi:10.1136/bmjqs-2024-017094 · Cited by 4 · see citing works
Self-citationAyorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y. (2020) · Health Services and Delivery Research · doi:10.3310/hsdr08330 · Cited by 4 · see citing works
Self-citationAujla, N., Ilori, T., Irabor, A. E., Obimakinde, A. M., Owoaje, E., Fayehun, O. A., Ajisola, M. M., Bolaji, S. O., Watson, S., Hofer, T. P., et al. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06491-4 · Cited by 4 · see citing works
“This paper focuses on the ‘process’ component of quality of care [10], which describes the clinical processes involving patients, such as the action of ordering a test or conducting an examination when necessary [11].”
BackgroundSelf-citationTools and Strategies for Continuous Quality Improvement and Patient Safety
Johnson, J. K., & Barach, P. R. (2017) · Springer eBooks · doi:10.1007/978-3-319-44010-1_9 · Cited by 4 · see citing works
Self-citationWatson, S., Chen, Y., Bion, J., Aldridge, C., Girling, A. J., & Lilford, R. J. (2018) · BMJ Open · doi:10.1136/bmjopen-2016-015561 · Cited by 4 · see citing works
Self-citationYu, X., Han, W., Jiang, J., Wang, Y., Xin, S., Wu, S., Sun, H. J., Wang, Z., & Zhao, Y. (2019) · Journal of Medical Internet Research · doi:10.2196/13576 · Cited by 4 · see citing works
“These indicators, unlike process or surrogate safety indicators (eg, error or culture), are identifiable and quantifiable and are more appropriate as natural endpoints in the story of patient safety [22].”
Lipshultz, S. E., Barach, P. R., & Wilkinson, J. D. (2017) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2017.07.749 · Cited by 4 · see citing works
“Methods for selecting and interpreting study endpoints in the evaluation of policy and service interventions remain contested (8).”
BackgroundSelf-citation
- Authors
- Lilford, R. J., Chilton, P. J., Hemming, K., Girling, A. J., Taylor, C. A., & Barach, P.
- Affiliations
- University of Birmingham (GB)
- University Medical Center Utrecht (NL)
- Corresponding author
- Lilford, R. J.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2010-08-27
- Volume
- 341
- Issue
- aug27 1
- Pages
- c4413-c4413
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W1996547462
- Cited by
- 146
- References
- 31
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 5 · 2023: 5 · 2022: 6 · 2021: 17 · 2020: 11 · 2019: 4 · 2018: 19 · 2017: 19 · 2016: 10 · 2015: 10 · 2014: 8 · 2013: 4 · 2012: 13
- FWCI
- 19.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (GR/S29874/01, EP/F063822/1)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- health policy evaluation, health service interventions, evaluation framework, cost-effectiveness
- MeSH terms
- Bayes Theorem, Cost-Benefit Analysis, Health Policy, Program Evaluation, Biomarkers, Organizational Policy, Outcome Assessment, Health Care, Process Assessment, Health Care, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review