The case for randomized controlled trials to assess the impact of clinical information systems

Liu, J. L. Y.; Wyatt, J. C.

doi:10.1136/jamia.2010.010306

Cited by 66Electronic Health Records SystemsAdvanced Causal Inference TechniquesMeta-analysis and systematic reviews

The case for randomized controlled trials to assess the impact of clinical information systems

Open access

Liu, J. L. Y., Wyatt, J. C.·2011·Journal of the American Medical Informatics Association

DOI 10.1136/jamia.2010.010306PMC PMC3116250

Open this paper in CoChat — free account Open PDF

Abstract

There is a persistent view of a significant minority in the medical informatics community that the randomized controlled trial (RCT) has a limited role to play in evaluating clinical information systems. A common reason voiced by skeptics is that these systems are fundamentally different from drug interventions, so the RCT is irrelevant. There is an urgent need to promote the use of RCTs, given the shift to evidence-based policy and the need to demonstrate cost-effectiveness of these systems. The authors suggest returning to first principles and argue that what is required is clarity about how to match methods to evaluation questions. The authors address common concerns about RCTs, and the extent to which they are fallacious, and also discuss the challenges of conducting RCTs in informatics and alternative study designs when randomized trials are infeasible. While neither a perfect nor universal evaluation method, RCTs form an important part of an evaluator's toolkit.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1136/jamia.2010.010306

Cites

125 of 125 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The study used a methodologically innovative design, captured contemporaneous multifaceted real time longitudinal data, and had theoretical grounding.
“We also used a methodologically innovative design,86 captured contemporaneous multifaceted real time longitudinal data, and had theoretical grounding.19 87”
From Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in early adopter hospitals (2011) · cites this paper as [86]
The Background and 'Question Appropriate study design' sections discuss how to choose methods for evaluating clinical information systems, but do not describe or assess the citing study's design. The 'Theory or case studies can reliably predict what works' section discusses theory-based approaches generally. I found no passage establishing that the citing study used an innovative design, collected the stated longitudinal data, or had theoretical grounding; the copy provides no page numbers for these passages. (LLM, conf=0.74)
Experimental — accuracy not yet measured
RCTs have only infrequently been used in EHR evaluations.
“Despite the exalted status of RCTs as the gold standard for assessing cause and effect or therapeutic impact, they have only infrequently been used in EHR evaluations [19].”
From Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems (2012) · cites this paper as [19]
Positivist approaches, including RCTs and before-after studies, are less able to conduct meaningful real-time formative evaluations that address such concerns.
“Positivist approaches, including RCTs and before-after studies, despite health care’s long tradition and continuing calls for their use [11,19], are less able to conduct meaningful real-time formative evaluations that address such concerns.”
From Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems (2012) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Methods for the Economic Evaluation of Health Care Programmes
Drummond, M. E., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., & Stoddart, G. L. (2005) · doi:10.1093/oso/9780198529446.001.0001 · Cited by 13,648 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371 · see citing works
Practical Statistics for Medical Research.
Walter, S., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,665 · see citing works
Practical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329 · see citing works
CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · see citing works
Design and analysis of randomized clinical trials requiring prolonged observation of each patient. II. Analysis and examples
Peto, R., Pike, M. C., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., & Smith, P. G. (1977) · British Journal of Cancer · doi:10.1038/bjc.1977.1 · Cited by 8,285 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomized trials
Schulz, K., Altman, D., & Moher, D. (2010) · Obstetrics and Gynecology · doi:10.1097/aog.0b013e3181d9d421 · Cited by 3,952 · see citing works
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · see citing works
Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936 · see citing works
Clinical Trials: A Practical Approach.
Hill, I. D., & Pocock, S. (1984) · Biometrics · doi:10.2307/2531186 · Cited by 2,918 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
Qualitative Research in Health Care
(2006) · Wiley eBooks · doi:10.1002/9780470750841 · Cited by 2,713 · see citing works
Lack of Effect of Long-Term Supplementation with Beta Carotene on the Incidence of Malignant Neoplasms and Cardiovascular Disease
Hennekens, C. H., Buring, J. E., Manson, J. E., Stampfer, M., Rosner, B., Cook, N. R., Belanger, C., LaMotte, F., Gaziano, J. M., Ridker, P. M., et al. (1996) · New England Journal of Medicine · doi:10.1056/nejm199605023341801 · Cited by 2,397 · see citing works
Equipoise and the Ethics of Clinical Research
Freedman, B. (1987) · New England Journal of Medicine · doi:10.1056/nejm198707163170304 · Cited by 2,339 · see citing works
Qualitative Research: Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research
Pope, C., & Mays, N. (1995) · BMJ · doi:10.1136/bmj.311.6996.42 · Cited by 2,336 · see citing works
Design and analysis of randomized clinical trials requiring prolonged observation of each patient. I. Introduction and design
Peto, R., Pike, M. C., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., & Smith, P. G. (1976) · British Journal of Cancer · doi:10.1038/bjc.1976.220 · Cited by 2,294 · see citing works
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015) · doi:10.1007/978-3-319-18539-2 · Cited by 2,237 · see citing works
Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors
Koppel, R. (2005) · JAMA · doi:10.1001/jama.293.10.1197 · Cited by 2,126 · see citing works
Effects of Computer-Based Clinical Decision Support Systems on Physician Performance and Patient Outcomes
Hunt, D. L., Haynes, R. B., Hanna, S., & Smith, K. (1998) · JAMA · doi:10.1001/jama.280.15.1339 · Cited by 1,684 · see citing works
Oxford Textbook of Medicine.
(1988) · Annals of Internal Medicine · doi:10.7326/0003-4819-108-3-499_7 · Cited by 1,583 · see citing works
Do women with pre-eclampsia, and their babies, benefit from magnesium sulphate? The Magpie Trial: a randomised placebo-controlled trial
Altman, D. G., Carroli, G., Duley, L., Farrell, B., Moodley, J., Neilson, J., & Smith, D. (2002) · The Lancet · doi:10.1016/s0140-6736(02)08778-0 · Cited by 1,488 · see citing works
The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview
Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun, T., McKinstry, B., Procter, R., Majeed, A., & Sheikh, A. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000387 · Cited by 1,442 · see citing works
A System of Logic, Ratiocinative and Inductive
Mill, J. S. (2023) · Routledge eBooks · doi:10.4324/9781003009450-5 · Cited by 1,430 · see citing works
Design and analysis of stepped wedge cluster randomized trials.
Hussey, M. A., & Hughes, J. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.05.007 · Cited by 1,297 · see citing works
Qualitative Research in Health Care
Pope, C., & Mays, N. 1. (2019) · Wiley eBooks · doi:10.1002/9781119410867 · Cited by 1,117 · see citing works
The Use and Interpretation of Quasi-Experimental Studies in Medical Informatics
Harris, A. D., McGregor, J. C., Perencevich, E. N., Furuno, J. P., Zhu, J., Peterson, D. E., & Finkelstein, J. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1749 · Cited by 1,023 · see citing works
Prevention of falls in the elderly trial (PROFET): a randomised controlled trial
Close, J., Ellis, M., Hooper, R., Glucksman, E., Jackson, S., & Swift, C. G. (1999) · The Lancet · doi:10.1016/s0140-6736(98)06119-4 · Cited by 962 · see citing works
The stepped wedge trial design: a systematic review
Brown, C. A., & Lilford, R. J. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-54 · Cited by 907 · see citing works
Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury—outcomes at 6 months
(2005) · The Lancet · doi:10.1016/s0140-6736(05)66552-x · Cited by 843 · see citing works
Randomized versus historical controls for clinical trials
Sacks, H., Chalmers, T. C., & Smith, H. (1982) · The American Journal of Medicine · doi:10.1016/0002-9343(82)90815-4 · Cited by 834 · see citing works
Further Observations on Penicillin
Abraham, E. P., Chain, E., Fletcher, C. M., Gardner, A. D., Heatley, N. G., Jennings, M. A., & Florey, H. W. (1941) · The Lancet · doi:10.1016/s0140-6736(00)72122-2 · Cited by 821 · see citing works
Early Treatment with Prednisolone or Acyclovir in Bell's Palsy
Sullivan, F., Swan, I. R. C., Donnan, P. T., Morrison, J. M., Smith, B. H., McKinstry, B., Davenport, R., Vale, L. D., Clarkson, J., Hammersley, V., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072006 · Cited by 769 · see citing works
Evidence-based policy: the promise of ‘realist synthesis’
Pawson, R. (2002) · Evaluation · doi:10.1177/135638902401462448 · Cited by 729 · see citing works
Interactive Health Communication Applications for people with chronic disease
Murray, E., Burns, J., See Tai, S., Lai, R., & Nazareth, I. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004274.pub4 · Cited by 722 · see citing works
Effects of Computer-based Clinical Decision Support Systems on Clinician Performance and Patient Outcome: A Critical Appraisal of Research
Johnston, M. E., Langton, K. B., Haynes, R. B., & Mathieu, A. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-2-199401150-00007 · Cited by 713 · see citing works
Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis
Clark, R. A., Inglis, S. C., McAlister, F. A., Cleland, J. G. F., & Stewart, S. (2007) · BMJ · doi:10.1136/bmj.39156.536968.55 · Cited by 672 · see citing works
Positive Therapeutic Effects of Intercessory Prayer in a Coronary Care Unit Population
Byrd, R. C. (1988) · Southern Medical Journal · doi:10.1097/00007611-198807000-00005 · Cited by 587 · see citing works
Tensions and Paradoxes in Electronic Patient Record Research: A Systematic Literature Review Using the Meta‐narrative Method
GREENHALGH, T., POTTS, H. W. W., WONG, G., BARK, P., & SWINGLEHURST, D. (2009) · The Milbank Quarterly · doi:10.1111/j.1468-0009.2009.00578.x · Cited by 541 · see citing works
Contamination in trials: is cluster randomisation the answer?
Torgerson, D. J. (2001) · BMJ · doi:10.1136/bmj.322.7282.355 · Cited by 508 · see citing works
Randomised, controlled, community-level HIV-prevention intervention for sexual-risk behaviour among homosexual men in US cities
Kelly, J. A., Murphy, D. A., Sikkema, K. J., McAuliffe, T. L., Roffman, R. A., Solomon, L. J., Winett, R. A., & Kalichman, S. C. (1997) · The Lancet · doi:10.1016/s0140-6736(97)07439-4 · Cited by 505 · see citing works
Introduction to Logic
Copi, I. M., Cohen, C., & Rodych, V. (2016) · Routledge eBooks · doi:10.4324/9781315144016 · Cited by 493 · see citing works
Screening for thrombophilia in high-risk situations: systematic review and cost-effectiveness analysis. The Thrombosis: Risk and Economic Assessment of Thrombophilia Screening (TREATS) study
Wu, O., Robertson, L., Twaddle, S., Lowe, G. D. O., Clark, P., Greaves, M., Walker, I. D., Langhorne, P., Brenkel, I., Regan, L., et al. (2006) · Health Technology Assessment · doi:10.3310/hta10110 · Cited by 489 · see citing works
Epidemiology and reporting of randomised trials published in PubMed journals
Chan, A.-W., & Altman, D. G. (2005) · The Lancet · doi:10.1016/s0140-6736(05)71879-1 · Cited by 474 · see citing works
Randomised trial of home-based psychosocial nursing intervention for patients recovering from myocardial infarction
Frasure-Smith, N., Lespérance, F., Prince, R. H., Verrier, P., Garber, R. A., Juneau, M., Wolfson, C., & Bourassa, M. G. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02142-9 · Cited by 461 · see citing works
Internet-based medical education: a realist review of what works, for whom and in what circumstances
Wong, G., Greenhalgh, T., & Pawson, R. (2010) · BMC Medical Education · doi:10.1186/1472-6920-10-12 · Cited by 443 · see citing works
Medicine and the Law
Brahams, D. (1983) · The Lancet · doi:10.1016/s0140-6736(83)90167-8 · Cited by 440 · see citing works
Informatics in Primary Care
(2002) · Computers in health care · doi:10.1007/978-1-4613-0069-4 · Cited by 425 · see citing works
Can We Systematically Review Studies That Evaluate Complex Interventions?
Shepperd, S., Lewin, S., Straus, S., Clarke, M., Eccles, M. P., Fitzpatrick, R., Wong, G., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000086 · Cited by 408 · see citing works
Experimental and quasi-experimental designs for evaluating guideline implementation strategies
Grimshaw, J. (2000) · Family Practice · doi:10.1093/fampra/17.suppl_1.s11 · Cited by 382 · see citing works
Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism.
Kaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380 · see citing works
Actor-Network Theory and its role in understanding the implementation of information technology developments in healthcare
Cresswell, K. M., Worth, A., & Sheikh, A. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-67 · Cited by 372 · see citing works
Theorising big IT programmes in healthcare: Strong structuration theory meets actor-network theory
Greenhalgh, T., & Stones, R. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2009.12.034 · Cited by 371 · see citing works
Health information technology: fallacies and sober realities
Karsh, B.-T., Weinger, M. B., Abbott, P. A., & Wears, R. L. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005637 · Cited by 365 · cited 1× · see citing works
“Clinical information systems can do no harm Karsh et al recently pointed out that ‘many [system] designers and policy makers believe that the risks of HIT [Health Information Technology] are minor and easily manageable,’ but in reality their belief is not warranted.(84) The authors called this ‘the ‘Risk Free HIT’ fal…”
Background
Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial
Eccles, M. (2002) · BMJ · doi:10.1136/bmj.325.7370.941 · Cited by 354 · see citing works
Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals
Puffer, S., Torgerson, D., & Watson, J. (2003) · BMJ · doi:10.1136/bmj.327.7418.785 · Cited by 351 · see citing works
Emerging paradigms of cognition in medical decision-making
Patel, V. L., Kaufman, D. R., & Arocha, J. F. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(02)00009-6 · Cited by 338 · see citing works
Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials
Collins, R., & MacMahon, S. W. (2001) · The Lancet · doi:10.1016/s0140-6736(00)03651-5 · Cited by 337 · see citing works
Translation of highly promising basic science research into clinical applications
Contopoulos-Ioannidis, D. G., Ntzani, E. E., & Ioannidis, J. P. A. (2003) · The American Journal of Medicine · doi:10.1016/s0002-9343(03)00013-5 · Cited by 323 · see citing works
Computer aided diagnosis of acute abdominal pain: a multicentre study.
Adams, I. D., Chan, M. Q., Clifford, P. C., Cooke, W. M., Dallos, V., de Dombal, F. T., Edwards, M., Hancock, D. M., Hewett, D. J., & McIntyre, N. (1986) · BMJ · doi:10.1136/bmj.293.6550.800 · Cited by 311 · see citing works
Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307 · see citing works
Cardiorespiratory and all-cause mortality after restrictions on sulphur content of fuel in Hong Kong: an intervention study
Hedley, A. J., Wong, C.-M., Thach, T., Ma, S., Lam, T.-H., & Anderson, H. R. (2002) · The Lancet · doi:10.1016/s0140-6736(02)11612-6 · Cited by 306 · see citing works
What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior?
Schedlbauer, A., Prasad, V., Mulvaney, C., Phansalkar, S., Stanton, W., Bates, D. W., & Avery, A. J. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2910 · Cited by 262 · see citing works
Effectiveness of health checks conducted by nurses in primary care: final results of the OXCHECK study
(1995) · BMJ · doi:10.1136/bmj.310.6987.1099 · Cited by 259 · see citing works
A randomised controlled trial of psychological debriefing for victims of road traffic accidents: Table I
Hobbs, M., Mayou, R., Harrison, B., & Worlock, P. H. (1996) · BMJ · doi:10.1136/bmj.313.7070.1438 · Cited by 251 · see citing works
Trials: the next 50 years
Peto, R., & Baigent, C. (1998) · BMJ · doi:10.1136/bmj.317.7167.1170 · Cited by 245 · see citing works
Written case simulations: Do they predict physicians' behavior?
Jones, T. V., Gerrity, M., & Earp, J. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90241-g · Cited by 233 · see citing works
Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation
Robertson, A., Cresswell, K., Takian, A., Petrakaki, D., Crowe, S., Cornford, T., Barber, N., Avery, A., Fernando, B., Jacklin, A., et al. (2010) · BMJ · doi:10.1136/bmj.c4564 · Cited by 230 · see citing works
A randomised control study of a fully automated internet based smoking cessation programme
Swartz, L. H. G., Noell, J. W., Schroeder, S., & Ary, D. V. (2006) · Tobacco Control · doi:10.1136/tc.2003.006189 · Cited by 228 · see citing works
Evaluating information technology in health care: barriers and challenges
Heathfield, H., Pitty, D., & Hanka, R. (1998) · BMJ · doi:10.1136/bmj.316.7149.1959 · Cited by 226 · see citing works
Enhancement of Clinicians' Diagnostic Reasoning by Computer-Based Consultation
Friedman, C. P., Elstein, A. S., Wolf, F. M., Murphy, G. C., Franz, T. M., Heckerling, P. S., Fine, P. L., Miller, T. M., & Abraham, V. (1999) · JAMA · doi:10.1001/jama.282.19.1851 · Cited by 219 · see citing works
A review of the clinical effectiveness and cost-effectiveness of routine anti-D prophylaxis for pregnant women who are rhesus-negative
Chilcott, J. B., Jones, M. L., Wight, J., Forman, K., Wray, J., Beverley, C., & Tappenden, P. (2003) · Health Technology Assessment · doi:10.3310/hta7040 · Cited by 214 · see citing works
The Value of Service Dogs for People With Severe Ambulatory Disabilities
Allen, K. (1996) · JAMA · doi:10.1001/jama.275.13.1001 · Cited by 204 · see citing works
The clinical value of computerized information services. A review of 98 randomized clinical trials
Balas, E. A. (1996) · Archives of Family Medicine · doi:10.1001/archfami.5.5.271 · Cited by 191 · see citing works
Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery
Hlatky, M. A., Califf, R. M., Harrell, F. E., Lee, K. L., Mark, D. B., & Pryor, D. B. (1988) · Journal of the American College of Cardiology · doi:10.1016/0735-1097(88)90086-1 · Cited by 180 · see citing works
Interactive Health Communication Applications for people with chronic disease (Withdrawn paper. 2004, art. no. CD004274.pub3)
Murray, E. J., Burns, J., Tai, S. S., Lai, R., & Nazareth, I. (2005) · UCL Discovery (University College London) · Cited by 171 · see citing works
Predicting outcome in coronary disease. Statistical models versus expert clinicians.
Lee, K. L., Pryor, D., Harrell, F., Califf, R., Behar, V. S., Floyd, W. L., Morris, J. J., Waugh, R., Whalen, R., & Rosati, R. (1986) · American Journal of Medicine · doi:10.1016/0002-9343(86)90807-7 · Cited by 158 · see citing works
Effect of cognitive-behavioural training on job-finding among long-term unemployed people
Proudfoot, J. G., Guest, D. E., Carson, J. F., Dunn, G., & Gray, J. A. (1997) · The Lancet · doi:10.1016/s0140-6736(96)09097-6 · Cited by 151 · see citing works
Experimentation and social interventions: a forgotten but important history
Oakley, A. (1998) · BMJ · doi:10.1136/bmj.317.7167.1239 · Cited by 150 · see citing works
Computerization Can Create Safety Hazards: A Bar-Coding Near Miss
McDonald, C. J. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-7-200604040-00010 · Cited by 146 · see citing works
Equipoise, design bias, and randomized controlled trials: the elusive ethics of new drug development
Fries, J. F., & Krishnan, E. (2004) · Arthritis Research & Therapy · doi:10.1186/ar1170 · Cited by 143 · see citing works
Impact of user fees on attendance at a referral centre for sexually transmitted diseases in Kenya
Moses, S. (1992) · The Lancet · doi:10.1016/0140-6736(92)91778-7 · Cited by 134 · see citing works
Randomised controlled trial of clinical decision support tools to improve learning of evidence based medicine in medical students
Leung, G. M., Johnston, J. M., Tin, K. Y. K., Wong, I. O. L., Ho, L.-M., Lam, W. W. T., & Lam, T.-H. (2003) · BMJ · doi:10.1136/bmj.327.7423.1090 · Cited by 130 · see citing works
Medical Device Technology
(2021) · Advancing microelectronics · doi:10.4071/2380-7016-48.3.1 · Cited by 129 · see citing works
In the Era of Systematic Reviews, Does the Size of an Individual Trial Still Matter?
Guyatt, G. H., Mills, E. J., & Elbourne, D. R. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0050004 · Cited by 119 · see citing works
Telehealthcare for asthma
McLean, S., Chandler, D., Nurmatov, U., Liu, J. L., Pagliari, C., Car, J., & Sheikh, A. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007717.pub2 · Cited by 119 · see citing works
Self-citation
Handbook of Evaluation Methods for Health Informatics
(2006) · Elsevier eBooks · doi:10.1016/b978-0-12-370464-1.x5000-x · Cited by 115 · see citing works
Randomised trial of educational visits to enhance use of systematic reviews in 25 obstetric units
Wyatt, J. C., Paterson-Brown, S., Johanson, R., Altman, D. G., Bradburn, M. J., & Fisk, N. M. (1998) · BMJ · doi:10.1136/bmj.317.7165.1041 · Cited by 114 · see citing works
Self-citation
Good intentions and received wisdom are not enough
MacIntyre, S., & PETTICREW, M. (2000) · Journal of Epidemiology & Community Health · doi:10.1136/jech.54.11.802 · Cited by 112 · see citing works
Patterns of Ordering Diagnostic Tests for Patients with Acute Low Back Pain
Carey, T. S., Garrett, J., & North Carolina Back Pain Project* (1996) · Annals of Internal Medicine · doi:10.7326/0003-4819-125-10-199611150-00004 · Cited by 110 · see citing works
Verifying quality and safety in health informatics services
Rigby, M., Forsström, J., Roberts, R., & Wyatt, J. C. (2001) · BMJ · doi:10.1136/bmj.323.7312.552 · Cited by 100 · see citing works
Self-citation
Health Information Management: Integrating Information Technology in Health Care Work
Berg, B. (2003) · Cited by 100 · see citing works
When and how to evaluate health information systems?
Wyatt, J. C. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(02)00108-9 · Cited by 95 · see citing works
Self-citation
Teledermatologic Consultation and Reduction in Referrals to Dermatologists: A Cluster Randomized Controlled Trial
Eminović, N., de Keizer, N. F., Wyatt, J. C., ter Riet, G., Peek, N., van Weert, H. C., Bruijnzeel-Koomen, C. A., & Bindels, P. J. E. (2009) · Archives of Dermatology · doi:10.1001/archdermatol.2009.44 · Cited by 89 · see citing works
Self-citation
Computer-aided quality assurance. A critical appraisal
Haynes, R. B. (1987) · Archives of Internal Medicine · doi:10.1001/archinte.147.7.1297 · Cited by 76 · see citing works
Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation
Goud, R., de Keizer, N. F., ter Riet, G., Wyatt, J. C., Hasman, A., Hellemans, I. M., & Peek, N. B. (2009) · BMJ · doi:10.1136/bmj.b1440 · Cited by 75 · see citing works
Self-citation
Effectiveness of health checks conducted by nurses in primary care: final results of the OXCHECK study. Imperial Cancer Research Fund OXCHECK Study Group.
(1995) · PubMed Central · Cited by 75 · see citing works
Effects of an Ambient Air Pollution Intervention and Environmental Tobacco Smoke on Children's Respiratory Health in Hong Kong
Peters, J., Hedley, A. J., Wong, C. M., Lam, T., Ong, S. G. K., Liu, J. Y., & Spiegelhalter, D. J. (1996) · International Journal of Epidemiology · doi:10.1093/ije/25.4.821 · Cited by 71 · see citing works
Self-citation
The impact of eHealth on the quality & safety of healthcare: a systematic overview & synthesis of the literature
Car, J., Black, A. D., Anandan, C., Cresswell, K. M., Pagliari, C., Mckinstry, B., Procter, R., Majeed, A., & Sheikh, A. (2008) · Cited by 66 · see citing works
Systematic reviews of clinical decision tools for acute abdominal pain
Liu, J., Wyatt, J. C., Deeks, J. J., Clamp, S. E., Keen, J., Verde, P. E., Ohmann, C., Wellwood, J. M., Dawes, M. G., & Altman, D. G. (2006) · Health Technology Assessment · doi:10.3310/hta10470 · Cited by 62 · see citing works
Self-citation
Research priorities in traditional Chinese medicine
Tang, J.-L. (2006) · BMJ · doi:10.1136/bmj.333.7564.391 · Cited by 57 · see citing works
Translation of basic research into useful treatments: how often does it occur?
Crowley, W. F. (2003) · The American Journal of Medicine · doi:10.1016/s0002-9343(03)00119-0 · Cited by 53 · see citing works
Implementation of Clinical Guidelines via a Computer Charting System: Effect on the Care of Febrile Children Less than Three Years of Age
Schriger, D. L., Baraff, L. J., Buller, K., Shendrikar, M. A., Nagda, S., Lin, E., Mikulich, V. J., & Cretin, S. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070186 · Cited by 53 · see citing works
A Plea for Controlled Trials in Medical Informatics
Tierney, W. M., Overhage, J. M., & McDonald, C. J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236170 · Cited by 51 · see citing works
The value of service dogs for people with severe ambulatory disabilities
Allen, K., & Blascovich, J. (1996) · International Journal of Trauma Nursing · doi:10.1016/s1075-4210(96)80059-9 · Cited by 51 · see citing works
Studies on the respiratory health of primary school children in urban communities of Hong Kong
Ong, S. G. K., Liu, J., Wong, C. M., Lam, T., Tam, A. Y. C., Daniel, L., & Hedley, A. J. (1991) · The Science of The Total Environment · doi:10.1016/0048-9697(91)90025-a · Cited by 48 · see citing works
Self-citation
Clinical websites are currently dangerous to health
Roberts, J., & Copeland, K. L. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00162-9 · Cited by 47 · see citing works
How does computer-aided diagnosis improve the management of acute abdominal pain?
Wellwood, J. M., Johannessen, S. I., & Spiegelhalter, D. J. (1992) · PubMed · Cited by 46 · see citing works
People, Organizational, and Social Issues: Evaluation as an exemplar
Kaplan, B., & Shaw, N. T. (2002) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638129 · Cited by 46 · see citing works
Do bedbugs transmit hepatitis B?
Vall-Mayans, M., Hall, A. J., Inskip, H. M., Chotard, J., Mendy, M. E., Lindsay, S. W., & Whittle, H. (1994) · The Lancet · doi:10.1016/s0140-6736(94)91838-4 · Cited by 44 · see citing works
Large-scale randomized evidence: trials and overviews
Gray, R. G., Collins, R., Peto, R., & Wheatley, K. (2008) · CRC Press eBooks · doi:10.1201/b13550-64 · Cited by 41 · see citing works
Making existing technology safer in healthcare
Newton, R. C., Mytton, O. T., Aggarwal, R., Runciman, W. B., Free, M. J., Fahlgren, B., Akiyama, M., Farlow, B., Yaron, S., Locke, G., et al. (2010) · BMJ Quality & Safety · doi:10.1136/qshc.2009.038539 · Cited by 39 · see citing works
Is medical informatics a mature science? A review of measurement practice in outcome studies of clinical systems
Friedman, C. P. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(02)00109-0 · Cited by 38 · see citing works
The quality of evidence in health informatics: How did the quality of healthcare IT evaluation publications develop from 1982 to 2005?
de Keizer, N. F., & Ammenwerth, E. (2007) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.11.009 · Cited by 35 · see citing works
Cluster randomized trials produced similar results to individually randomized trials in a meta-analysis of enhanced care for depression
Gilbody, S. M., Bower, P., Torgerson, D. J., & Richards, D. A. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.015 · Cited by 31 · see citing works
The application of a computerized problem-oriented medical record system and its impact on patient care
Ho, L., MCGHEE, S., Hedley, A. J., & Leong, J. K. (1999) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(99)00019-2 · Cited by 30 · see citing works
Current evaluations of information technology in health care are often inadequate.
Heathfield, H. A., & Buchan, I. E. (1996) · BMJ · doi:10.1136/bmj.313.7063.1008 · Cited by 29 · see citing works
Invited Commentary: Studies in Mechanism and Prevention: Striking a Proper Balance
Wynder, E. L. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117045 · Cited by 28 · see citing works
An efficacy-driven approach to the research and development of traditional Chinese medicine.
Tang, J., & Leung, P. C. (2001) · PubMed · Cited by 21 · see citing works
Health informatics as a tool to improve quality in non-acute care — new opportunities and a matching need for a new evaluation paradigm
Rigby, M. (1999) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(99)00038-6 · Cited by 17 · see citing works
Evaluation of patient care information systems
Heathfield, H., de Mul, M., Berg, M., & Stoop, A. P. (2003) · Routledge health management series · doi:10.4324/9780203563298.ch11 · Cited by 11 · see citing works
E-health: implementation and evaluation research in Scotland – a scoping exercise
Clark, J. S., Mair, F. S., O’Donnell, C. A., & Liu, J. K. (2008) · Journal of Telemedicine and Telecare · doi:10.1258/jtt.2008.003004 · Cited by 10 · see citing works
Self-citation
Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research
Black, A. D., Car, J., Majeed, A., & Sheikh, A. (2008) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v16i3.690 · Cited by 9 · see citing works
FDA final guidance on software validation.
Donawa, M. E. (2002) · PubMed · Cited by 4 · see citing works
Pre-Medline evidence of stroke mortality trends
Kd, B. (2003) · The Lancet · doi:10.1016/s0140-6736(03)12650-5 · Cited by 1 · see citing works

Flagged first, then by citations

2 of the 127 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

Kaplan B. Evaluating informatics applicationsdsome alternative approaches, and call for methodological pluralism. Int J Med Inform 2001;64:39e56.
Friedman CP, Wyatt JC. Evaluation Methods in Bio-Medical Informatics. 2nd edn. New York: Springer-Verlag, 2006.
Rothman KJ, Greenland S. Modern Epidemiology. 2nd edn. Philadelphia: Lippincott-Raven, 1998.
Moher D, Hopewell S, Schulz KF, et al. The CONSORT Statement 2010 updated guidelines for reporting randomized trials: explanation and elaboration. BMJ 2010;340:c869.
Liu JLY, Wyatt JC. The Impact of Telepsychiatry, Teledermatology, Teleent and Teleradiology on the Health Services: A Rapid Review. Portland, OR: Medicaid Evidence-Based Decisions Program, 2008.

Sorted by citations

Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials
Roshanov, P. S., Fernandes, N., Wilczynski, J. M., Hemens, B. J., You, J. J., Handler, S. M., Nieuwlaat, R., Souza, N. M., Beyene, J., Spall, H. G. C. V., et al. (2013) · BMJ · doi:10.1136/bmj.f657 · Cited by 458 · see citing works
Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in early adopter hospitals
Sheikh, A., Cornford, T., Barber, N., Avery, A., Takian, A., Lichtner, V., Petrakaki, D., Crowe, S., Marsden, K., Robertson, A., et al. (2011) · BMJ · doi:10.1136/bmj.d6054 · Cited by 314 · see citing works
Clinical Decision Support: a 25 Year Retrospective and a 25 Year Vision
Sittig, D. F., Wright, A. T., & Middleton, B. (2016) · Yearbook of Medical Informatics · doi:10.15265/iys-2016-s034 · Cited by 276 · see citing works
Why National eHealth Programs Need Dead Philosophers: Wittgensteinian Reflections on Policymakers’ Reluctance to Learn from History
GREENHALGH, T., RUSSELL, J., ASHCROFT, R. E., & PARSONS, W. (2011) · Milbank Quarterly · doi:10.1111/j.1468-0009.2011.00642.x · Cited by 119 · see citing works
Does health informatics have a replication crisis?
Coiera, E. W., Ammenwerth, E., Georgiou, A., & Magrabi, F. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy028 · Cited by 105 · see citing works
Health Professions’ Digital Education: Review of Learning Theories in Randomized Controlled Trials by the Digital Health Education Collaboration
Bajpai, S., Semwal, M., Bajpai, R. C., Car, J., & Ho, A. H. Y. (2019) · Journal of Medical Internet Research · doi:10.2196/12912 · Cited by 87 · see citing works
How can clinicians, specialty societies and others evaluate and improve the quality of apps for patient use?
Wyatt, J. C. (2018) · BMC Medicine · doi:10.1186/s12916-018-1211-7 · Cited by 61 · see citing works
Self-citation
Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study
Bonten, T. N., Rauwerdink, A., Wyatt, J. C., Kasteleyn, M. J., Witkamp, L., Riper, H. M., van Gemert‐Pijnen, J. E. W. C., Cresswell, K. M., Sheikh, A., Schijven, M. P., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/17774 · Cited by 81 · see citing works
Self-citation
The effect of computerized provider order entry systems on clinical care and work processes in emergency departments: a systematic review of the quantitative literature.
Georgiou, A., Prgomet, M., Paoloni, R., Creswick, N., Hordern, A., Walter, S., & Westbrook, J. (2013) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2013.01.028 · Cited by 84 · see citing works
Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial
Lau, A. Y. S., Sintchenko, V., Crimmins, J., Magrabi, F., Gallego, B., & Coiera, E. W. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000433 · Cited by 69 · see citing works
Policing in the Era of Big Data
Ridgeway, G. (2018) · Annual Review of Criminology · doi:10.1146/annurev-criminol-062217-114209 · Cited by 55 · see citing works
Improving diagnostic accuracy using EHR in emergency departments: A simulation-based study
Ben‐Assuli, O., Sagi, D., Leshno, M., Ironi, A., & Ziv, A. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.03.004 · Cited by 61 · see citing works
The use of computerized clinical decision support systems in emergency care: a substantive review of the literature
Bennett, P., & Hardiker, N. R. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw151 · Cited by 57 · see citing works
Electronic Symptom Reporting Between Patient and Provider for Improved Health Care Service Quality: A Systematic Review of Randomized Controlled Trials. Part 1: State of the Art
Johansen, M. A., Henriksen, E. K. K., Horsch, A., Schuster, T., & Berntsen, G. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2214 · Cited by 54 · see citing works
Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems
Takian, A., Petrakaki, D., Cornford, T., Sheikh, A., & Barber, N. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-105 · Cited by 51 · see citing works
What makes a good clinical app? Introducing the RCP Health Informatics Unit checklist
Wyatt, J. C., Thimbleby, H., Rastall, P., Hoogewerf, J., Wooldridge, D., & Williams, J. (2015) · Clinical Medicine · doi:10.7861/clinmedicine.15-6-519 · Cited by 52 · see citing works
Self-citation
Standards for reporting randomized controlled trials in medical informatics: a systematic review of CONSORT adherence in RCTs on clinical decision support
Augestad, K. M., Berntsen, G., Lassen, K., Bellika, J. G., Wootton, R., & Lindsetmo, R. O. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000411 · Cited by 52 · see citing works
Evidence synthesis, digital scribes, and translational challenges for artificial intelligence in healthcare
Coiera, E. W., & Liu, S. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100860 · Cited by 52 · see citing works
Maintaining a Consistent Big Picture: Meaningful Use of a Web-based POC EHR System
Keenan, G. M., Yakel, E., Yao, Y., Xu, D., Szalacha, L. A., Tschannen, D. J., Ford, Y., Chen, Y.-C., Johnson, A. E., Lopez, K. D., et al. (2012) · International Journal of Nursing Knowledge · doi:10.1111/j.2047-3095.2012.01215.x · Cited by 43 · see citing works
Enhancing trust in clinical decision support systems: a framework for developers
Jones, C. H. D., Thornton, J., & Wyatt, J. C. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100247 · Cited by 46 · see citing works
Self-citation
Systematic review of clinical decision support interventions with potential for inpatient cost reduction
Fillmore, C. L., Bray, B. E., & Kawamoto, K. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-135 · Cited by 37 · see citing works
Emergency Physicians’ Knowledge and Attitudes of Clinical Decision Support in the Electronic Health Record: A Survey‐based Study
Ballard, D. W., Rauchwerger, A. S., Reed, M., Vinson, D. R., Mark, D. G., Offerman, S. R., Chettipally, U. K., Graetz, I., Dayan, P. S., Kuppermann, N., et al. (2013) · Academic Emergency Medicine · doi:10.1111/acem.12109 · Cited by 37 · see citing works
Reducing Communication Delays and Improving Quality of Care with a Tuberculosis Laboratory Information System in Resource Poor Environments: A Cluster Randomized Controlled Trial
Blaya, J. A., Shin, S. S., Yagui, M., Contreras, C., Cegielski, P., Yale, G., Suarez, C., Asencios, L., Bayona, J., Kim, J., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090110 · Cited by 36 · see citing works
Development and Assessment of an E-Learning Course on Breast Imaging for Radiographers: A Stratified Randomized Controlled Trial
Moreira, I. C., Ventura, S. M. R., Ramos, I. M., & Rodrigues, P. P. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3344 · Cited by 34 · see citing works
Telemedicine and the assessment of clinician time: a scoping review
Kidholm, K., Jensen, L. K., Johansson, M., & Montori, V. M. (2023) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462323002830 · Cited by 34 · see citing works
Validity of the Model for Assessment of Telemedicine: A Delphi study
Kidholm, K., Jensen, L. K., Kjølhede, T., Nielsen, E., & Hørup, M. B. (2016) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x16686553 · Cited by 31 · see citing works
A review of measurement practice in studies of clinical decision support systems 1998–2017
Scott, P., Brown, A. W., Adedeji, T., Wyatt, J. C., Georgiou, A., Eisenstein, E. L., & Friedman, C. P. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz035 · Cited by 27 · see citing works
Self-citation
Complex self-management interventions in chronic disease unravelled: a review of lessons learned from an individual patient data meta-analysis
Jonkman, N. H., Groenwold, R. H. H., Trappenburg, J. C. A., Hoes, A. W., & Schuurmans, M. J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.01.004 · Cited by 27 · see citing works
Assessing the Views of Professionals, Patients, and Care Partners Concerning the Use of Computer Tools in Memory Clinics: International Survey Study
van Gils, A. M., Visser, L. N. C., Hendriksen, H. M. A., Georges, J., Muller, M., Bouwman, F. H., van der Flier, W. M., & Rhodius‐Meester, H. F. M. (2021) · JMIR Formative Research · doi:10.2196/31053 · Cited by 24 · see citing works
A Scoping Review of Economic Evaluations Alongside Randomised Controlled Trials of Home Monitoring in Chronic Disease Management
Kidholm, K., & Kristensen, M. B. D. (2017) · Applied Health Economics and Health Policy · doi:10.1007/s40258-017-0351-9 · Cited by 20 · see citing works
Advocate
Ariel, B. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108278423.022 · Cited by 17 · see citing works
Electronic Health Records Improve the Quality of Care in Underserved Populations: A Literature Review
Weinfeld, J. M., Davidson, L. W., & Mohan, V. (2012) · Journal of Health Care for the Poor and Underserved · doi:10.1353/hpu.2012.0134 · Cited by 11 · see citing works
Mixed Methods: A Paradigm for Holistic Evaluation of Health IT.
Scott, P. (2016) · PubMed · Cited by 8 · see citing works
Expanding the boundaries of clinical informatics for interdisciplinary systems research
King, N. E. (2014) · Health Systems · doi:10.1057/hs.2013.18 · Cited by 7 · see citing works
Should we embed randomized controlled trials within action research: arguing from a case study of telemonitoring
Day, K., Kenealy, T. W., & Sheridan, N. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0175-6 · Cited by 7 · see citing works
Discussion of “Combining Health Data Uses to Ignite Health System Learning”
Denaxas, S., Friedman, C. P., Geissbühler, A., Hemingway, H., Kalra, D., Kimura, M., Kuhn, K. A., Payne, H. A., de Quirós, F. G. B., & Wyatt, J. C. (2015) · Methods of Information in Medicine · doi:10.3414/me15-12-0004 · Cited by 7 · see citing works
Self-citation
Evaluation of Biomedical and Health Information Resources
Friedman, C. P., & Wyatt, J. C. (2013) · Springer eBooks · doi:10.1007/978-1-4471-4474-8_11 · Cited by 6 · see citing works
Self-citation
A Quantitative Study Investigating the Effects of Computerised Clinical Decision Support in the Emergency Department
Bennett, P., & Hardiker, N. R. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-658-3-53 · Cited by 5 · see citing works
Learner's satisfaction within a breast imaging eLearning course for radiographers
Moreira, I. C., Ventura, S. M. R., Ramos, I. M., & Rodrigues, P. P. (2013) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2013.6627791 · Cited by 5 · see citing works
Discussion of “Evidence-based Health Informatics: How Do We Know What We Know?”
Al-Shorbaji, N., Hanmer, L. A., Hussein, R., Magrabi, F., Moen, A., Moura, L. A., Park, H., & Scott, P. (2015) · Methods of Information in Medicine · doi:10.3414/me14-02-0119 · Cited by 5 · see citing works
Mixed Methods: A Paradigm for Holistic Evaluation of Health IT
Scott Philip, J. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-635-4-102 · Cited by 4 · see citing works
Prediction model study focusing on eHealth in the management of urinary incontinence: the Personalised Advantage Index as a decision-making aid
Loohuis, A. M. M., Burger, H., Wessels, N. J., Dekker, J. H., Malmberg, A. G., Berger, M. Y., Blanker, M. H., & van der Worp, H. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-051827 · Cited by 3 · see citing works
Medische apps: zorg voor de toekomst?
Loohuis, A. M. M., & Chavannes, N. H. (2017) · Huisarts en Wetenschap · doi:10.1007/s12445-017-0274-6 · Cited by 3 · see citing works
Health Informatics
Parry, D. (2013) · Springer eBooks · doi:10.1007/978-3-642-30574-0_34 · Cited by 3 · see citing works
Evaluation of informatics systems: beyond RCTs and beyond the hospital
Lehmannm, H., & Lucila Ohno-machado, A. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000145 · Cited by 3 · see citing works
Evaluation of Biomedical and Health Information Resources
Friedman, C. P., & Wyatt, J. C. (2021) · Springer eBooks · doi:10.1007/978-3-030-58721-5_13 · Cited by 3 · see citing works
Self-citation
Better value digital health: the medium, the market and the role of openness
Reynolds, C. (2013) · Clinical Medicine · doi:10.7861/clinmedicine.13-4-336 · Cited by 2 · see citing works
An Immune Memory and Negative Selection to Visualize Clinical Pathways from Electronic Health Record Data
Berquedich, M., Kamach, O., Malek Masmoudi, A., & Deshayes, L. (2020) · Springer eBooks · doi:10.1007/978-3-030-45240-7_6 · Cited by 2 · see citing works
Health Informatics for Healthcare Quality Improvement: A Literature Review of Issues, Challenges and Findings
Parthasarathy, R., & Steinbach, T. A. (2015) · Journal of the Association for Information Systems · Cited by 2 · see citing works
Self-management interventions for patients with a chronic disease: what works and in whom?
Jonkman, N. H. (2016) · Utrecht University Repository (Utrecht University) · Cited by 2 · see citing works
Developing and implementing a new health information technology innovation to improve patient safety in the Canadian context
Hohl, C. M., Okpani, A. I., & Kuziemsky, C. E. (2025) · Healthcare Management Forum · doi:10.1177/08404704251346951 · Cited by 1 · see citing works
How artificial intelligence is disrupting the field of medical knowledge and evaluation: a narrative review
Cresswell, K. M., & Williams, R. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdh-2026-000104 · Cited by 1 · see citing works
Avoiding Adverse Consequences of E-Health
O’Hanlon, S. (2012) · IGI Global eBooks · doi:10.4018/978-1-4666-2657-7.ch002 · Cited by 1 · see citing works
Design of Interventional Studies
Friedman, C. P., Wyatt, J. C., & Ash, J. S. (2022) · Health informatics · doi:10.1007/978-3-030-86453-8_11 · Cited by 1 · see citing works
Self-citation
Supporting cancer patients in managing distress: new insights in the use of the Distress Thermometer & Problem List and effects of web-based support programs
Admiraal, J. M. (2020) · doi:10.33612/diss.112109033 · Cited by 1 · see citing works
Clinical decision support systems in the care of hospitalised patients with diabetes
Nirantharakumar, K. (2013) · University of Birmingham Institutional Research Archive (University of Birmingham)
Clinical Decision Support Systems for Brain Tumour Diagnosis: Classification and Evaluation Approaches
Vicente, J. (2012) · doi:10.4995/thesis/10251/17468
Health Information Systems
Breant, C. M., & Systems, S. E. F. T. I. Y. S. O. H. I. (2012) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1639435
Health information systems. What are the underlying links to enhanced health care delivery?
Breant, C. M. (2012) · PubMed
Organisational change associated with introduction of an electronic health record in residential aged care homes
Diment, K. (2015) · Research Online (University of Wollongong)
Self-management support for outpatients with cancer pain
Hochstenbach (2017) · doi:10.26481/dis.20170630lh
Medication optimisation
Gonzalvo, C. M. (2018) · doi:10.26481/dis.20180412cm
App-based treatment for female urinary incontinence
Loohuis, A. M. M. (2021) · doi:10.33612/diss.182829039
Navigating through digital health outcome evaluation methodologies: an algorithm based on a scoping review and emerging methodologies. (Preprint)Preprint
Perrin, C., Randriambelonoro, M., & Geissbühler, A. (2019) · doi:10.2196/preprints.15611
Digital tools to support dementia diagnosis
van Gils, A. M. (2024) · doi:10.5463/thesis.750
Cognitive Load and Usability Assessment in Health Information Systems
Senthilrajah, T., & Ahangama, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93724-8_13

Evaluation of informatics systems: beyond RCTs and beyond the hospital
Lehmannm, H., & Lucila Ohno-machado, A. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000145 · Cited by 3 · see citing works
Evaluating information technology in health care: barriers and challenges
Heathfield, H., Pitty, D., & Hanka, R. (1998) · BMJ · doi:10.1136/bmj.316.7149.1959 · Cited by 226 · see citing works
Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism.
Kaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380 · see citing works
The clinical value of computerized information services. A review of 98 randomized clinical trials
Balas, E. A. (1996) · Archives of Family Medicine · doi:10.1001/archfami.5.5.271 · Cited by 191 · see citing works
The Trial Bank system: capturing randomized trials for evidence-based medicine.
Sim, I., Olasov, B., & Carini, S. (2003) · Europe PMC (PubMed Central) · Cited by 19 · see citing works

Authors: Liu, J. L. Y., & Wyatt, J. C.
Affiliations: University of Dundee (GB)
Corresponding author: Liu, J. L. Y.
ORCID: Wyatt, J. C.
Venue: Journal of the American Medical Informatics Association
Publisher: Oxford University Press (OUP)
Published: 2011-03-01
Volume: 18
Issue: 2
Pages: 173-180
ISSN: 1067-5027 (ISSN-L)
1527-974X
DOI: 10.1136/jamia.2010.010306
PMC: PMC3116250
OpenAlex: W2164176905
Cited by: 66
References: 127
Citations by year: 2026: 1 · 2025: 2 · 2024: 1 · 2023: 1 · 2022: 3 · 2021: 4 · 2020: 3 · 2019: 4 · 2018: 3 · 2017: 5 · 2016: 8 · 2015: 7 · 2014: 2 · 2013: 9 · 2012: 9
FWCI: 12.30
Type: journal article
Language: en
Open access: bronze
doi.org
Locations: Journal of the American Medical Informatics Association (publishedVersion)
Europe PMC (PubMed Central) (publishedVersion)
PubMed Central (publishedVersion)
PubMed (publishedVersion)
Discovery Research Portal (University of Dundee) (publishedVersion)
DigitalGeorgetown (Georgetown University Library) (submittedVersion)
Funders: Scottish Funding Council
Topics: Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords: randomized controlled trials, clinical information systems, evidence-based policy, cost-effectiveness, evaluation methodology
MeSH terms: Health Services Research, Humans, Information Systems, Research Design, Randomized Controlled Trials as Topic, Evidence-Based Practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The case for randomized controlled trials to assess the impact of clinical information systems

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks