CoChat

Does health informatics have a replication crisis?

Open access
Coiera, Enrico, Ammenwerth, Elske, Georgiou, Andrew, Magrabi, Farah · 2018 · Journal of the American Medical Informatics Association
Cited by 106

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

Abstract

Objective: Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces. Methods: Narrative review of recent literature on research replication challenges. Results: While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting. Conclusions: The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocy028

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 15 references

Research works printed but not linked: 24

  1. Popper, 1963, Conjectures and Refutations
  2. Mathieu, 2009, Comparison of registered and published primary outcomes in randomized controlled trials, JAMA, 302, 977, 10.1001/jama.2009.1242 · DOI · Paper page
  3. Chalmers, 1990, Underreporting research is scientific misconduct, JAMA, 263, 1405, 10.1001/jama.1990.03440100121018 · DOI · Paper page
  4. Curtis, 2015, Replication – why we need to publish our findings, Pharmacol Res Perspect, 3, e00164, 10.1002/prp2.164 · DOI
  5. Anderson, 2016, Response to Comment on “Estimating the reproducibility of psychological science”, Science, 351, 1037, 10.1126/science.aad9163 · DOI · Paper page
  6. Peng, 2009, Reproducible research and Biostatistics, Biostatistics, 10, 405, 10.1093/biostatistics/kxp014 · DOI · Paper page
  7. Pashler, 2012, Is the replicability crisis overblown? Three arguments examined, Perspect Psychol Sci, 7, 531, 10.1177/1745691612463401 · DOI · Paper page
  8. Ammenwerth, 2016, Evidence-Based Health Informatics
  9. Madigan, 2013, Evaluating the impact of database heterogeneity on observational study results, Am J Epidemiol, 178, 645, 10.1093/aje/kwt010 · DOI · Paper page
  10. Han, 2005, Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system, Pediatrics, 116, 1506, 10.1542/peds.2005-1287 · DOI · Paper page
  11. Brunette, 2013, Implementation of computerized physician order entry for critical patients in an Academic Emergency Department is not associated with a change in mortality rate, Western J Emerg Med, 14, 114, 10.5811/westjem.2012.9.6601 · DOI
  12. Al-Dorzi, 2011, Impact of computerized physician order entry (CPOE) system on the outcome of critically ill adult patients: a before-after study, BMC Med Inform Decis Making, 11, 71, 10.1186/1472-6947-11-71 · DOI
  13. Longhurst, 2010, Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system, Pediatrics, 126, 14, 10.1542/peds.2009-3271 · DOI · Paper page
  14. Ammenwerth, 2006, Impact of CPOE on mortality rates – contradictory findings, important messages, Methods Archive, 45, 586
  15. Johnson, 2016, MIMIC-III, a freely accessible critical care database, Scientific Data, 3, 160035, 10.1038/sdata.2016.35 · DOI · Paper page
  16. Kelly, 2006, Replicating empirical research in behavioral ecology: how and why it should be done but rarely ever is, Quart Rev Biol, 81, 221, 10.1086/506236 · DOI · Paper page
  17. Palmer, 2000, Quasi-replication and the contract of error: lessons from sex ratios, heritabilities and fluctuating asymmetry, Ann Rev Ecol Syst, 31, 441, 10.1146/annurev.ecolsys.31.1.441 · DOI · Paper page
  18. Brender, 2013, STARE-HI – Statement on Reporting of Evaluation Studies in Health Informatics: Explanation and Elaboration, Appl Clin Inform, 4, 331, 10.4338/ACI-2013-04-RA-0024 · DOI · Paper page
  19. Dunn, 2016, Conflict of interest disclosure in biomedical research: a review of current practices, biases, and the role of public registries in improving transparency, Res Integrity Peer Rev, 1, 1, 10.1186/s41073-016-0006-7 · DOI · Paper page
  20. Almugbel, 2018, Reproducible Bioconductor workflows using browser-based interactive notebooks and containers, J Am Med Inform Assoc, 25, 4, 10.1093/jamia/ocx120 · DOI
  21. Coiera, 2017, Linking quality indicators to clinical trials: an automated approach, Int J Qual Healthcare, 10.1093/intqhc/mzx076 · DOI
  22. Cotterill, 2018, Getting messier with TIDieR: embracing context and complexity in intervention reporting, BMC Med Res Methodol, 18, 12, 10.1186/s12874-017-0461-y · DOI · Paper page
  23. Walker, 2015, Emerging trends in peer review—a survey, Front Neurosci, 9
  24. Moher, 2015, Four proposals to help improve the medical research literature, PLOS Med, 12, e1001864, 10.1371/journal.pmed.1001864 · DOI · Paper page

Cited by

See all 106

Metadata

Authors
Coiera, E., Ammenwerth, E., Georgiou, A., & Magrabi, F.
Venue
Journal of the American Medical Informatics Association
Published
2018-04-13
Volume
25
Issue
8
Pages
963-968
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W2797035815
Cited by
106
References
15
FWCI
10.91
Type
journal article
Language
en
Open access
hybrid
Funders
National Health and Medical Research Council (APP1134919)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Does health informatics have a replication crisis? https://app.cochat.ai/papers/10.1093/jamia/ocy028
The paper
Coiera, E., Ammenwerth, E., Georgiou, A., & Magrabi, F. (2018). Does health informatics have a replication crisis? Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocy028

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.