The MIMIC Code Repository: enabling reproducibility in critical care research
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Objective: Lack of reproducibility in medical studies is a barrier to the generation of a robust knowledge base to support clinical decision-making. In this paper we outline the Medical Information Mart for Intensive Care (MIMIC) Code Repository, a centralized code base for generating reproducible studies on an openly available critical care dataset. Materials and Methods: Code is provided to load the data into a relational structure, create extractions of the data, and reproduce entire analysis plans including research studies. Results: Concepts extracted include severity of illness scores, comorbid status, administrative definitions of sepsis, physiologic criteria for sepsis, organ failure scores, treatment administration, and more. Executable documents are used for tutorials and reproduce published studies end-to-end, providing a template for future researchers to replicate. The repository’s issue tracker enables community discussion about the data and concepts, allowing users to collaboratively improve the resource. Discussion: The centralized repository provides a platform for users of the data to interact directly with the data generators, facilitating greater understanding of the data. It also provides a location for the community to collaborate on necessary concepts for research progress and share them with a larger audience. Consistent application of the same code for underlying concepts is a key step in ensuring that research studies on the MIMIC database are comparable and reproducible. Conclusion: By providing open source code alongside the freely accessible MIMIC-III database, we enable end-to-end reproducible analysis of electronic health records.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The MIMIC Code Repository was originally created for MIMIC-III to enable researchers to share and reuse code24.”
From MIMIC-IV, a freely accessible electronic health record dataset (2023) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250 · cited 1×
“21 The precise onset of sepsis is not typically documented in the EHR, and is, in fact, a difficult item to capture clinically.”
BackgroundVincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., & Thijs, L. G. (1996) · Intensive Care Medicine · doi:10.1007/bf01709751 · Cited by 11,674 · cited 1×
“Some scores assess multiple organ systems: the SOFA score 12 and LODS 13 both assess 6 organ systems for failure.”
BackgroundCoding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data
Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., Saunders, L. D., Beck, C. A., Feasby, T. E., & Ghali, W. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000182534.19832.83 · Cited by 10,669 · cited 1×
“27 Finally, Quan et al. 28 proposed an enhanced ICD-9 coding methodology based on examining inconsistencies among previous definitions.”
MethodComorbidity measures for use with administrative data.
Elixhauser, A., Steiner, C., Harris, D. R., & Coffey, R. M. (1998) · Medical Care · doi:10.1097/00005650-199801000-00004 · Cited by 9,921 · cited 4×
“Van Walraven et al. 29 later aggregated comorbidities codified by Elixhauser et al. 26 into a single point score for in-hospital mortality prediction, which is also available in the repository.”
BackgroundMethodInfluentialAngus, D. C., Linde-Zwirble, W. T., Lidicker, J., Clermont, G., Carcillo, J., & Pinsky, M. R. (2001) · Critical Care Medicine · doi:10.1097/00003246-200107000-00002 · Cited by 8,503 · cited 3×
“The criteria as proposed by Angus et al. 23 were validated in a later study by Iwashyna et al. 25 These 3 criteria – explicit coding, those proposed by Angus et al., 23 and those proposed by Martin et al. 24 – are available in the repository.”
MethodMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“3,4 The database comprises detailed clinical information regarding > 60000 stays in ICUs at the Beth Israel Deaconess Medical Center in Boston, Massachusetts, collected as part of routine clinical care.”
BackgroundSelf-citationLe Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773
Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury
Mehta, R. L., Kellum, J. A., Shah, S. V., Molitoris, B. A., Ronco, C., Warnock, D. G., Levin, A., & the Acute Kidney Injury Network (2007) · Critical Care · doi:10.1186/cc5713 · Cited by 7,048 · cited 1×
“Examples include the Model for End-stage Liver Disease, 14 the Risk/Injury/Failure/Loss/End-stage renal disease criteria, 15 the Acute Kidney Injury Network classification, 16 and the Kidney Disease: Improving Global Outcomes criteria.”
BackgroundThe epidemiology of sepsis in the United States from 1979 through 2000
Martin, G. S., Mannino, D. M., Eaton, S., & Moss, M. (2003) · New England Journal of Medicine · doi:10.1056/nejmoa022139 · Cited by 5,841 · cited 4×
“The criteria as proposed by Angus et al. 23 were validated in a later study by Iwashyna et al. 25 These 3 criteria – explicit coding, those proposed by Angus et al., 23 and those proposed by Martin et al. 24 – are available in the repository.”
BackgroundInfluential1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 1×
“Concerns about the reproducibility of results in science are becoming increasingly prominent in both scientific and mainstream literature.(1) Some commentators have gone so far as to call the current state a crisis, citing causes such as pressure to publish positive results, the cost of replicating studies such as dou…”
BackgroundIPython: A System for Interactive Scientific Computing
Perez, F., & Granger, B. E. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.53 · Cited by 4,009 · cited 1×
“30,31 We have found executable documents particularly valuable for research in cross-disciplinary fields such as health care, because they facilitate collaboration between data analysts and domain experts.”
BackgroundSeymour, C. W., Liu, V. X., Iwashyna, T. J., Brunkhorst, F. M., Rea, T. D., Scherag, A., Rubenfeld, G., Kahn, J. M., Shankar-Hari, M., Singer, M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0288 · Cited by 3,939 · cited 1×
“In their quantitative evaluation of septic patients, Sey-mour et al. 22 first identified patients suspected of having infection by cross-referencing antibiotic use with requests for a microbiology assessment.”
BackgroundTargeting the KEAP1-NRF2 system to prevent kidney disease progression
Nezu, M., Suzuki, N., & Yamamoto, M. (2017) · American Journal of Nephrology · doi:10.1159/000475890 · Cited by 3,447
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., Bastos, P. G., Sirio, C. A., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · Chest · doi:10.1378/chest.100.6.1619 · Cited by 3,439
Jupyter Notebooks – a publishing format for reproducible computational workflows
Thomas, K., Benjamin, R.-K., Fernando, P., Brian, G., Matthias, B., Jonathan, F., Kyle, K., Jessica, H., Jason, G., Sylvain, C., et al. (2016) · Positioning and Power in Academic Publishing: Players, Agents and Agendas · doi:10.3233/978-1-61499-649-1-87 · Cited by 3,063 · cited 1×
“30,31 We have found executable documents particularly valuable for research in cross-disciplinary fields such as health care, because they facilitate collaboration between data analysts and domain experts.”
BackgroundThe APACHE III Prognostic System
Teres, D., & Lemeshow, S. (1992) · Chest · doi:10.1378/chest.102.6.1919b · Cited by 2,880
Model for end-stage liver disease (MELD) and allocation of donor livers
Wiesner, R., Edwards, E., Freeman, R., Harper, A., Kim, R., Kamath, P., Kremers, W., Lake, J., Howard, T., Merion, R. M., et al. (2003) · Gastroenterology · doi:10.1053/gast.2003.50016 · Cited by 2,528
MacLeod, A. M. (2009) · Kidney International · Cited by 2,219
van Walraven, C., Austin, P. C., Jennings, A., Quan, H., & Forster, A. J. (2009) · Medical Care · doi:10.1097/mlr.0b013e31819432e5 · Cited by 2,200
Eckardt, K.-U., & Kasiske, B. L. (2009) · Kidney International · doi:10.1038/ki.2009.188 · Cited by 1,167
Le Gall, J.-R. (1996) · JAMA · doi:10.1001/jama.1996.03540100046027 · Cited by 735
Best Practices for Scientific Computing
Wilson, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., et al. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001745 · Cited by 723
The healthcare cost and utilization project: an overview.
Steiner, C., Elixhauser, A., & Schnaier, J. A. (2002) · PubMed · Cited by 568
Gall, J. R. L. (1996) · JAMA · doi:10.1001/jama.276.10.802 · Cited by 543
Longo, D. L., & Drazen, J. M. (2016) · New England Journal of Medicine · doi:10.1056/nejme1516564 · Cited by 419
Iwashyna, T. J., Odden, A., Rohde, J., Bonham, C., Kuhn, L., Malani, P., Chen, L., & Flanders, S. (2014) · Medical Care · doi:10.1097/mlr.0b013e318268ac86 · Cited by 358 · cited 2×
“The criteria as proposed by Angus et al. 23 were validated in a later study by Iwashyna et al. 25 These 3 criteria – explicit coding, those proposed by Angus et al., 23 and those proposed by Martin et al. 24 – are available in the repository.”
MethodJohnson, A. E. W., Kramer, A. A., & Clifford, G. D. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31828a24fe · Cited by 297
Self-citationTen Simple Rules for Taking Advantage of Git and GitHub
Perez-Riverol, Y., Gatto, L., Wang, R., Sachsenberg, T., Uszkoreit, J., Leprevost, F. D. V., Fufezan, C., Ternent, T., Eglen, S. J., Katz, D. S., et al. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004947 · Cited by 230
Self-citationThe first international consensus conference on continuous renal replacement therapy
Kellum, J. A., Mehta, R. L., Angus, D. C., Palevsky, P., & Ronco, C. (2002) · Kidney International · doi:10.1046/j.1523-1755.2002.00613.x · Cited by 205
Data Authorship as an Incentive to Data Sharing
Bierer, B. E., Crosas, M., & Pierce, H. H. (2017) · New England Journal of Medicine · doi:10.1056/nejmsb1616595 · Cited by 169
Johnson, A., Pollard, T., & Mark, R. (2015) · PhysioNet · doi:10.13026/c2xw26 · Cited by 165
Self-citationUshering in a New Era of Open Science Through Data Sharing: The Wall Must Come Down
Ross, J. S., & Krumholz, H. M. (2013) · JAMA · doi:10.1001/jama.2013.1299 · Cited by 118 · cited 1×
“quite well described.(35,36) To achieve full transparency, researchers must be able to provide both the data used for analysis and the code used to process it.”
BackgroundAPACHE 1978-2001: The Development of a Quality Assurance System Based on Prognosis
Knaus, W. A. (2002) · Archives of Surgery · doi:10.1001/archsurg.137.1.37 · Cited by 97 · cited 1×
“7 The principal aim of these scores is to risk-adjust patient populations for benchmarking and research purposes, such as comparing cohorts in clinical trials and observational studies.”
BackgroundPredicting intervention onset in the ICU with switching state space models.
Ghassemi, M., Wu, M., Hughes, M. C., Szolovits, P., & Doshi‐Velez, F. (2017) · PubMed Central · Cited by 50
Hsu, D. J., Feng, M., Kothari, R. P., Zhou, H., Chen, K. P., & Celi, L. A. (2015) · CHEST Journal · doi:10.1378/chest.15-0516 · Cited by 47
Self-citationTen Simple Rules for Taking Advantage of git and GitHubPreprint
Pérez‐Riverol, Y., Gatto, L., Wang, R., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/048744 · Cited by 31 · cited 1×
“Source code control allows for transparency both in the authorship of the code and in the nature of any changes.(34)”
BackgroundThe power of capturing and using information at the point of care
Gruber, W. H., Powell, A. C., & Torous, J. B. (2016) · Healthcare · doi:10.1016/j.hjdsi.2016.09.004 · Cited by 9 · cited 1×
“The Health Information Technology for Economic and Clinical Health Act has catalyzed the transition of hospitals and care institutions from paper-based to electronic-based systems.(2) Vast quantities of digital data are now routinely collected by modern hospital monitoring systems, even more so in intensive care units…”
BackgroundMIT-LCP/mimic-code: MIMIC Code v1.4.1
Johnson, A. E. W., Pollard, T., a-chahin, Blundell, J., Gow, B. J., erinhong, Schubert, M., Dang, K., Paris, N., JackieMe, et al. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.821871 · Cited by 9
Self-citationMIT-LCP/mimic-code: MIMIC-III v1.4
Pollard, T., Johnson, A., Blundell, J., et al. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.821872 · Cited by 3
Self-citation
Cited by
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401
“The MIMIC Code Repository was originally created for MIMIC-III to enable researchers to share and reuse code 24 .”
MethodSelf-citationMIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
“We believe that publicly accessible code using publicly available data will accelerate research in the field and ensure reproducibility of future studies 26,27 .”
MethodSelf-citationSerpa Neto, A., Deliberato, R. O., Johnson, A. E. W., Bos, L. D., Amorim, P., Pereira, S. M., Cazati, D. C., Cordioli, R. L., Correa, T. D., Pollard, T. J., et al. (2018) · Intensive Care Medicine · doi:10.1007/s00134-018-5375-6 · Cited by 613
Self-citationOpen science saves lives: lessons from the COVID-19 pandemic
Besançon, L., Peiffer-Smadja, N., Segalas, C., Jiang, H., Masuzzo, P., Smout, C., Billy, E., Deforet, M., & Leyrat, C. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01304-y · Cited by 249
“An open code repository has been created [90] and many scientific journals are now asking researchers to publish the source code alongside their findings.”
BackgroundTransthoracic echocardiography and mortality in sepsis: analysis of the MIMIC-III database
Feng, M., McSparron, J. I., Kien, D. T., Stone, D. J., Roberts, D. H., Schwartzstein, R. M., Vieillard-Baron, A., & Celi, L. A. (2018) · Intensive Care Medicine · doi:10.1007/s00134-018-5208-7 · Cited by 239
“Discussions around reliability of specific data elements are captured in a community white board to guide future investigators who may be interested in exploring similar concepts [21].”
Self-citation
Metadata
- Authors
- Johnson, A. E. W., Stone, D. J., Celi, L. A., & Pollard, T. J.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Virginia (US)
- Beth Israel Deaconess Medical Center (US)
- Corresponding author
- Johnson, Alistair E W
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-09-27
- Volume
- 25
- Issue
- 1
- Pages
- 32-39
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6381763
- OpenAlex
- W2757504960
- Cited by
- 442
- References
- 36
- Citations by year
- 2026: 44 · 2025: 67 · 2024: 56 · 2023: 72 · 2022: 57 · 2021: 62 · 2020: 39 · 2019: 24 · 2018: 17 · 2017: 3
- FWCI
- 17.77
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (NIH-R01-EB017205, NIH-R01-EB001659)
- Topics
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- reproducibility, critical care research, intensive care unit data, open-source code, electronic health records, sepsis definitions, disease severity index, organ failure scores, data extraction, research data sharing, community collaboration
- MeSH terms
- Critical Care, Humans, ROC Curve, Severity of Illness Index, Therapeutics, Reproducibility of Results, Comorbidity, Databases, Factual, Sepsis, Access to Information, Biomedical Research, Data Mining, Electronic Health Records, Organ Dysfunction Scores
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.