CoChat
Sign Up
Cited by 101Machine Learning in HealthcareElectronic Health Records SystemsArtificial Intelligence in Healthcare and Education

Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets

Open access
Chen, Jonathan H., Alagappan, Muthuraman, Goldstein, Mary K., Asch, Steven M., Altman, Russ B.·2017·International Journal of Medical Informatics
DOI 10.1016/j.ijmedinf.2017.03.006PMC PMC5459355
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVE: Determine how varying longitudinal historical training data can impact prediction of future clinical decisions. Estimate the "decay rate" of clinical data source relevance. MATERIALS AND METHODS: We trained a clinical order recommender system, analogous to Netflix or Amazon's "Customers who bought A also bought B..." product recommenders, based on a tertiary academic hospital's structured electronic health record data. We used this system to predict future (2013) admission orders based on different subsets of historical training data (2009 through 2012), relative to existing human-authored order sets. RESULTS: ). Algorithmically learned models from even the older (2009) data was still more effective than existing human-authored order sets (ROC AUC 0.81, precision 16% recall 35%). Training with more longitudinal data (2009-2012) was no better than using only the most recent (2012) data, unless applying a decaying weighting scheme with a "half-life" of data relevance about 4 months. DISCUSSION: Clinical practice patterns (automatically) learned from electronic health record data can vary substantially across years. Gold standards for clinical decision support are elusive moving targets, reinforcing the need for automated methods that can adapt to evolving information. CONCLUSIONS AND RELEVANCM: Prioritizing small amounts of recent data is more effective than using larger amounts of older data towards future clinical predictions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ijmedinf.2017.03.006

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 68

  1. [1]Richardson WC Berwick DM Bisgard JC Bristow LR Buck CR Cassel CK Chassin MR Coye MJ Detmer DE Grossman JH James B Lawrence DM Leape LL Levin A Robinson-Beale R Scherger JE Southam A Wakefield M Warden GL Crossing the Quality Chasm: A New Health System for the 21st Century Institute of Medicine, Committee on Quality of Health Care in America Committee on Quality of Health Care in America Washington DC 2001 10.1136/bmj.323.7322.1192 · DOI
  2. [2]Lauer MS Bonds D Eliminating the “expensive” adjective for clinical trials Am Heart J 167 2014 419 20 10.1016/j.ahj.2013.12.003 24655687 · DOI
  3. [3]Tricoci P Allen JM Kramer JM Califf RM Smith SC Scientific evidence underlying the ACC/AHA clinical practice guidelines JAMA 301 2009 831 41 10.1001/jama.2009.205 19244190 · DOI · Paper page
  4. [4]Durack DT The weight of medical knowledge N Engl J Med 298 1978 773 775 10.1097/00006534-197811000-00140 342963 · DOI
  5. [6]Kaushal R Shojania KG Bates DW Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review Arch Intern Med 163 2003 1409 1416 http://www.ncbi.nlm.nih.gov/pubmed/12824090 12824090
  6. [7]Overhage J Tierney W A randomized trial of “corollary orders” to prevent errors of omission J Am Med Informatics Assoc 4 1997 364 75 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=61254&tool=pmcentrez&rendertype=abstract accessed September 25, 2012 PMC61254 9292842
  7. [8]Tierney WM Overhage JM Takesue BY Harris LE Murray MD Vargo DL McDonald CJ Computerizing Guidelines to Improve Care and Patient Outcomes: The Example of Heart Failure J Am Med Informatics Assoc 2 1995 316 322 10.1136/jamia.1995.96073834 PMC116272 7496881 · DOI
  8. [9]Chen JH Fang DZ Tim Goodnough L Evans KH Lee Porter M Shieh L Why providers transfuse blood products outside recommended guidelines in spite of integrated electronic best practice alerts J Hosp Med 10 2015 1 7 10.1002/jhm.2236 25044190 · DOI
  9. [10]Finlay GD Rothman MJ Smith Ra Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system J Hosp Med 9 2014 116 9 10.1002/jhm.2132 24357519 PMC4321057 · DOI
  10. [11]Kaushal R Shojania KG Bates DW Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review Arch Intern Med 163 2003 1409 16 10.1001/archinte.163.12.1409 12824090 · DOI · Paper page
  11. [12]Cowden D Barbacioru C Kahwash E Saltz J Order sets utilization in a clinical order entry system AMIA Annu Symp Proc 2003 819 http://www.ncbi.nlm.nih.gov/pubmed/14728324 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1479990/pdf/amia2003_0819.pdf 14728324 PMC1479990
  12. [13]Payne TH Hoey PJ Nichol P Lovis C Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system J Am Med Inform Assoc 10 2003 322 9 10.1197/jamia.M1090 12668686 PMC181982 · DOI
  13. [14]Bobb AM Payne TH Gross PA Viewpoint: controversies surrounding use of order sets for clinical decision support in computerized provider order entry J Am Med Inform Assoc 14 2007 41 7 10.1197/jamia.M2184 17068352 PMC2215063 · DOI
  14. [15]Ballard DW Kim AS Huang J Park DK Kene MV Chettipally UK Iskin HR Hsu J Vinson DR Mark DG Reed ME Implementation of Computerized Physician Order Entry Is Associated With Increased Thrombolytic Administration for Emergency Department Patients With Acute Ischemic Stroke Ann Emerg Med 2015 1 10 10.1016/j.annemergmed.2015.07.018 PMC5111545 26362574 · DOI
  15. [16]Ballesca Ma LaGuardia JC Lee PC Hwang AM Park DK Gardner MN Turk BJ Kipnis P Escobar GJ An electronic order set for acute myocardial infarction is associated with improved patient outcomes through better adherence to clinical practice guidelines J Hosp Med 9 2014 155 161 10.1002/jhm.2149 24493376 · DOI
  16. [17]Micek ST Roubinian N Heuring T Bode M Williams J Harrison C Murphy T Prentice D Ruoff BE Kollef MH Before-after study of a standardized hospital order set for the management of septic shock Crit Care Med 34 2006 2707 13 10.1097/01.CCM.0000241151.25426.D7 16943733 · DOI
  17. [18]Jacobs BR Hart KW Rucker DW Reduction in Clinical Variance Using Targeted Design Changes in Computerized Provider Order Entry (CPOE) Order Sets: Impact on Hospitalized Children with Acute Asthma Exacerbation Appl Clin Inform 3 2012 52 63 10.4338/ACI-2011-01-RA-0002 23616900 PMC3613011 · DOI
  18. [19]Bates DW Kuperman GJ Wang S Gandhi T Kittler A Volk L Spurr C Khorasani R Tanasijevic M Middleton B Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality J Am Med Inform Assoc 10 2003 523 30 10.1197/jamia.M1370 12925543 PMC264429 · DOI · Paper page
  19. [20]Goldstein MK Hoffman BB Coleman RW Musen MA Tu SW Advani A Shankar R O’Connor M Implementing clinical practice guidelines while taking account of changing evidence: ATHENA DSS, an easily modifiable decision-support system for managing hypertension in primary care Proc AMIA Symp 2000 300 4 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2243943&tool=pmcentrez&rendertype=abstract accessed August 29, 2014 11079893 PMC2243943
  20. [21]Hripcsak G Albers DJ Perotte a Exploiting time in electronic health record correlations J Am Med Informatics Assoc 18 2011 i109 i115 10.1136/amiajnl-2011-000463 PMC3241180 22116643 · DOI
  21. [22]Ortiz E Rhodes S Morton SC Eccles MP Grimshaw JM Woolf SH Validity of the Agency for Healthcare Research and Quality Clinical Practice Guidelines 286 2001 1461 1467 11572738
  22. [23]ONC Health information technology: standards, implementation specifications, and certification criteria for electronic health record technology, 2014 edition; revisions to the permanent certification program for health information technology. Final rule Fed Regist 77 2012 54163 292 http://www.ncbi.nlm.nih.gov/pubmed/22946139 accessed July 30, 2014 22946139
  23. [24]Longhurst Ca Harrington Ra Shah NH A “Green Button” For Using Aggregate Patient Data At The Point Of Care Health Aff 33 2014 1229 1235 10.1377/hlthaff.2014.0099 25006150 · DOI
  24. [25]Frankovich J Longhurst CA Sutherland SM Evidence-based medicine in the EMR era N Engl J Med 365 2011 1758 9 10.1056/NEJMp1108726 22047518 · DOI
  25. [26]Smith M Saunders R Stuckhardt L McGinnis JM Best care at lower cost: the path to continuously learning health care in America Institute of Medicine, Committee on the Learning Health Care System in America Washington DC 2012 10.5860/CHOICE.51-3277 24901184 · DOI
  26. [28]de Lissovoy G Big data meets the electronic medical record: a commentary on “identifying patients at increased risk for unplanned readmission” Med Care 51 2013 759 60 10.1097/MLR.0b013e3182a67209 23942217 · DOI
  27. [29]Sittig DF Wright A Osheroff Ja Middleton B Teich JM Ash JS Campbell E Bates DW Grand challenges in clinical decision support J Biomed Inform 41 2008 387 92 10.1016/j.jbi.2007.09.003 18029232 PMC2660274 · DOI · Paper page
  28. [30]Doddi S Marathe a Ravi SS Torney DC Discovery of association rules in medical data Med Inform Internet Med 26 2001 25 33 http://www.ncbi.nlm.nih.gov/pubmed/11583406 11583406
  29. [31]Klann J Schadow G Downs SM A method to compute treatment suggestions from local order entry data AMIA Annu Symp Proc 2010 2010 387 91 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3041352&tool=pmcentrez&rendertype=abstract 21347006 PMC3041352
  30. [32]Klann J Schadow G McCoy JM A recommendation algorithm for automating corollary order generation AMIA Annu Symp Proc 2009 2009 333 7 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2815486&tool=pmcentrez&rendertype=abstract 20351875 PMC2815486
  31. [33]Wright A Sittig DF Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system AMIA Annu Symp Proc 2006 2006 819 823 http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=17238455 PMC1839652 17238455
  32. [34]Zhang Y Padman R Levin JE Paving the COWpath: data-driven design of pediatric order sets J Am Med Inform Assoc 21 2014 e304 e311 10.1136/amiajnl-2013-002316 24674844 PMC4173172 · DOI
  33. [35]Klann JG Szolovits P Downs SM Schadow G Decision support from local data: creating adaptive order menus from past clinician behavior J Biomed Inform 48 2014 84 93 10.1016/j.jbi.2013.12.005 24355978 PMC4004673 · DOI
  34. [36]Wright AP Wright AT McCoy AB Sittig DF The use of sequential pattern mining to predict next prescribed medications J Biomed Inform 53 2014 73 80 10.1016/j.jbi.2014.09.003 25236952 · DOI
  35. [37]Wright A Chen ES Maloney FL An automated technique for identifying associations between medications, laboratory results and problems J Biomed Inform 43 2010 891 901 10.1016/j.jbi.2010.09.009 20884377 · DOI
  36. [38]Chen JH Altman RB Mining for clinical expertise in (undocumented) order sets to power an order suggestion system AMIA Jt Summits Transl Sci Proc AMIA Summit Transl Sci 2013 2013 34 8 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3845792&tool=pmcentrez&rendertype=abstract accessed October 27, 2014 PMC3845792 24303232
  37. [39]Chen JH Podchiyska T Altman RB OrderRex: clinical order decision support and outcome predictions by data-mining electronic medical records J Am Med Informatics Assoc 23 2016 339 348 10.1093/jamia/ocv091 PMC5009921 26198303 · DOI
  38. [40]Linden G Smith B York J Amazon.com recommendations: item-to-item collaborative filtering IEEE Internet Comput 7 2003 76 80 10.1109/MIC.2003.1167344 · DOI · Paper page
  39. [41]Moskovitch R Choi H Hripcsak G Tatonetti N Prognosis of Clinical Outcomes with Temporal Patterns and Experiences with One Class Feature Selection IEEE/ACM Trans Comput Biol Bioinforma 5963 2016 1 1 10.1109/TCBB.2016.2591539 PMC5486920 27429447 · DOI
  40. [42]Coiera E Wang Y Magrabi F Concha OP Gallego B Runciman W Predicting the cumulative risk of death during hospitalization by modeling weekend, weekday and diurnal mortality risks BMC Health Serv Res 14 2014 226 10.1186/1472-6963-14-226 24886152 PMC4053268 · DOI
  41. [43]Gallego B Magrabi F Concha OP Wang Y Coiera E Insights into temporal patterns of hospital patient safety from routinely collected electronic data Heal Inf Sci Syst 3 2015 S2 10.1186/2047-2501-3-S1-S2 PMC4383060 25870757 · DOI
  42. [44]Lemeshow S Le Gall JR Modeling the severity of illness of ICU patients. A systems update JAMA 272 1994 1049 55 http://www.ncbi.nlm.nih.gov/pubmed/8089888 accessed July 31, 2014 8089888
  43. [45]Kansagara D Risk Prediction Models for Hospital Readmission<subtitle>A Systematic Review</subtitle> JAMA J Am Med Assoc 306 2011 1688 10.1001/jama.2011.1515 PMC3603349 22009101 · DOI
  44. [46]Henry KE Hager DN Pronovost PJ Saria S A targeted real-time early warning score (TREWScore) for septic shock Sci Transl Med 7 2015 299ra122 10.1126/scitranslmed.aab3719 26246167 · DOI · Paper page
  45. [47]Goldstein BA Navar AM Pencina MJ Ioannidis JP Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review J Am Med Informatics Assoc 2016 2016 ocw042 10.1093/jamia/ocw042 PMC5201180 27189013 · DOI · Paper page
  46. [49]Lowe HJ Ferris Ta Hernandez PM Weber SC STRIDE--An integrated standards-based translational research informatics platform AMIA Annu Symp Proc 2009 2009 391 5 20351886 PMC2815452
  47. [50]Hernandez P Podchiyska T Weber S Ferris T Lowe H Automated mapping of pharmacy orders from two electronic health record systems to RxNorm within the STRIDE clinical data warehouse AMIA Annu Symp Proc 2009 2009 244 8 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2815471&tool=pmcentrez&rendertype=abstract 20351858 PMC2815471
  48. [51]Wright A Bates DW Distribution of Problems, Medications and Lab Results in Electronic Health Records: The Pareto Principle at Work Appl Clin Inform 1 2010 32 37 10.4338/ACI-2009-12-RA-0023 21991298 PMC3189502 · DOI
  49. [52]Chen JH Altman RB Automated physician order recommendations and outcome predictions by data-mining electronic medical records AMIA Jt Summits Transl Sci Proc AMIA Summit Transl Sci 2014 2014 206 10 http://www.ncbi.nlm.nih.gov/pubmed/25717414 PMC4333710 25717414
  50. [53]Finlayson SG Lependu P Shah NH Building the graph of medicine from millions of clinical narratives Sci Data 1 2014 140032 10.1038/sdata.2014.32 25977789 PMC4322575 · DOI
  51. [54]Manrai AK Bhatia G Strymish J Kohane IS Jain SH Medicine’s uncomfortable relationship with math: calculating positive predictive value JAMA Intern Med 174 2014 991 3 10.1001/jamainternmed.2014.1059 24756486 PMC4955674 · DOI · Paper page
  52. [55]Kendall MG A New Measure of Rank Correlation Biometrika 30 1938 81 93 10.2307/2332226 · DOI
  53. [56]Webber W Moffat A Zobel J A similarity measure for indefinite rankings ACM Trans Inf Syst 28 2010 1 38 10.1145/1852102.1852106 · DOI
  54. [58]Chen JH Goldstein MK Asch SM Mackey L Altman RB Predicting inpatient clinical order patterns with probabilistic topic models vs conventional order sets J Am Med Informatics Assoc 2016 ocw136 10.1093/jamia/ocw136 PMC5391730 27655861 · DOI
  55. [60]Leskovec J Rajaraman A Ullman JD Mining Data Streams Min Massive Datasets 2014 131 162 10.4018/978-1-60566-010-3.ch194 · DOI
  56. [61]Kerr JR Swine influenza J Clin Pathol 62 2009 577 578 10.1136/jcp.2009.067710 19433408 · DOI
  57. [62]Who WHO Guidelines for Pharmacological Management of Pandemic Influenza A(H1N1) 2009 and other Influenza Viruses, WHO Guidel Pharmacol Manag Pandemic Influ A(H1N1) 2009 Other Influ Viruses 2010 1 32 http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:WHO,2010#7 23741777
  58. [63]Mandell La Wunderink RG Anzueto A Bartlett JG Campbell GD Dean NC Dowell SF File TM Musher DM Niederman MS Torres A Whitney CG Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults Clin Infect Dis 44 Suppl 2 2007 S27 72 10.1086/511159 17278083 PMC7107997 · DOI
  59. [64]Focaccia R Gomes Da Conceicao OJ Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia Am J Respir Crit Care Med 171 2005 388 416 10.1164/rccm.200405-644ST 15699079 · DOI
  60. [65]Teirstein P Boarded to Death—Why Maintenance of Certification Is Bad for Doctors and Patients N Engl J Med 372 2015 106 8 10.1056/NEJMp1407422 25564895 · DOI
  61. [66]Irons MB Nora LM Maintenance of Certification 2.0—Strong Start, Continued Evolution N Engl J Med 372 2015 104 106 10.1056/NEJMp1409923 25564894 · DOI
  62. [67]Rhee C Gohil S Klompas M Regulatory mandates for sepsis care--reasons for caution N Engl J Med 370 2014 1673 1676 10.1056/NEJMp1400276 24738642 PMC4718398 · DOI
  63. [68]Wang H Zhang C Zhang J Kong L Zhu H Yu J The prognosis analysis of different metastasis pattern in patients with different breast cancer subtypes: a SEER based study 2016 PMC5432264 28038448
  64. [69]Bhatia HK Singh H Grewal N Natt NK Sofosbuvir: A novel treatment option for chronic hepatitis C infection J Pharmacol Pharmacother 5 2014 278 284 10.4103/0976-500X.142464 25422576 PMC4231565 · DOI
  65. [71]Austen-Smith D Banks JS Information Aggregation, Rationality, and the Condorcet Jury Theorem Am Polit Sci Rev 90 1996 34 45 10.2307/2082796 · DOI
  66. [72]Chen JH Altman RB Data-Mining Electronic Medical Records for Clinical Order Recommendations: Wisdom of the Crowd or Tyranny of the Mob? AMIA Jt Summits Transl Sci Proc AMIA Summit Transl Sci 2015 2015 435 9 http://www.ncbi.nlm.nih.gov/pubmed/26306281 PMC4525236 26306281
  67. [73]Brin S Page L The anatomy of a large-scale hypertextual Web search engine BT - Computer Networks and ISDN Systems 30 1998 107 117 10.1016/S0169-7552(98)00110-X · DOI · Paper page
  68. [74]Excell D Bayesian inference - The future of online fraud protection Comput Fraud Secur 2012 2012 8 11 10.1016/S1361-3723(12)70018-0 · DOI

Cited by

Sorted by citations

Authors
Chen, J. H., Alagappan, M., Goldstein, M. K., Asch, S. M., & Altman, R. B.
Affiliations
  • Stanford Medicine (US)
  • Beth Israel Deaconess Medical Center (US)
  • VA Palo Alto Health Care System (US)
Corresponding author
Chen, Jonathan H.
ORCID
Venue
International Journal of Medical Informatics
Publisher
Elsevier BV
Published
2017-06
Volume
102
Pages
71-79
ISSN
1386-5056 (ISSN-L)
DOI
10.1016/j.ijmedinf.2017.03.006
PMC
PMC5459355
OpenAlex
W2595962374
Cited by
101
References
1
Citations by year
2026: 5 · 2025: 7 · 2024: 9 · 2023: 10 · 2022: 16 · 2021: 14 · 2020: 13 · 2019: 10 · 2018: 16 · 2017: 1
FWCI
6.52
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Medical Informatics (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institute of General Medical Sciences (R01GM102365, R24GM061374)
  • National Institute of Environmental Health Sciences (K01ES026837)
  • U.S. National Library of Medicine (R01LM005652)
  • National Center for Research Resources (UL1RR025744)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
clinical decision support, recommender systems, longitudinal electronic health records, clinical practice patterns, clinical prediction
MeSH terms
Patient Care, Hospitalization, Humans, Inpatients, Practice Patterns, Physicians', Information Storage and Retrieval, Decision Support Systems, Clinical, Medical Order Entry Systems, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review