The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research
Open accessAbstract
The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, a major concern is that data not collected systematically for research will be poorer quality, which could have negative impacts on findings generated from these data.24–27”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 2,127
The “Meaningful Use” Regulation for Electronic Health Records
Blumenthal, D., & Tavenner, M. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006114 · Cited by 2,059
A review of uses of health care utilization databases for epidemiologic research on therapeutics
Schneeweiß, S., & Avorn, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.012 · Cited by 1,290
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 1×
“A review of the literature of studies evaluating the data quality of EHRs for clinical research identified 5 dimensions of data quality assessed: completeness, correctness, concordance, plausibility, and currency.(26) The authors identified many studies with a wide variety of techniques to assess these dimensions and,…”
Measuring Diagnoses: ICD Code Accuracy
O’Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,156
Assessing Quality Using Administrative Data
Iezzoni, L. I. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00048 · Cited by 981
Next-generation phenotyping of electronic health records
Hripcsak, G., & Albers, D. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001145 · Cited by 795
Self-citationSafran, C., Bloomrosen, M., Hammond, W. E., Labkoff, S., Markel-Fox, S., Tang, P., & Detmer, D. (2007) · JAMIA Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2273 · Cited by 714
the eMERGE Team, McCarty, C. A., Chisholm, R. L., Chute, C. G., Kullo, I. J., Jarvik, G. P., Larson, E. B., Li, R., Masys, D. R., Ritchie, M. D., et al. (2011) · BMC Medical Genomics · doi:10.1186/1755-8794-4-13 · Cited by 701
Desiderata for Controlled Medical Vocabularies in the Twenty-First Century
Cimino, J. J. (1998) · Methods of Information in Medicine · doi:10.1055/s-0038-1634558 · Cited by 656
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
“Adoption of unified data formats, such as Fast Healthcare Interoperability Resources [84], offer the potential for better aggregation of data, although improved interoperability does not necessarily fix the problem of inconsistent semantic coding in EHR data [85].”
Ahmed, Z., Mohamed, K., Zeeshan, S., & Dong, X. (2020) · Database · doi:10.1093/database/baaa010 · Cited by 1,035
“Use of EHR in comparative effectiveness research (46) Reporting caveats in existing healthcare systems.”
BackgroundMethodGoldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
“As others have noted, EHR data come with many challenges.(4) EHRs include all patients that touch a medical system, primarily capture data only when patients are ill, and collect metrics that clinicians deem to be necessary at each clinic visit.”
BackgroundAutomated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834
Electronic health records to facilitate clinical research
Cowie, M. R., Blomster, J. I., Curtis, L. H., Duclaux, S., Ford, I., Fritz, F., Goldman, S., Janmohamed, S., Kreuzer, J., Leenay, M., et al. (2017) · Clinical Research in Cardiology · doi:10.1007/s00392-016-1025-6 · Cited by 651
“Excessive or busy workloads may also contribute to errors in clinician data entry [43].”
BackgroundHarnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
“The general limits of using health-care records to conduct research are widely discussed 109 : one challenge is unobserved patient outcomes, which can be handled for time- to-event analysis but requires excluding patients for categorical outcomes.”
BackgroundLaunching PCORnet, a national patient-centered clinical research network
Fleurence, R., Curtis, L. H., Califf, R. M., Platt, R., Selby, J. V., & Brown, J. S. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002747 · Cited by 654
“17 Second, in order to achieve multi-institutional querying, PCORnet will have to deal with data harmonization.”
BackgroundKahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1244 · Cited by 697
“24–27 Transaction-oriented systems rarely include prespecified, unambiguous data definitions, and uniform (unbiased) data collection procedures.”
BackgroundThe National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment
Haendel, M. A., Chute, C. G., Bennett, T. D., Eichmann, D. A., Guinney, J., Kibbe, W. A., Payne, P. R. O., Pfaff, E. R., Robinson, P. N., Saltz, J. H., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa196 · Cited by 553
“125 EHR data captured in real time are often wrong (eg, incorrect diagnosis) or may have originated from a different patient.”
BackgroundSelf-citationBig Data and Large Sample Size: A Cautionary Note on the Potential for Bias
Kaplan, R. M., Chambers, D., & Glasgow, R. E. (2014) · Clinical and Translational Science · doi:10.1111/cts.12178 · Cited by 442
Personalized Medicine and the Power of Electronic Health Records
Abul-Husn, N. S., & Kenny, E. E. (2019) · Cell · doi:10.1016/j.cell.2019.02.039 · Cited by 355
“The models in which health system data may be used to support personalized medicine research are varied, and there are special ethical and scientific (Hersh et al., 2013; Sutherland et al., 2016; Wolford et al., 2018) considerations to take into account when linking genomic data with EHRs.”
BackgroundMethodWei, W., Bastarache, L. A., Carroll, R. J., Marlo, J. E., Osterman, T. J., Gamazon, E. R., Cox, N. J., Roden, D. M., & Denny, J. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175508 · Cited by 364
Real-world Data for Clinical Evidence Generation in Oncology
Khozin, S., Blumenthal, G. M., & Pazdur, R. (2017) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djx187 · Cited by 361
Natural Language Processing in Oncology: A Review
Yim, W.-W., Yetisgen, M., Harris, W. P., & Kwan, S. W. (2016) · JAMA Oncology · doi:10.1001/jamaoncol.2016.0213 · Cited by 298
Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse
Verheij, R., Ćurčin, V., Delaney, B., & McGilchrist, M. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9134 · Cited by 241
“purpose may not be suitable for another and that there are serious issues to be considered in the use or reuse of EHR data [24-28].”
BackgroundChakraborty, C., Bhattacharya, M., Pal, S., & Lee, S. (2023) · Current Research in Biotechnology · doi:10.1016/j.crbiot.2023.100164 · Cited by 311
Beckmann, J. S., & Lew, D. (2016) · Genome Medicine · doi:10.1186/s13073-016-0388-7 · Cited by 267
“However, EHRs are not only heterogeneous but they can have incomplete, inconsistent, or inaccurate data, as well as additional limitations (reviewed elsewhere [86 – 88]) that further hinder their applicability, scalability, and semantic and technical interoperability.”
BackgroundExtracting research-quality phenotypes from electronic health records to support precision medicine
Wei, W., & Denny, J. C. (2015) · Genome Medicine · doi:10.1186/s13073-015-0166-y · Cited by 251
“EHRs derive from selected populations and their data contain biases [34,45,106]; in particular, they are biased toward sick individuals.”
BackgroundSteele, A. J., Cakiroglu, S. A., Shah, A. D., Denaxas, S., Hemingway, H., & Luscombe, N. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0202344 · Cited by 246
Fundamentals of Clinical Data Science
Kubben, P., Dumontier, M., & Dekker, A. L. A. J. (2018) · Springer eBooks · doi:10.1007/978-3-319-99713-1 · Cited by 258
Bernert, R. A., Hilberg, A. M., Melia, R., Kim, J. P., Shah, N. H., & Abnousi, F. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17165929 · Cited by 243
“Hersh et al. [139] raised concerns regarding biases due to EMR data being collected only at hospital visits, incomplete records or missing data, and other considerations relevant to accurate coding that emphasize advanced statistical methods be used for correction.”
MethodCorey, K. M., Kashyap, S., Lorenzi, E., Lagoo-Deenadayalan, S. A., Heller, K., Whalen, K., Balu, S., Heflin, M. T., McDonald, S. R., Swaminathan, M., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002701 · Cited by 204
Clinical Data Reuse or Secondary Use: Current Status and Potential Future Progress
Meystre, S. M., Lovis, C., Bürkle, T., Tognola, G., Budrionis, A., & Lehmann, C. U. (2017) · Yearbook of Medical Informatics · doi:10.15265/iy-2017-007 · Cited by 204
“A similar European initiative proposed recommendations for the trustworthy reuse of health data [52], and Hersh and colleagues published recommendations [53] and caveats for clinical data reuse in comparative effectiveness research [54].”
MethodWill Generative Artificial Intelligence Deliver on Its Promise in Health Care?
Wachter, R. M., & Brynjolfsson, E. (2023) · JAMA · doi:10.1001/jama.2023.25054 · Cited by 206
Goldstein, B. A., Bhavsar, N. A., Phelan, M., & Pencina, M. J. (2016) · American Journal of Epidemiology · doi:10.1093/aje/kww112 · Cited by 179
Machine Learning in Healthcare
Callahan, A., & Shah, N. H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-809523-2.00019-4 · Cited by 178
Use of Latent Class Analysis and k-Means Clustering to Identify Complex Patient Profiles
Grant, R. W., McCloskey, J. K., Hatfield, M. K., Uratsu, C. S., Ralston, J. D., Bayliss, E. A., & Kennedy, C. J. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.29068 · Cited by 191
Bias in pharmacoepidemiologic studies using secondary health care databases: a scoping review
Prada-Ramallal, G., Takkouche, B., & Figueiras, A. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0695-y · Cited by 161
A Data Quality Assessment Guideline for Electronic Health Record Data Reuse
Weiskopf, N. G., Bakken, S., Hripcsak, G., & Weng, C. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.218 · Cited by 152
Self-citationvan de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135
Rusanov, A., Weiskopf, N. G., Wang, S., & Weng, C. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-51 · Cited by 137
“There are, however, potential caveats to the secondary use of EHR data [10].”
BackgroundChallenges in and Opportunities for Electronic Health Record-Based Data Analysis and Interpretation
Kim, M. K., Rouphael, C., McMichael, J. P., Welch, N., & Dasarathy, S. (2023) · Gut and Liver · doi:10.5009/gnl230272 · Cited by 142
“25”
Transparent Reporting of Data Quality in Distributed Data Networks
Kahn, M. G., Brown, J. S., Chun, A. T., Davidson, B. N., Meeker, D., Ryan, P. B., Schilling, L. M., Weiskopf, N. G., Williams, A. E., & Zozus, M. N. (2015) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1052 · Cited by 140
Haneuse, S., & Daniels, M. J. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1203 · Cited by 118
Using Electronic Health Records To Generate Phenotypes For Research
Pendergrass, S. A., & Crawford, D. C. (2019) · Current Protocols in Human Genetics · doi:10.1002/cphg.80 · Cited by 120
“These data are prone to biases and error (Hersh et al., 2013).”
BackgroundKohane, I. S., Aronow, B. J., Avillach, P., Beaulieu-Jones, B. K., Bellazzi, R., Bradford, R. L., Brat, G. A., Cannataro, M., Cimino, J. J., García-Barrio, N., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/22219 · Cited by 113
“Finally, we did not focus on the specific limitations of EHR-derived studies, which have been amply documented [7,8], or on the methods to minimize the impact of these limitations,”
BackgroundSelf-citationRodríguez, F., Chung, S., Blum, M. R., Coulet, A., Basu, S., & Palaniappan, L. P. (2019) · Journal of the American Heart Association · doi:10.1161/jaha.118.011874 · Cited by 107
Population segmentation based on healthcare needs: a systematic review
Chong, J. L., Lim, K. K., & Matchar, D. B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1105-6 · Cited by 106
Employing Computers for the Recruitment into Clinical Trials: A Comprehensive Systematic Review
Köpcke, F., & Prokosch, H. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3446 · Cited by 105
Sick patients have more data: the non-random completeness of electronic health records.
Weiskopf, N. G., Rusanov, A., & Weng, C. (2013) · Europe PMC (PubMed Central) · Cited by 98
Giannoula, A., Gutiérrez‐Sacristán, A., Bravo, Á., Sanz, F., & Furlong, L. I. (2018) · Scientific Reports · doi:10.1038/s41598-018-22578-1 · Cited by 98
Biases introduced by filtering electronic health records for patients with “complete data.”
Weber, G. M., Adams, W. G., Bernstam, E. V., Bickel, J. P., Fox, K. P., Marsolo, K., Raghavan, V. A., Turchin, A., Zhou, X., Murphy, S. N., et al. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx071 · Cited by 93
Self-citationAnderson, A., Kerr, W. T., Thames, A. D., Li, T., Xiao, J., & Cohen, M. S. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.12.006 · Cited by 95
Phelan, M., Bhavsar, N. A., & Goldstein, B. A. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.243 · Cited by 95
“However, since they are designed for clinical care and billing purposes, rather than research, the data from them are inherently different from traditional data sources.(2) They present an opportunistic and non-random snapshot of patient interactions, capturing only the information that is relevant to each specific en…”
BackgroundTackling COVID-19 Through Responsible AI Innovation: Five Steps in the Right Direction
Leslie, D. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4bb9d7a7 · Cited by 103
Development and validation of early warning score system: A systematic literature review
Fu, L.-H., Schwartz, J., Moy, A., Knaplund, C., Kang, M.-J., Schnock, K. O., Garcia, J. P., Jia, H., Dykes, P. C., Cato, K., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103410 · Cited by 102
Hersh, W., Gorman, P., Biagioli, F. E., Mohan, V., Gold, J. A., & Mejicano, G. C. (2014) · Advances in Medical Education and Practice · doi:10.2147/amep.s63903 · Cited by 94
Self-citationIdentifying and mitigating biases in EHR laboratory tests
Pivovarov, R., Albers, D. J., Sepulveda, J. L., & Elhadad, N. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.03.016 · Cited by 91
“Automating feature selection from EHR variables is a difficult task, as the EHR is an inherently biased data source: EHR data are collected with the primary goal of delivering and documenting patient care, not with the primary goal of creating a curated research dataset [4] [5].”
Using electronic health records for clinical trials: Where do we stand and where can we go?
Cord, K. A. M., & Hemkens, L. G. (2019) · Canadian Medical Association Journal · doi:10.1503/cmaj.180841 · Cited by 88
“29”
BackgroundHarnessing EHR data for health research
Tang, A. S., Woldemariam, S. R., Miramontes, S., Norgeot, B., Oskotsky, T. T., & Sirota, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-03074-8 · Cited by 87
Big data are coming to psychiatry: a general introduction
Monteith, S., Glenn, T., Geddes, J. R., & Bauer, M. (2015) · International Journal of Bipolar Disorders · doi:10.1186/s40345-015-0038-9 · Cited by 95
“…(Wells et al. 2013), inconsistent use of medical terminology (Halamka 2014), redundant data in text (Cohen et al. 2013), varying levels of detail (Hersh et al. 2013), lack of completeness and fragmentation of medical record across providers (Bourgeois et al. 2010), impact of reimbursement…”
BackgroundMadhavan, S., Bastarache, L. A., Brown, J. S., Butte, A. J., Dorr, D. A., Embí, P. J., Friedman, C. P., Johnson, K. B., Moore, J. H., Kohane, I. S., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa287 · Cited by 87
“Utilizing EHRs in support of reporting for the COVID-19 pandemic unveiled the fragility of the health data infrastructure in support of a public health emergency, in a similar way as others have discussed for comparative effectiveness research[18] and healthcare quality improvement[19].”
MethodSelf-citationData Quality Assessment Framework to Assess Electronic Medical Record Data for use in Research
Reimer, A. P., Milinovich, A., & Madigan, E. (2016) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2016.03.006 · Cited by 78
Big Healthcare Data Analytics: Challenges and Applications
Lee, C., Luo, Z., Ngiam, K. Y., Zhang, M., Zheng, K., Chen, G., Ooi, B. C., & Yip, W. L. J. (2017) · Scalable computing and communications · doi:10.1007/978-3-319-58280-1_2 · Cited by 89
Schulz, S., Stegwee, R., & Chronaki, C. (2019) · Fundamentals of Clinical Data Science · doi:10.1007/978-3-319-99713-1_3 · Cited by 86
Harrison, C. M., Henderson, J., Miller, G. C., & Britt, H. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0172935 · Cited by 83
Bower, J. K., Patel, S., Rudy, J. E., & Felix, A. S. (2017) · Current Epidemiology Reports · doi:10.1007/s40471-017-0130-z · Cited by 88
A longitudinal analysis of data quality in a large pediatric data research network
Khare, R., Utidjian, L., Ruth, B. J., Kahn, M. G., Burrows, E. K., Marsolo, K. A., Patibandla, N., Razzaghi, H., Colvin, R., Ranade, D., et al. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx033 · Cited by 83
Advancing Symptom Science Through Use of Common Data Elements
Redeker, N. S., Anderson, R. A., Henry, S. B., Corwin, E. J., Docherty, S. L., Dorsey, S. G., Heitkemper, M. M., McCloskey, D. J., Moore, S. M., Pullen, C. H., et al. (2015) · Journal of Nursing Scholarship · doi:10.1111/jnu.12155 · Cited by 85
Hernandez‐Boussard, T., Monda, K. L., Crespo, B. C., & Riskin, D. J. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz119 · Cited by 82
Bishop, L., Movaghar, A., Greenberg, J. S., Page, D., DaWalt, L. S., Brilliant, M. H., & Seltzer, M. M. (2018) · Autism Research · doi:10.1002/aur.1960 · Cited by 78
“First, as is the case with all EHR data, EHR data from the Marshfield Clinic may contain inaccurate or incomplete data [Hersh et al., 2013; Weiskopf & Weng, 2013] and may more accurately capture serious diseases (i.e., diabetes, colon cancer, epilepsy) than other disorders that are more difficult to…”
MethodCox, J., Edsberg, L. E., Koloms, K., & VanGilder, C. A. (2022) · PubMed Central · doi:10.1097/won.0000000000000834 · Cited by 14
The Georges Pompidou University Hospital Clinical Data Warehouse: A 8-years follow-up experience
Jannot, A., Zapletal, É., Avillach, P., Mamzer, M., Burgun, A., & Patrice Degoulet, A. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.02.006 · Cited by 76
Modern Clinical Text Mining: A Guide and Review
Percha, B. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-030421-030931 · Cited by 81
Machine Learning for Endometrial Cancer Prediction and Prognostication
Bhardwaj, V., Sharma, A., Snijesh, V. P., Gul, I., Zhang, X., Lobie, P. E., Qin, P., & Pandey, V. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.852746 · Cited by 82
“Data are often segmented in a variety of medical imaging archival programs, diagnostic systems, EHRs, automated monitoring software, and insurance databases, making it impossible to integrate (135, 136). ix.”
BackgroundIdentifying Risk Factors for Pressure Injury in Adult Critical Care Patients
Cox, J., Schallom, M. E., & Jung, C. (2020) · American Journal of Critical Care · doi:10.4037/ajcc2020243 · Cited by 75
Pragmatic Precision Psychiatry—A New Direction for Optimizing Treatment Selection
Kessler, R. C., & Luedtke, A. R. (2021) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2021.2500 · Cited by 75
Machine Learning for Predicting Readmission Risk Among the Frail - Explainable AI for Healthcare
Mohanty, S. D., Lekan, D. A., McCoy, T. P., Jenkins, M., & Manda, P. (2021) · Patterns · doi:10.1016/j.patter.2021.100395 · Cited by 70
“81 The stacking classifier used here to combine predictions from individual models was successful in increasing prediction accuracy; however, this increase comes at the cost of explain-ability.”
MethodEstimates of SARS-CoV-2 Omicron BA.2 Subvariant Severity in New England
Strasser, Z. H., Greifer, N., Hadavand, A., Murphy, S. N., & Estiri, H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.38354 · Cited by 67
Cohen, B., Vawdrey, D. K., Liu, J., Caplan, D. W., Furuya, E. Y., Mis, F. W., & Larson, E. (2015) · Policy Politics & Nursing Practice · doi:10.1177/1527154415603358 · Cited by 68
“…reported by others: variations in data definitions or coding over time, inaccurate or inaccessible data elements, access to sufficient informatics and programming expertise to analyze data, and the complexity of linking multiple and varied data sources (Halamka, 2014; Hersh et al., 2013a, 2013b).”
BackgroundTu, K., Widdifield, J., Young, J., Oud, W., Ivers, N. M., Butt, D. A., Leaver, C. A., & Jaakkimainen, L. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0195-x · Cited by 66
“Previous studies that have looked at using electronic health record data for quality measurement [15] or comparative effectiveness research [16] have recognized the complexity of this type of evaluation and have provided conceptual frameworks on how to assess the data and issues to consider.”
BackgroundSecondary Use and Analysis of Big Data Collected for Patient Care
Martin-Sanchez, F. J., Aguiar‐Pulido, V., López–Campos, G., Peek, N. B., & Sacchi, L. (2017) · Yearbook of Medical Informatics · doi:10.15265/iy-2017-008 · Cited by 70
“It must be noticed that in most of the cases when we refer to the use of EHR data, we mostly refer to data that is derived from an “ad-hoc” patient data repository or data warehouse, rather than the operational EHR itself [79].”
MethodBozkurt, S., Cahan, E. M., Seneviratne, M. G., Sun, R., Lossio-Ventura, J. A., Ioannidis, J. P. A., & Hernandez‐Boussard, T. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa164 · Cited by 76
“27 Clearly, many sensitive variables, such as socioeconomic or housing status may not be available in EHRs, hence the low reporting we found on these variables is not surprising.”
BackgroundEfficacy of COVID-19 vaccines in patients taking immunosuppressants
Shen, C., Risk, M., Schiopu, E., Hayek, S. S., Xie, T., Holevinski, L., Akin, C., Freed, G. L., & Zhao, L. (2022) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2021-222045 · Cited by 65
The impact of routine data quality assessments on electronic medical record data quality in Kenya
Muthee, V., Bochner, A. F., Osterman, A., Liku, N., Akhwale, W., Kwach, J., Prachi, M., Wamicwe, J., Odhiambo, J., Onyango, F., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0195362 · Cited by 64
Big Data in Head and Neck Cancer
Resteghini, C., Trama, A., Borgonovi, E., Hosni, H., Corrao, G., Orlandi, E. M., Calareso, G., Cecco, L. D., Piazza, C., Mainardi, L. T., et al. (2018) · Current Treatment Options in Oncology · doi:10.1007/s11864-018-0585-2 · Cited by 62
Daniel Capurro, N., Yetişgen, M., Eaton, E. G. V., Black, R. S., & Tarczy-Hornoch, P. Z. (2014) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1079 · Cited by 67
Application of an Ontology for Characterizing Data Quality for a Secondary Use of EHR Data
Speedie, S. M., Simon, G., Kumar, V., Westra, B. L., & Johnson, S. (2016) · Applied Clinical Informatics · doi:10.4338/aci-2015-08-ra-0107 · Cited by 64
“A number of studies have shown that EHR data contain errors that can affect research results [7–9].”
BackgroundIlan, Y. (2020) · Frontiers in Digital Health · doi:10.3389/fdgth.2020.569178 · Cited by 68
“Sources may suffer from the absence of a unifying data format across centers (100).”
BackgroundElectronic Health Records and Quality of Care
Yanamadala, S., Morrison, D. E., Curtin, C. M., McDonald, K. M., & Hernandez‐Boussard, T. (2016) · Medicine · doi:10.1097/md.0000000000003332 · Cited by 75
“Electronic health records were originally built for billing purposes, not for research and quality improvement efforts.(5) Accordingly, the impact of EHRs on quality healthcare delivery has focused on physician performance and billing precision.”
BackgroundHarvey, R. A., Hayden, J., Kamble, P., Bouchard, J. R., & Huang, J. (2016) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4121 · Cited by 62
Davies, S. J. D., Grundmeier, R. W., Campos, D., Hayes, K. L., Bell, J., Alessandrini, E. A., Bajaj, L., Chamberlain, J., Gorelick, M. H., Enríquez, R., et al. (2018) · Applied Clinical Informatics · doi:10.1055/s-0038-1651496 · Cited by 61
Hersh, W., Cimino, J. J., Payne, P., Embí, P. J., Logan, J. R., Weiner, M. G., Bernstam, E. V., Lehmann, H. P., Hripcsak, G. M., Hartzog, T. H., et al. (2013) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1018 · Cited by 62
Self-citationCortés, A., Dendrou, C. A., Motyer, A., Jostins, L., Vukcevic, D., Dilthey, A., Donnelly, P., Leslie, S., Fugger, L. H., & McVean, G. (2017) · Nature Genetics · doi:10.1038/ng.3926 · Cited by 58
Validity of Cardiovascular Data From Electronic Sources
Ahmad, F. S., Chan, C., Rosenman, M. B., Post, W. S., Fort, D. G., Greenland, P., Liu, K., Kho, A., & Allen, N. B. (2017) · Circulation · doi:10.1161/circulationaha.117.027436 · Cited by 60
Kurian, A. W., Mitani, A., Desai, M., Yu, P. P., Seto, T., Weber, S., Olson, C., Kenkare, P., Gomez, S. L., de Bruin, M. A., et al. (2013) · Cancer · doi:10.1002/cncr.28395 · Cited by 61
“Given the unique potential of EMRs to enhance the understanding of cancer outcomes, studies to optimize the clinical and research uses of EMRs should remain a high priority.(33,34) Some limitations may be addressed through EMR changes, with structured fields facilitating data extraction; others require new data source…”
BackgroundClinical Research Informatics and Electronic Health Record Data
Horvath, M. M., Rusincovitch, S. A., & Richesson, R. (2014) · Yearbook of Medical Informatics · doi:10.15265/iy-2014-0009 · Cited by 59
“Essentially, generalized evidence synthesis (e.g., comparative effectiveness research or meta-analyses) of any kind will be dependent upon shared semantics and data quality [46, 88-91].”
BackgroundInfluentialElectronic Health Records and Community Health Surveillance of Childhood Obesity
Flood, T. L., Zhao, Y., Tomayko, E. J., Tandias, A., Carrel, A. L., & Hanrahan, L. P. (2015) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.10.020 · Cited by 58
Developing real‐world evidence from real‐world data: Transforming raw data into analytical datasets
Bastarache, L. A., Brown, J. S., Cimino, J. J., Dorr, D. A., Embí, P. J., Payne, P., Wilcox, A. B., & Weiner, M. G. (2021) · Learning Health Systems · doi:10.1002/lrh2.10293 · Cited by 58
Self-citationA Review of Automatic Phenotyping Approaches using Electronic Health Records
Alzoubi, H., Alzubi, R., Ramzan, N., West, D., Al-Hadhrami, T., & Alazab, M. (2019) · Electronics · doi:10.3390/electronics8111235 · Cited by 56
“However, with the improvement and development in information technology, increasingly these records are adopted for secondary uses such as quality and safety measurement, clinical decision support, genome-wide association studies (GWAS), and clinical research [1,2].”
BackgroundExpectations for Artificial Intelligence (AI) in Psychiatry
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E. D., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01378-5 · Cited by 61
“…were not developed for medical research, and there are many data quality issues related to missing data, inaccuracy, coding errors, biases, timeliness, redundancies, types of healthcare facilities, provenance or ownership trails, and lack of interoperability between vendor products [22, 68, 69].”
BackgroundImplications of non-stationarity on predictive modeling using EHRs
Jung, K., & Shah, N. H. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.10.006 · Cited by 55
“However, the observational nature of EHR derived data raises several practical issues in the development of such models [7-9].”
BackgroundAdie, Y., Kats, D. J., Tlimat, A., Perzynski, A. T., Dalton, J. E., Gunzler, D. D., & Tarabichi, Y. (2019) · CHEST Journal · doi:10.1016/j.chest.2019.11.033 · Cited by 54
“EHR data fields were not primarily designed for research and are often incomplete or inaccurate as a result.(33) The family history of lung cancer, for instance, is a field that is defaulted to a negative value.”
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., & Bauer, M. (2016) · International Journal of Bipolar Disorders · doi:10.1186/s40345-016-0051-7 · Cited by 51
“…issues in EMR data including inconsistency, redundancy, inaccuracy, missing data, interoperability between vendor products, and potential biases from measured and non-measured confounders (Monteith et al. 2015; Bayley et al. 2013; Kaplan et al. 2014; Hersh et al. 2013; Hripcsak et al. 2011).”
BackgroundInformatics and machine learning to define the phenotype
Basile, A. O., & Ritchie, M. D. (2018) · Expert Review of Molecular Diagnostics · doi:10.1080/14737159.2018.1439380 · Cited by 51
“…entered), miscommunication with a patient (inaccurate information is provided by the patient or there is a miscommunication between patient and clinician), and timeliness of data entry (medical professional may chart data much later than it was originally observed due to time constraints) [61].”
BackgroundLeveraging electronic health records for clinical research
Raman, S. R., Curtis, L. H., Temple, R., Andersson, T., Ezekowitz, J. A., Ford, I., James, S. K., Marsolo, K. A., Mirhaji, P., Rocca, M., et al. (2018) · American Heart Journal · doi:10.1016/j.ahj.2018.04.015 · Cited by 56
“At the same time, concerns arise about whether the quality of the data is suitable for research.(5,9) The quality of EHR data directly impacts the research that may follow and can be illustrated by our ability to characterize missingness.”
BackgroundPredictive modeling in urgent care: a comparative study of machine learning approaches
Tang, F., Xiao, C., Wang, F., & Zhou, J. (2018) · JAMIA Open · doi:10.1093/jamiaopen/ooy011 · Cited by 53
“917 for LOS ranges (1, 2), (2, 3), (3, 5), (5, 8), (8, 14), (14, 21), and (21, 30).”
MethodDeep Learning for Electronic Health Records Analytics
Harerimana, G., Kim, J. W., Yoo, H., & Jang, B. (2019) · IEEE Access · doi:10.1109/access.2019.2928363 · Cited by 56
Gallego, B., Walter, S. R., Day, R. O., Dunn, A. G., Sivaraman, V., Shah, N. H., Longhurst, C., & Coiera, E. W. (2015) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.15.12 · Cited by 49
Metadata
- Authors
- Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., & Saltz, J. H.
- Affiliations
- Oregon Health & Science University (US)
- AstraZeneca (United Kingdom) (GB)
- The Ohio State University Wexner Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Johns Hopkins University (US)
- Columbia University (US)
- Medical University of South Carolina (US)
- National Institutes of Health (US)
- Emory University (US)
- Corresponding author
- Hersh, William R.
- Logan, Judith R.
- Cimino, James J.
- ORCID
- Venue
- Medical Care
- Published
- 2013-08
- Volume
- 51
- Issue
- Supplement 8Suppl 3
- Pages
- S30-S37
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC3748381
- OpenAlex
- W2137767404
- Cited by
- 567
- References
- 68
- Citations by year
- 2026: 43 · 2025: 44 · 2024: 30 · 2023: 43 · 2022: 43 · 2021: 58 · 2020: 50 · 2019: 59 · 2018: 50 · 2017: 45 · 2016: 45 · 2015: 27 · 2014: 24 · 2013: 6
- FWCI
- 58.67
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000454, UL1TR000090)
- Centers for Disease Control and Prevention
- National Center for Research Resources (UL1RR025008, UL1RR024156, UL1RR025005, UL1RR024148, UL1RR024140)
- National Center for Advancing Translational Sciences
- H2020 European Research Council
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- electronic health records, comparative effectiveness research, data quality, data completeness, data provenance, data granularity, data reuse
- MeSH terms
- Data Collection, Data Interpretation, Statistical, Humans, Insurance Claim Review, Research Design, Comparative Effectiveness Research, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
