Cited by 567Electronic Health Records SystemsMedical Coding and Health InformationMachine Learning in Healthcare

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.

Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research

Open access

Hersh, William R., Weiner, Mark G., Embi, Peter J., Logan, Judith R., Payne, Philip R.O., Bernstam, Elmer V., Lehmann, Harold P., Hripcsak, George, Hartzog, Timothy H., Cimino, James J., Saltz, Joel H.·2013·Medical Care

DOI 10.1097/mlr.0b013e31829b1dbdPMC PMC3748381

Open this paper in CoChat — free account

Abstract

The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1097/mlr.0b013e31829b1dbd

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 2,127
The “Meaningful Use” Regulation for Electronic Health Records
Blumenthal, D., & Tavenner, M. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006114 · Cited by 2,059
A review of uses of health care utilization databases for epidemiologic research on therapeutics
Schneeweiß, S., & Avorn, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.012 · Cited by 1,290
Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 1×
“A review of the literature of studies evaluating the data quality of EHRs for clinical research identified 5 dimensions of data quality assessed: completeness, correctness, concordance, plausibility, and currency.(26) The authors identified many studies with a wide variety of techniques to assess these dimensions and,…”
Measuring Diagnoses: ICD Code Accuracy
O’Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,156
Assessing Quality Using Administrative Data
Iezzoni, L. I. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00048 · Cited by 981
Next-generation phenotyping of electronic health records
Hripcsak, G., & Albers, D. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001145 · Cited by 795
Self-citation
Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper.
Safran, C., Bloomrosen, M., Hammond, W. E., Labkoff, S., Markel-Fox, S., Tang, P., & Detmer, D. (2007) · JAMIA Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2273 · Cited by 714
The eMERGE Network: A consortium of biorepositories linked to electronic medical records data for conducting genomic studies
the eMERGE Team, McCarty, C. A., Chisholm, R. L., Chute, C. G., Kullo, I. J., Jarvik, G. P., Larson, E. B., Li, R., Masys, D. R., Ritchie, M. D., et al. (2011) · BMC Medical Genomics · doi:10.1186/1755-8794-4-13 · Cited by 701
Desiderata for Controlled Medical Vocabularies in the Twenty-First Century
Cimino, J. J. (1998) · Methods of Information in Medicine · doi:10.1055/s-0038-1634558 · Cited by 656
Self-citation

Flagged first, then by citations

Show all 68 references

Sorted by citations

Items 201–300 of 564

Can we predict a national profile of non-attendance paediatric urology patients: a multi-institutional electronic health record study
Bush, R. A., Vemulakonda, V. M., Corbett, S. T., & Chiang, G. J. (2014) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v21i3.59 · Cited by 21
Utilizing Data Consortia to Monitor Safety and Effectiveness of Biosimilars and Their Innovator Products
AMCP Task Force on Biosimilar Collective Intelligence Systems (2015) · Journal of Managed Care & Specialty Pharmacy · doi:10.18553/jmcp.2015.21.1.23 · Cited by 20
Shared Genetic Etiology of Autoimmune Diseases in Patients from a Biorepository Linked to De-identified Electronic Health Records
RESTREPO, N. A., Butkiewicz, M., McGrath, J. A., & Crawford, D. C. (2016) · Frontiers in Genetics · doi:10.3389/fgene.2016.00185 · Cited by 21
“The paucity of clinical data for a proportion of patients in any given EHR is a known caveat for use of these data in a research setting (Hersh et al., 2013).”
Using voice to create inpatient progress notes: effects on note timeliness, quality, and physician satisfaction
Payne, T. H., Alonso, W. D., Markiel, J. A., Lybarger, K., Lordon, R. J., Yetişgen, M., Zech, J. M., & White, A. A. (2018) · JAMIA Open · doi:10.1093/jamiaopen/ooy036 · Cited by 20
Design and analytic considerations for using patient-reported health data in pragmatic clinical trials: report from an NIH Collaboratory roundtable
Rockhold, F. W., Tenenbaum, J. D., Richesson, R., Marsolo, K. A., & O'Brien, E. C. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz226 · Cited by 20
“Some data elements may be found only in free text or may be structured data found in fields other than the 1 expected.(8) For example, a researcher might be interested in ejection fraction information.”
Background
The challenges and opportunities in using real-world data to drive advances in healthcare in East Asia: expert panel recommendations
Crane, G., Lim, J. C. W., Gau, C., Xie, J., & Chu, L. K. (2022) · Current Medical Research and Opinion · doi:10.1080/03007995.2022.2096354 · Cited by 19
“Deriving useful RWD from systems not designed for research (e.g. EMR and billing systems) can be challenging because data may be incomplete, missing, inaccurate, or inaccessible 18 (e.g. genomic test results may be an incompatible format such as PDF).”
Data linkages between patient-powered research networks and health plans: a foundation for collaborative research
Agiro, A., Chen, X., Eshete, B., Sutphen, R., Clark, E. B., Burroughs, C., Nowell, W. B., Curtis, J. R., Loud, S., McBurney, R. N., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz012 · Cited by 19
“ABOUT Network PPRN memberships overlapping with health plans 1228 Breast or Ovarian Cancer 519 373 72 (68-76) 309 60 (55-64) Breast Cancer 474 311 66 (61-70) 275 58 (53-63) Ovarian Cancer 62 42 68 (55-79) 39 63 (50-75) ArthritisPower PPRN memberships overlapping with health plans 1600 Arthritis or Psoriasis 935 467 50…”
Background
Comparative analysis, applications, and interpretation of electronic health record-based stroke phenotyping methods
Thangaraj, P., Kummer, B. R., Lorberbaum, T., Elkind, M. S. V., & Tatonetti, N. P. (2019) · BioData Mining · doi:10.1186/s13040-020-00230-x · Cited by 10
Characterization of the phenotypic consequences of the Duffy-null genotype
Hysong, M. R., Shuey, M. M., Huffman, J. E., Auer, P. L., Reiner, A. P., & Raffield, L. M. (2025) · Blood Advances · doi:10.1182/bloodadvances.2024014399 · Cited by 20
Utilizing Big Data analytics and electronic health record data in HIV prevention, treatment, and care research: a literature review
Qiao, S., Li, X., Olatosi, B., & Young, S. D. (2021) · AIDS Care · doi:10.1080/09540121.2021.1948499 · Cited by 18
“Potential strategies include mandating or providing incentives for adoption of standards-based clinical data across various sites (Hersh et al., 2013); developing guidelines and training for optimal data entry, structure, and extraction (Hersh et al., 2013); and promoting data sharing by providing…”
Background
Data Elements and Validation Methods Used for Electronic Surveillance of Healthcare-Associated Infections: A Systematic Review
Cato, K., Cohen, B., & Larson, E. (2015) · American Journal of Infection Control · doi:10.1016/j.ajic.2015.02.006 · Cited by 13
Methodological and Ethical Issues in Pediatric Medication Safety Research
Carpenter, D. M., González, D., Retsch‐Bogart, G. Z., Sleath, B. L., & Wilfond, B. S. (2017) · PEDIATRICS · doi:10.1542/peds.2017-0195 · Cited by 18
A conceptual framework for evaluating data suitability for observational studies
Shang, N., Weng, C., & Hripcsak, G. M. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx095 · Cited by 23
“Many research studies require the use of multiple resources to identify an adequate study cohort or to formulate a complete patient story.(41,48) In such scenarios, database-level and person-level record linkages are important, and data custodians should provide means to facilitate such linkages.”
BackgroundSelf-citation
Challenges of Using ICD-9-CM and ICD-10-CM Codes for Soft-Tissue Sarcoma in Databases for Health Services Research.
Hess, L. M., Zhu, Y. E., Sugihara, T., Fang, Y., Collins, N., & Nicol, S. J. (2019) · Europe PMC (PubMed Central) · Cited by 18
Workflow Differences Affect Data Accuracy in Oncologic EHRs: A First Step Toward Detangling the Diagnosis Data Babel
Diaz‐Garelli, F., Strowd, R. E., III, Lawson, V., Mayorga, M. E., Wells, B. J., Lycan, T. W., & Topaloĝlu, Ü. (2020) · JCO Clinical Cancer Informatics · doi:10.1200/cci.19.00114 · Cited by 18
“5,38 However, the literature is still unclear on which factors affect data quality.”
BackgroundInfluential
Facilitating biomedical researchers’ interrogation of electronic health record data: Ideas from outside of biomedical informatics
Hruby, G. W., Matsoukas, K., Cimino, J. J., & Weng, C. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.03.004 · Cited by 19
“However, this task faces significant human and technological barriers [10, 16, 17].”
BackgroundSelf-citation
Major Issues in Adoption of Electronic Health Records
Yi, M. (2018) · Journal of Digital Information Management · doi:10.6025/jdim/2018/16/4/180-191 · Cited by 20
Aggregating Electronic Health Record Data for COVID-19 Research—Caveat Emptor
Brown, J. S., Bastarache, L. A., & Weiner, M. G. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.17175 · Cited by 17
Self-citation
The geographic distribution of cardiovascular health in the stroke prevention in healthcare delivery environments (SPHERE) study
Roth, C., Payne, P., Weier, R. C., Shoben, A. B., Fletcher, E. N., Lai, A. M., Kelley, M. M., Plascak, J. J., & Foraker, R. E. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.01.013 · Cited by 17
Self-citation
Predicting Causes of Data Quality Issues in a Clinical Data Research Network.
Khare, R., Ruth, B. J., Miller, M. W., Tucker, J. S., Utidjian, L., Razzaghi, H., Patibandla, N., Ek, B., & Bailey, L. C. (2018) · Europe PMC (PubMed Central) · Cited by 17
Real-time imputation of missing predictor values in clinical practice
Nijman, S. W. J., Hoogland, J., Groenhof, T. K. J., Brandjes, M., Jacobs, J. J. L., Bots, M. L., Asselbergs, F. W., Moons, K. G. M., & Debray, T. P. A. (2020) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztaa016 · Cited by 17
Ecological Assessment of Clinicians’ Antipsychotic Prescription Habits in Psychiatric Inpatients: A Novel Web- and Mobile Phone–Based Prototype for a Dynamic Clinical Decision Support System
Berrouiguet, S., Barrigón, M. L., Brandt, S. A., Nitzburg, G. C., Ovejero, S., Álvarez-García, R., Carballo, J. J., Walter, M., Billot, R., Lenca, P., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.5954 · Cited by 17
“This practice, although prevalent in clinical settings, is discouraged in clinical guidelines [25].”
Variation in Documenting Diagnosable Chronic Kidney Disease in General Medical Practice: Implications for Quality Improvement and Research
Kitsos, A. R., Peterson, G. M., Jose, M., Khanam, M. A., Castelino, R. L., & Radford, J. C. (2019) · Journal of Primary Care & Community Health · doi:10.1177/2150132719833298 · Cited by 17
How the Provenance of Electronic Health Record Data Matters for Research: A Case Example Using System Mapping
Johnson, K. E., Kamineni, A., Fuller, S., Olmstead, D., & Wernli, K. J. (2014) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1058 · Cited by 39
Secure Secondary Use of Clinical Data with Cloud-based NLP Services
Christoph, J., Griebel, L., Leb, I., Engel, I., Köpcke, F., Toddenroth, D., Prokosch, H. U., Laufer, J., Marquardt, K., & Sedlmayr, M. (2014) · Methods of Information in Medicine · doi:10.3414/me13-01-0133 · Cited by 16
“Just recently, for example, Hersh et al. have published seven caveats for the use of operational EHR data in comparative effectiveness research [1].”
Background
An architecture for research computing in health to support clinical and translational investigators with electronic patient data
Campion, T. R., Sholle, E., Pathak, J., Johnson, S. B., Leonard, J. P., & Cole, C. L. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab266 · Cited by 20
“1,2 Challenges include repurposing transactional (eg, care, billing) data for analytical purposes, finding and using the right electronic tools, understanding strengths and limitations of underlying data, obtaining regulatory approval, and maintaining compliance.”
R Packages for Data Quality Assessments and Data Monitoring: A Software Scoping Review with Recommendations for Future Developments
Mariño, J., Kasbohm, E., Struckmann, S., Kapsner, L. A., & Schmidt, C. O. (2022) · Applied Sciences · doi:10.3390/app12094238 · Cited by 20
Extracting forced vital capacity from the electronic health record through natural language processing in rheumatoid arthritis‐associated interstitial lung disease
England, B. R., Roul, P., Yang, Y., Hershberger, D. M., Sayles, H. R., Rojas, J., Cannon, G. W., Sauer, B. C., Curtis, J. R., Baker, J. F., et al. (2023) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5744 · Cited by 18
“13 Even in the EHR, FVC values can be difficult to obtain because they may not be stored as discrete data elements amenable to a simple extraction.”
Background
Consore: A Powerful Federated Data Mining Tool Driving a French Research Network to Accelerate Cancer Research
Guérin, J., Nahid, A., Tassy, L., Deloger, M., Bocquet, F., Thézenas, S., Désandes, E., Deley, M. L., Durando, X., Jaffré, A., et al. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21020189 · Cited by 16
Assessment of Potentially Inappropriate Prescribing of Opioid Analgesics Requiring Prior Opioid Tolerance
Jeffery, M. M., Chaisson, C. E., Hane, C. A., Rumanes, L., Tucker, J. M., Hang, L., McCoy, R. G., Chen, C. L., Bicket, M. C., Hooten, W. M., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.2875 · Cited by 16
Clinical Informatics Researcher's Desiderata for the Data Content of the Next Generation Electronic Health Record
Kennell, T., Willig, J. H., & Cimino, J. J. (2017) · Applied Clinical Informatics · doi:10.4338/aci-2017-06-r-0101 · Cited by 16
“3,4 Clinical informaticians are reporting some of these de ﬁ - ciencies, 5 whilemakingdowithextractingandinferringpatient information from current EHRs for various purposes.”
BackgroundSelf-citation
Accuracy of medical billing data against the electronic health record in the measurement of colorectal cancer screening rates
Rudrapatna, V. A., Glicksberg, B. S., Avila, P., Harding-Theobald, E. R., Wang, C. W., & Butte, A. J. (2020) · BMJ Open Quality · doi:10.1136/bmjoq-2019-000856 · Cited by 16
Data quality assessment in healthcare, dimensions, methods and tools: a systematic review
Hosseinzadeh, E., Afkanpour, M., Momeni, M., & Tabesh, H. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03136-y · Cited by 22
Evidence‐Based Medicine in Otolaryngology Part 14: Falsehood and Bias
Mallur, P. S., Ikeda, A. K., Patel, A., Raol, N., Ahanotu, A., Suarez‐Goris, D., Randolph, G. W., & Shin, J. J. (2023) · Otolaryngology · doi:10.1002/ohn.215 · Cited by 15
Considerations for using research data to verify clinical data accuracy.
Fort, D. G., Weng, C., Henry, S. B., & Wilcox, A. B. (2014) · Europe PMC (PubMed Central) · Cited by 15
Research IT maturity models for academic health centers: Early development and initial evaluation
Knosp, B. M., Barnett, W. K., Anderson, N., & Embí, P. J. (2018) · Journal of Clinical and Translational Science · doi:10.1017/cts.2018.339 · Cited by 15
Lost in Translation: Diagnosis Records Show More Inaccuracies After Biopsy in Oncology Care EHRs.
Diaz‐Garelli, F., Strowd, R. E., III, Wells, B. J., Ahmed, T., Merrill, R., & Topaloĝlu, Ü. (2019) · PubMed Central · Cited by 15
Literature-Based Discovery of Confounding in Observational Clinical Data.
Malec, S. A., Wei, P., Xu, H., Bernstam, E. V., Myneni, S., & Cohen, T. (2016) · PubMed Central · Cited by 14
Self-citation
A Framework for Visualizing Data Quality for Predictive Models and Clinical Quality Measures.
Johnson, S., Pruinelli, L., Hoff, A., Kumar, V., Simon, G., Steinbach, M. S., & Westra, B. L. (2019) · PubMed Central · Cited by 14
Integration of Registries with EHRs to Accelerate Generation of Real-World Evidence for Clinical Practice and Learning Health Systems Research
Franklin, P. D., Lurie, J. D., Tosteson, T. D., Tosteson, A. N. A., & Task Force on Musculoskeletal Registries (2020) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.19.01464 · Cited by 14
Clinical study applying machine learning to detect a rare disease: results and lessons learned
Hersh, W., Cohen, A. M., Nguyen, M., Bensching, K. L., & DeLoughery, T. G. (2022) · JAMIA Open · doi:10.1093/jamiaopen/ooac053 · Cited by 15
“6 Although there are limitations to relying on EHR data for clinical research, 7 it was the only data available to us for these patients.”
MethodSelf-citation
An Observational Analysis of Provider Adherence to AUA Guidelines on the Management of Benign Prostatic Hyperplasia
Auffenberg, G. B., Gonzalez, C. M., Wolf, J. S., Clemens, J. Q., Meeks, W. D., & McVary, K. T. (2014) · The Journal of Urology · doi:10.1016/j.juro.2014.06.016 · Cited by 14
A practical framework for operationalising responsible and equitable artificial intelligence in health care: tackling bias, inequity, and implementation challenges
Welch, M. L., Grant, B., Deutschman, C., McElcheran, C., Badzynski, A., Bell, J. A. H., Hope, A. J., Grant, R. C., Truong, T., Lane, K., et al. (2026) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100957 · Cited by 13
Postpartum readmission for hypertension and pre‐eclampsia: development and validation of a predictive model
Venkatesh, K. K., Jelovsek, J. E., Hoffman, M., Beckham, A. J., Bitar, G., Friedman, A. M., Boggess, K., & Stamilio, D. M. (2023) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17572 · Cited by 13
“38 The available EHR data did not include candidate variables that could have improved model performance, such as use of antihypertensive medication at delivery hospital discharge.”
Racial Disparities in Invasive ICU Treatments Among Septic Patients: High Resolution Electronic Health Records Analysis from MIMIC-IV
Mohammed, S., Matos, J., Doutreligne, M., Celi, L. A., & Struja, T. (2023) · The Yale Journal of Biology and Medicine · doi:10.59249/wdji8829 · Cited by 13
MENDS-on-FHIR: leveraging the OMOP common data model and FHIR standards for national chronic disease surveillance
Essaid, S., Andre, J., Brooks, I. M., Hohman, K. H., Hull, M. L., Jackson, S. L., Kahn, M. G., Kraus, E. M., Mandadi, N., Martinez, A. K., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae045 · Cited by 19
“With any data translation, mapping fidelity is a concern [40], although mapping errors may have smaller-than-anticipated impact on analytic results [41–43].”
Background
What Should Oversight of Clinical Decision Support Systems Look Like?
Evans, E. L., & Whicher, D. M. (2018) · The AMA Journal of Ethic · doi:10.1001/amajethics.2018.857 · Cited by 13
Data electronically extracted from the electronic health record require validation
Scheid, L. M., Brown, L. S., Clark, C., & Rosenfeld, C. R. (2019) · Journal of Perinatology · doi:10.1038/s41372-018-0311-8 · Cited by 13
Improving the Quality and Utility of Electronic Health Record Data through Ontologies
Lin, A., Arabandi, S., Beale, T., Duncan, W., Hicks, A., Hogan, W. R., Jensen, M., Koppel, R., Martínez-Costa, C., Nytrø, Ø., et al. (2023) · Standards · doi:10.3390/standards3030023 · Cited by 11
“Further deﬁciencies of EHR data that hinder their reuse in translational research scenarios are summarized in Table 2 [8].”
Background
Extracting Electronic Health Record Data in a Practice-Based Research Network: Lessons Learned from Collaborations with Translational Researchers
Cole, A., Stephens, K. A., Keppel, G. A., Estiri, H., & Baldwin, L. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1206 · Cited by 14
“This process requires complex coordination across multiple domains, including: 1) understanding the context of EHR data; 2) creating and maintaining a governance structure to support exchange of EHR data; and 3) defining data parameters that are used in order to extract data from the EHR.1,2,3,4 We use the Northwest-A…”
BackgroundMethodInfluential
Diagnosis, misdiagnosis, lucky guess, hearsay, and more: an ontological analysis
Hogan, W. R., & Ceusters, W. M. (2016) · Journal of Biomedical Semantics · doi:10.1186/s13326-016-0098-5 · Cited by 15
“For example, Hersh et al. note that data recorded in billing workflows for financial purposes are less accurate than clinical data [10].”
Background
A dynamic risk model for inpatient falls
Choi, Y., Staley, B. J., Henriksen, C. H., Xu, D., Lipori, G. P., Brumback, B., & Winterstein, A. G. (2018) · American Journal of Health-System Pharmacy · doi:10.2146/ajhp180013 · Cited by 13
Variation in Laboratory Test Naming Conventions in EHRs Within and Between Hospitals
Wiitala, W. L., Vincent, B. M., Burns, J., Prescott, H. C., Waljee, A. K., Cohen, G. R., & Iwashyna, T. J. (2018) · Medical Care · doi:10.1097/mlr.0000000000000996 · Cited by 13
Predicting Glaucoma Surgical Outcomes Using Neural Networks and Machine Learning on Electronic Health Records
Barry, S., & Wang, S. Y. (2024) · Translational Vision Science & Technology · doi:10.1167/tvst.13.6.15 · Cited by 12
A Comparison of Health Plan- and Provider-Delivered Chronic Care Management Models on Patient Clinical Outcomes
Luo, Z., Chen, Q., Annis, A., Piatt, G., Green, L. A., Tao, M., & Holtrop, J. S. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3617-2 · Cited by 12
Major hemorrhage in chronic lymphocytic leukemia patients in the US Veterans Health Administration system in the pre‐ibrutinib era: Incidence and risk factors
Georgantopoulos, P., Yang, H., Norris, L. B., & Bennett, C. L. (2019) · Cancer Medicine · doi:10.1002/cam4.2134 · Cited by 12
Integrating electronic healthcare records of armed forces personnel: Developing a framework for evaluating health outcomes in England, Scotland and Wales
Leightley, D., Chui, Z., Jones, M., Landau, S., McCrone, P., Hayes, R. D., Wessely, S. C., Fear, N. T., & Goodwin, L. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.02.012 · Cited by 12
Assessing Quality of Surgical Real-World Data from an Automated Electronic Health Record Pipeline
Corey, K., Helmkamp, J. K., Simons, M. G., Curtis, L. H., Marsolo, K. A., Balu, S., Gao, M. C., Nichols, M., Watson, J. A., Mureebe, L., et al. (2020) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2019.12.005 · Cited by 13
Generalization in Clinical Prediction Models: The Blessing and Curse of Measurement Indicator Variables
Futoma, J., Simons, M. G., Doshi‐Velez, F., & Kamaleswaran, R. (2021) · Critical Care Explorations · doi:10.1097/cce.0000000000000453 · Cited by 12
“org July 2021 • Volume 3 • Number 7 Furthermore, we developed our models on retrospective EHR data, which may limit the applicability of the model if used prospectively (40).”
Method
Prevalence and Characterization of Yoga Mentions in the Electronic Health Record
Penrod, N. M., Lynch, S., Thomas, S., Seshadri, N., & Moore, J. H. (2019) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2019.06.190115 · Cited by 11
Harmonization of the ICHOM Quality Measures to Enable Health Outcomes Measurement in Multimorbid Patients
Blom, M., Khalid, M. D., van Lettow, B., Hutink, H., Larsson, S., Huff, S., & Ingvar, M. (2020) · Frontiers in Digital Health · doi:10.3389/fdgth.2020.606246 · Cited by 11
“Data quality issues (6, 14, 15) and challenges in repurposing data have caused concerns on validity (6, 14) of the wealth of data available in clinical information systems (16).”
Background
Network-medicine framework for studying disease trajectories in U.S. veterans
do Valle, Í. F., Ferolito, B. R., Gerlovin, H., Costa, L., Demissie, S., Linares, F., Cohen, J., GAGNON, D. R., Gaziano, J. M., Begoli, E., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-15764-9 · Cited by 11
“Possible errors in diagnostic codes, admission dates and incomplete recording might cause variations in the resulting disease associations 30,31 .”
Background
A Road Map for Peer Review of Real-World Evidence Studies on Safety and Effectiveness of Treatments
Winterstein, A. G., Ehrenstein, V., Brown, J. S., Stürmer‎, T., & Smith, M. Y. (2023) · Diabetes Care · doi:10.2337/dc22-2037 · Cited by 11
“However, even though N3C follows good processes for data set creation, the complexities in using EHR data from multiple disparate health systems for evaluation of treatment effects (54), and the novelty of the data source and processes that created and curated it, pose significant challenges for determining whether it…”
Background
Using primary care electronic health record data for comparative effectiveness research: experience of data quality assessment and preprocessing in The Netherlands
Huang, Y., Voorham, J., & Haaijer‐Ruskamp, F. M. (2016) · Journal of Comparative Effectiveness Research · doi:10.2217/cer-2015-0022 · Cited by 11
A tale of three subspecialties: Diagnosis recording patterns are internally consistent but Specialty-Dependent
Diaz‐Garelli, F., Strowd, R. E., III, Ahmed, T., Wells, B. J., Merrill, R., Laurini, J. A., Pasche, B., & Topaloĝlu, Ü. (2019) · JAMIA Open · doi:10.1093/jamiaopen/ooz020 · Cited by 11
“, ICD code inaccuracy rates went from 20–70% in the 1970s to 20% in 1980s), their reliability remains questioned.(3,14) The complex nature of clinical knowledge, variability of clinical workflows, and clinical data being generated for care and billing rather than research(3,15,16) have led to development of workaround…”
Background
Developing a pediatric pain data repository
Donado, C., Lobo, K., Berde, C. B., & Bourgeois, F. T. (2019) · JAMIA Open · doi:10.1093/jamiaopen/ooz062 · Cited by 11
“Information collected in the clinical setting may be incomplete, inaccurate, or have insufficient detail to yield useful information for research purposes.(15) Additionally, EHR data are typically presented in free text format or may present multiple sources of data for a given measure.”
Background
Linguistic and ontological challenges of multiple domains contributing to transformed health ecosystems
Kreuzthaler, M., Brochhausen, M., Zayas, C. E., Blobel, B. G. M. E., & Schulz, S. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1073313 · Cited by 12
“, for billing, disease reporting and quality assurance, often redundantly and therefore unwillingly, which explains biases and errors (12).”
Background
Rediscovering drug side effects: the impact of analytical assumptions on the detection of associations in EHR data.
Diaz‐Garelli, F., Bernstam, E. V., Mse, Rahbar, M. H., & Johnson, T. R. (2015) · Europe PMC (PubMed Central) · Cited by 11
Self-citation
Scalable joint modeling of longitudinal and point process data for disease trajectory prediction and improving Management of Chronic Kidney Disease
Futoma, J., Sendak, M., Cameron, C. B., & Heller, K. (2016) · Uncertainty in Artificial Intelligence · Cited by 11
Innovation on Machine Learning in Healthcare Services—An Introduction
Pattnayak, P., & Panda, A. R. (2021) · Studies in computational intelligence · doi:10.1007/978-981-33-4698-7_1 · Cited by 10
Fate or coincidence: do COPD and major depression share genetic risk factors?
Martucci, V. L., Richmond, B. W., Davis, L. K., Blackwell, T. S., Cox, N. J., Samuels, D. C., Edwards, D. R. V., & Aldrich, M. C. (2021) · Human Molecular Genetics · doi:10.1093/hmg/ddab068 · Cited by 12
Datenreiche Medizin und das Problem der Einwilligung
Richter, G., Loh, W., Buyx, A., von Kielmansegg, S. G., & GmbH, S.-V. (2021) · Springer eBooks · doi:10.1007/978-3-662-62987-1 · Cited by 11
A Shared Opportunity for Improving Electronic Medical Record Data
Baier, A. W., Snyder, D., Leahy, I. C., Patak, L. S., & Brustowicz, R. M. (2017) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002134 · Cited by 11
Applied Methods for Estimating Transition Probabilities from Electronic Health Record Data
Rodriguez, P. J., Ward, Z. J., Long, M. W., Austin, S. B., & Wright, D. R. (2021) · Medical Decision Making · doi:10.1177/0272989x20985752 · Cited by 10
“While these changes are specific to the application of ED, similar problems with using administrative data for research have been widely documented.(5,31,32)”
Background
The tree-based pipeline optimization tool: Tackling biomedical research problems with genetic programming and automated machine learning
Hernandez, J. G., Saini, A. K., Ghosh, A., & Moore, J. H. (2025) · Patterns · doi:10.1016/j.patter.2025.101314 · Cited by 10
“Furthermore, there is no guarantee that the available data accurately capture underlying patterns, as EHR data often contain inconsistencies and inaccuracies, 13,14 making it challenging for ML approaches to learn these patterns.”
Pharmacoepidemiology in the Era of Real-World Evidence
Toh, S. (2017) · Current Epidemiology Reports · doi:10.1007/s40471-017-0123-y · Cited by 11
“Electronic health recorddatabasesare enrichedwithmoreclinical information,but they generally do not have complete capture of events that occur across multiple health systems [17].”
Sociodemographic Characteristics and Mental and Physical Health Diagnoses of Yazidi Refugees Who Survived the Daesh Genocide and Resettled in Canada
Hassan, N., Coakley, A., AlMasri, I., Talavlikar, R., Aucoin, M. W., Grewal, R., Khalaf, A. K., Murad, S., McBrien, K., Ronksley, P. E., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.23064 · Cited by 10
Considerations for pooling real-world data as a comparator cohort to a single arm trial: a simulation study on assessment of heterogeneity
Backenroth, D., Royce, T. J., Pinheiro, J. C., Samant, M. K., & Humblet, O. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02002-7 · Cited by 10
“, differences in quality and organization of the datasets) [11, 12].”
Background
Database quality assessment in research in paramedicine: a scoping review
McDonald, N., Little, N., Kriellaars, D. J., Doupe, M. B., Giesbrecht, G. G., & Pryce, R. T. (2023) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/s13049-023-01145-2 · Cited by 11
“Electronic health records in all contexts have benefits and limitations, but all require consistent ways of describing, assessing, and integrating information about data quality [20–22] .”
Modelling Relevance towards Multiple Inclusion Criteria when Ranking Patients.
Limsopatham, N., Macdonald, C., & Ounis, I. (2014) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2661829.2661958 · Cited by 10
“Existing works on patient ranking mostly focus on estimating the relevance of the patients without considering how many of the inclusion criteria the retrieved patients cover.”
Background
Cross border semantic interoperability for clinical research: the EHR4CR semantic resources and services.
Daniel, C., Ouagne, D., Sadou, É., Forsberg, K., Gilchrist, M. M., Zapletal, É., Paris, N., Hussain, S., Jaulent, M., & Kalra, D. (2016) · PubMed Central · Cited by 10
Routinedaten – das ungenutzte Potenzial in der Versorgungsforschung
Djalali, S., Markun, S., & Rosemann, T. J. (2017) · Praxis · doi:10.1024/1661-8157/a002630 · Cited by 10
“Dies schmälert insbesondere die Verwendbarkeit von funktionalen Routinedaten für klinische Forschungsfragen [17].”
BackgroundInfluential
Identifying the “Blip on the Radar Screen”: Leveraging Big Data in Defining Drug Safety and Efficacy in Pediatric Practice
Christensen, M. L., & Davis, R. L. (2018) · The Journal of Clinical Pharmacology · doi:10.1002/jcph.1141 · Cited by 10
Validation of electronic medical data: Identifying diabetes prevalence in general practice
Henderson, J., Barnett, S., Ghosh, A., Pollack, A. J., Hodgkins, A. J., Win, K. T., Miller, G. C., & Bonney, A. D. (2018) · Health Information Management Journal · doi:10.1177/1833358318798123 · Cited by 10
Access to Routinely Collected Clinical Data for Research: A Process Implemented at an Academic Medical Center
Guerrero, S. C., Sridhar, S., Edmonds, C., Solis, C. F., Zhang, J., McPherson, D. D., & Bernstam, E. V. (2019) · Clinical and Translational Science · doi:10.1111/cts.12614 · Cited by 9
Self-citation
Comparative Effectiveness of Embedded Mental Health Services in Pain Management Clinics vs Standard Care
Gillman, A. G., Zhang, D., Jarquin, S., Karp, J. F., Jeong, J., & Wasan, A. D. (2019) · Pain Medicine · doi:10.1093/pm/pnz294 · Cited by 11
Preferences for Accessing Electronic Health Records for Research Purposes: Views of Parents Who Have a Child With a Known or Suspected Genetic Condition
Raspa, M., Paquin, R. S., Brown, D. S., Andrews, S. M., Edwards, A., Moultrie, R., Wagner, L., Frisch, M., Turner‐Brown, L., & Wheeler, A. (2020) · Value in Health · doi:10.1016/j.jval.2020.06.016 · Cited by 9
Assessment of a Clinical Trial–Derived Survival Model in Patients With Metastatic Castration-Resistant Prostate Cancer
Coquet, J., Bievre, N., Billaut, V., Seneviratne, M. G., Magnani, C. J., Bozkurt, S., Brooks, J. D., & Hernandez‐Boussard, T. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.31730 · Cited by 10
What Oncologists Want: Identifying Challenges and Preferences on Diagnosis Data Entry to Reduce EHR-Induced Burden and Improve Clinical Data Quality
Diaz‐Garelli, F., Strowd, R. E., III, Ahmed, T., Lycan, T. W., Daley, S., Wells, B. J., & Topaloĝlu, Ü. (2021) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00174 · Cited by 10
Exploring the Intersection Between Health Professionals’ Learning and eHealth Data: Protocol for a Comprehensive Research Program in Practice Analytics in Health Care
Janssen, A. L., Talic, S., Gašević, D., Kay, J., & Shaw, T. J. (2021) · JMIR Research Protocols · doi:10.2196/27984 · Cited by 14
“purposes rather than quality improvement or other secondary applications, and that the data often do not indicate a complete clinical story [14].”
Background
The importance of health insurance claims data in creating learning health systems: evaluating care for high-need high-cost patients using the National Patient-Centered Clinical Research Network (PCORNet)
Smith, M. A., Vaughan-Sarrazin, M., Yu, M., Wang, X., Nordby, P. A., Vogeli, C., Jaffery, J., & Metlay, J. (2019) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocz097 · Cited by 12
“This suggests that for some sites, concerted effort will be required to improve the completeness and reliability of EHR data through mechanisms such as participation in health information exchanges.(10) As suggested by Hersh et al, these efforts would be complemented by policies “calling for, mandating, or providing i…”
Background
Association Between End-Stage Liver Disease and Incident Heart Failure in an Integrated Health System
Chuzi, S. E., Tanaka, Y., Bavishi, A. A., Bruce, M., VanWagner, L. B., Wilcox, J. E., Ahmad, F. S., Ladner, D. P., Lagu, T. C., & Khan, S. S. (2023) · Journal of General Internal Medicine · doi:10.1007/s11606-023-08199-z · Cited by 9
“001 Self 913 (18) 790 (32) 123 (5) Medicare 856 (17) 333 (13) 523 (21) Medicaid 172 (3) 67 (3) 105 (4) Private 1961 (39) 959 (38) 1002 (40) Unknown 1102 (22) 353 (14) 749 (30) Index year 1.”
Identifying Symptoms Prior to Pancreatic Ductal Adenocarcinoma Diagnosis in Real-World Care Settings: Natural Language Processing Approach
Xie, F., Chang, J., Luong, T. Q., Wu, B. U., Lustigova, E., Shrader, E. E., & Chen, W. (2023) · JMIR AI · doi:10.2196/51240 · Cited by 8
“However, structured data can be inaccurate [11,12] and incomplete [13], especially for signs and symptoms.”
Background
Automating surgical procedure extraction for society of surgeons adult cardiac surgery registry using pretrained language models
Lee, J., Sharma, I., Arcaro, N., Blackstone, E. H., Gillinov, A. M., Svensson, L. G., Karamlou, T., & Chen, D. J. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae054 · Cited by 9
“16,17 For the latter, our institution has created a shared data model to map different versions of the registry into a single dataset.”
Method
Multi-domain rule-based phenotyping algorithms enable improved GWAS signal
Newbury, A., Elhussein, A., & Gürsoy, G. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01815-8 · Cited by 9
“Additionally, hospital billing practices can sometimes cause clinical concepts to be altered 5,6 .”
Assessing Different Approaches to Leveraging Historical Smoking Exposure Data to Better Select Lung Cancer Screening Candidates: A Retrospective Validation Study
Kats, D. J., Adie, Y., Tlimat, A., Greco, P. J., Kaelber, D. C., & Tarabichi, Y. (2020) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaa192 · Cited by 9
Clinical concept recognition: Evaluation of existing systems on EHRs
Lossio-Ventura, J. A., Sun, R., Boussard, S., & Hernandez‐Boussard, T. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.1051724 · Cited by 9
“Information from these unstructured data is sparse and conversion of these unstructured data to structured data is labor-intensive and expensive (Hersh et al., 2013).”
Background
A Framework for Leveraging “Big Data” to Advance Epidemiology and Improve Quality: Design of the VA Colonoscopy Collaborative
Gupta, S. N., Liu, L., Patterson, O. V., Earles, A., Bustamante, R., Gawron, A. J., Thompson, W. K., Scuba, W., Denhalter, D. W., Martı́nez, M. E., et al. (2018) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.198 · Cited by 9
“While the promise is real, extracting research-grade data from EHRs is a challenge [1].”
Background
Patient- and area-level predictors of prostate cancer among South Carolina veterans: a spatial analysis
Georgantopoulos, P., Eberth, J. M., Cai, B., Emrich, C. T., Rao, G., Bennett, C. L., Haddock, K. S., & Hébert, J. R. (2020) · Cancer Causes & Control · doi:10.1007/s10552-019-01263-2 · Cited by 9
Investigating Bias from Missing Data in an Electronic Health Records-Based Study of Weight Loss After Bariatric Surgery
Koffman, L. J., Levis, A. W., Arterburn, D. E., Coleman, K. J., Herrinton, L. J., Cooper, J., Ewing, J. J., Fischer, H. J., Fraser, J. R. P., Johnson, E., et al. (2021) · Obesity Surgery · doi:10.1007/s11695-021-05226-y · Cited by 9

Authors: Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., & Saltz, J. H.
Affiliations: Oregon Health & Science University (US)
AstraZeneca (United Kingdom) (GB)
The Ohio State University Wexner Medical Center (US)
The University of Texas Health Science Center at Houston (US)
Johns Hopkins University (US)
Columbia University (US)
Medical University of South Carolina (US)
National Institutes of Health (US)
Emory University (US)
Corresponding author: Hersh, William R.
Logan, Judith R.
Cimino, James J.
ORCID: Hersh, William R.
Weiner, Mark G.
Logan, Judith R.
Payne, Philip R.O.
Bernstam, Elmer V.
Lehmann, Harold P.
Hripcsak, George
Cimino, James J.
Saltz, Joel H.
Venue: Medical Care
Publisher: Ovid Technologies (Wolters Kluwer Health)
Published: 2013-08
Volume: 51
Issue: Supplement 8Suppl 3
Pages: S30-S37
ISSN: 0025-7079 (ISSN-L)
DOI: 10.1097/mlr.0b013e31829b1dbd
PMC: PMC3748381
OpenAlex: W2137767404
Cited by: 567
References: 68
Citations by year: 2026: 43 · 2025: 44 · 2024: 30 · 2023: 43 · 2022: 43 · 2021: 58 · 2020: 50 · 2019: 59 · 2018: 50 · 2017: 45 · 2016: 45 · 2015: 27 · 2014: 24 · 2013: 6
FWCI: 58.67
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Locations: Medical Care (publishedVersion)
PubMed (publishedVersion)
PubMed Central (acceptedVersion)
Funders: National Institutes of Health (UL1TR000454, UL1TR000090)
Centers for Disease Control and Prevention
National Center for Research Resources (UL1RR025008, UL1RR024156, UL1RR025005, UL1RR024148, UL1RR024140)
National Center for Advancing Translational Sciences
H2020 European Research Council
Topics: Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords: electronic health records, comparative effectiveness research, data quality, data completeness, data provenance, data granularity, data reuse
MeSH terms: Data Collection, Data Interpretation, Statistical, Humans, Insurance Claim Review, Research Design, Comparative Effectiveness Research, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks