The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research
Open accessAbstract
The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, a major concern is that data not collected systematically for research will be poorer quality, which could have negative impacts on findings generated from these data.24–27”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 2,127
The “Meaningful Use” Regulation for Electronic Health Records
Blumenthal, D., & Tavenner, M. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006114 · Cited by 2,059
A review of uses of health care utilization databases for epidemiologic research on therapeutics
Schneeweiß, S., & Avorn, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.012 · Cited by 1,290
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 1×
“A review of the literature of studies evaluating the data quality of EHRs for clinical research identified 5 dimensions of data quality assessed: completeness, correctness, concordance, plausibility, and currency.(26) The authors identified many studies with a wide variety of techniques to assess these dimensions and,…”
Measuring Diagnoses: ICD Code Accuracy
O’Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,156
Assessing Quality Using Administrative Data
Iezzoni, L. I. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00048 · Cited by 981
Next-generation phenotyping of electronic health records
Hripcsak, G., & Albers, D. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001145 · Cited by 795
Self-citationSafran, C., Bloomrosen, M., Hammond, W. E., Labkoff, S., Markel-Fox, S., Tang, P., & Detmer, D. (2007) · JAMIA Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2273 · Cited by 714
the eMERGE Team, McCarty, C. A., Chisholm, R. L., Chute, C. G., Kullo, I. J., Jarvik, G. P., Larson, E. B., Li, R., Masys, D. R., Ritchie, M. D., et al. (2011) · BMC Medical Genomics · doi:10.1186/1755-8794-4-13 · Cited by 701
Desiderata for Controlled Medical Vocabularies in the Twenty-First Century
Cimino, J. J. (1998) · Methods of Information in Medicine · doi:10.1055/s-0038-1634558 · Cited by 656
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Items 301–400 of 564
Moosavinasab, S., Sezgın, E., Sun, H., Hoffman, J., Huang, Y., & Lin, S. M. (2021) · ACI Open · doi:10.1055/s-0041-1729982 · Cited by 9
Gensheimer, M. F., Narasimhan, B., Henry, A. S., Wood, D., & Rubin, D. L. (2022) · JCO Clinical Cancer Informatics · doi:10.1200/cci.22.00019 · Cited by 9
Electronic Health Records in Dentistry: Clinical Challenges and Ethical Issues
Cederberg, R. A., Walji, M. F., & Valenza, J. (2014) · Health informatics · doi:10.1007/978-3-319-08973-7_1 · Cited by 9
Chaturvedi, R. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651256 · Cited by 8
“Not only are structured records plagued with inconsistent assignment of diagnostic codes [10], they also miss out recording a lot of pertinent information [1].”
BackgroundAzova, S., Laffel, L. M., Lennerz, B. S., Petty, C. R., Wolfsdorf, J. I., Rhodes, E. T., & Garvey, K. C. (2025) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2025.112041 · Cited by 9
“EMR data may be inaccurate, incomplete, or transformed in ways that may impact its interpretation and utility [48].”
Ruan‐Iu, L., Rivers, A. S., Barzilay, R., Moore, T. M., Tien, A. Y., & Diamond, G. S. (2022) · Archives of Suicide Research · doi:10.1080/13811118.2022.2106925 · Cited by 8
Breilyn, M. S., Kenny, E. E., & Abul‐Husn, N. S. (2022) · Molecular Genetics and Metabolism · doi:10.1016/j.ymgme.2022.106971 · Cited by 8
A Realism-Based View on Counts in OMOP's Common Data Model
Ceusters, W. M., & Blaisure, J. C. (2017) · Studies in health technology and informatics · doi:10.3233/978-1-61499-761-0-55 · Cited by 8
Chan, P. Y., Zhao, Y., Lim, S., Perlman, S. E., & McVeigh, K. H. (2018) · Preventing Chronic Disease · doi:10.5888/pcd15.180371 · Cited by 8
A novel method for clinical risk prediction with low-quality data
Wang, Z., Poon, J., Wang, S., Sun, S., & Poon, S. K.-S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102052 · Cited by 8
Electronic Health Record Network Research in Infectious Diseases
Jhaveri, R. R., John, J. D., & Rosenman, M. B. (2021) · Clinical Therapeutics · doi:10.1016/j.clinthera.2021.09.002 · Cited by 10
Gallardo‐Pizarro, A., Peyrony, O., Chumbita, M., Monzó, P., Aiello, T. F., Teijón‐Lumbreras, C., Gras, E., Mensa, J., Soriano, À., & García‐Vidal, C. (2024) · Expert Review of Anti-infective Therapy · doi:10.1080/14787210.2024.2322445 · Cited by 8
Chakrabarti, S., Sen, A. K., Huser, V., Hruby, G. W., Rusanov, A., Albers, D. J., & Weng, C. (2017) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-017-0005-6 · Cited by 8
“There are often severely limiting data quality issues with the EHR, such as missing values, sampling biases, and incorrectness [52, 53], and this invariably implies that a certain margin of error exists in research using EHRs.”
BackgroundA Data Element-Function Conceptual Model for Data Quality Checks
Rogers, J. R., Callahan, T. J., Kang, T., Bauck, A. E., Khare, R., Brown, J. S., Kahn, M. G., & Weng, C. (2019) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.289 · Cited by 9
Shoenbill, K., Song, Y.-Q., Craven, M., Johnson, H., Smith, M. A., & Mendonça, E. (2020) · Preventive Medicine · doi:10.1016/j.ypmed.2020.106061 · Cited by 9
“( 49) Despite difficulties, multiple studies have effectively used EHR data for many clinical research purposes if data are found to be sufficient for the task.”
BackgroundNarindrarangkura, P., Kim, M. S., & Boren, S. A. (2021) · ACI Open · doi:10.1055/s-0041-1735470 · Cited by 8
Gui, X., Lv, H., Wang, X., Lv, L., Xiao, Y., & Wang, L. (2025) · BioData Mining · doi:10.1186/s13040-025-00458-5 · Cited by 10
“Clinicians must navigate extensive electronic health records (EHRs) to identify patients with specific diagnoses or treatments [2], such as esophageal varices or those undergoing Transjugular Intrahepatic Portosystemic Shunt (TIPS).”
Implementation of a Maternal Child Knowledgebase.
Santillan, D. A., Santillan, M. K. O., Davis, H., Crooks, M., Flanagan, P. J., Ortman, C., Faro, E. Z., Hunter, S. K., & Knosp, B. M. (2022) · PubMed Central · Cited by 7
Pre-hospital peripheral intravenous catheter insertion practice: An integrative review
Golling, E., van de Mortel, T., Barr, N., & Zimmerman, P.-A. P. (2022) · Australasian Emergency Care · doi:10.1016/j.auec.2022.08.006 · Cited by 7
Healthcare Transformation: The Electronic Health Record
Edberg, D. T., & Wendel, J. (2018) · Springer eBooks · doi:10.1007/978-3-319-93003-9_7 · Cited by 7
In the room where it happens: do physicians need feedback on their real-world communication skills?
Zabar, S. R., Hanley, K., Wilhite, J. A., Altshuler, L. A., Kalet, A. L., & Gillespie, C. C. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2019-010384 · Cited by 7
Optimal multiwave validation of secondary use data with outcome and exposure misclassification
Lotspeich, S. C., Amorim, G., Shaw, P. A., Tao, R., & Shepherd, B. E. (2023) · Canadian Journal of Statistics · doi:10.1002/cjs.11772 · Cited by 7
Using the Electronic Health Record to Enhance Care in Pediatric Rheumatology.
Taxter, A., & Natter, M. (2022) · Rheumatic Disease Clinics of North America · doi:10.1016/j.rdc.2021.08.004 · Cited by 4
Sun, M. T., Singh, K., & Wang, S. Y. (2022) · Journal of Glaucoma · doi:10.1097/ijg.0000000000002122 · Cited by 7
“29 Within the limits of our automated pipeline, we were unable to extract additional clinical details, such as speci fi c clinical exam fi ndings or target IOPs which may have provided additional valuable insights.”
MethodA valuable approach to the use of electronic medical data in primary care research: Panning for gold
Barnett, S., Henderson, J., Hodgkins, A. J., Harrison, C. M., Ghosh, A., Dijkmans-Hadley, B. R., Britt, H. C., & Bonney, A. D. (2016) · Health Information Management Journal · doi:10.1177/1833358316669888 · Cited by 7
“However, there are still many caveats regarding the use of EMD, which may influence the validity of research using this technique (Hersh et al., 2013).”
BackgroundMethodAdvanced Research and Data Methods in Women's Health
Macedonia, C. R., Johnson, C. T., & Rajapakse, I. (2017) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000001865 · Cited by 7
Rutzner, S., Fietkau, R., Ganslandt, T., Prokosch, H., & Łubgan, D. (2017) · Frontiers in Oncology · doi:10.3389/fonc.2017.00016 · Cited by 7
“Furthermore, EHRs often do not tell a complete patient story, whether it may be those of a single institution or those aggregated across institutions (23).”
BackgroundConsumer-Mediated Data Exchange for Research: Current State of US Law, Technology, and Trust
Bracha, Y., Bagwell, J., Furberg, R., & Wald, J. S. (2019) · JMIR Medical Informatics · doi:10.2196/12348 · Cited by 7
“The paper does not address data completeness, provenance, harmonization, and other utility barriers in using EHR data for research, as these have been documented elsewhere [1,8,30-33].”
Gallis, J. A., Kusibab, K., Egger, J. R., Olsen, M. K., Askew, S., Steinberg, D., & Bennett, G. G. (2020) · Obesity · doi:10.1002/oby.22958 · Cited by 7
“EHR data are used to enroll and characterize study participants as well as to assess study end points (2,3,5).”
BackgroundMethodShah, R. U., Mutharasan, R. K., Ahmad, F. S., Rosenblatt, A., Gay, H. C., Steinberg, B. A., Yandell, M. D., Tristani‐Firouzi, M., Klewer, J., Mukherjee, R., et al. (2020) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.006516 · Cited by 7
Organic generation of real-world real-time data for clinical evidence in radiation oncology
Bertolet, A., Wals-Zurita, A., Miras, H., & Macías, J. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104301 · Cited by 7
Embí, P. J., Payne, P., & Friedman, C. P. (2021) · Learning Health Systems · doi:10.1002/lrh2.10302 · Cited by 7
“These limitations have manifested themselves in a number of recent scientific studies that have had to be retracted or adjusted due to incorrect assumptions and conclusions.(5,6) One thing is clear: The complexities of leveraging and learning from data require careful attention to several key principles in order to av…”
BackgroundSelf-citationMorales, D. J. V., Bansal, V., Heavner, S. F., Castro, J. C., Sharma, M., Tekin, A., Bogojevic, M., Zec, S., Sharma, N., Cartin-Ceba, R., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1089087 · Cited by 7
Uncertainties in outcome modelling in radiation oncology
Dünger, L., Mäusel, E., Zwanenburg, A., & Löck, S. (2025) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2025.100774 · Cited by 7
“Fourthly, data may be partly missing or incomplete for part of the patients [69,76].”
The Ethics of Leveraging Routinely Collected Patient Data for AI Development: Mixed Methods Study
Maris, M., Klopotowska, J. E., Ronald Cornet, A., van den Hoven, M., Lieverse, J. E., Fernández-Llaneza, D., & Bak, M. (2025) · Journal of Medical Internet Research · doi:10.2196/79863 · Cited by 7
“Because EHR data are collected during routine care for clinical purposes, they often have quality limitations when reused for research, lacking the consistency required by clinical research standards [8,9].”
Cox regression is robust to inaccurate EHR-extracted event time: an application to EHR-based GWAS
Irlmeier, R. T., Hughey, J., Bastarache, L. A., Denny, J. C., & Chen, Q. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac086 · Cited by 7
Lemas, D. J., Du, X., Rouhizadeh, M., Lewis, B., Frank, S., Wright, L., Spirache, A., Gonzalez, L., Cheves, R., Magalhães, M., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-58299-x · Cited by 8
Miller, M. R., Morris, R., Fisicaro, N., & Curtis, K. A. (2017) · Emergency Medicine Australasia · doi:10.1111/1742-6723.12817 · Cited by 6
“ICU days (median [IQR]) 2 (4) 4 (8) 3 (9) 0.”
Healthcare data classification — Cloud-based architecture concept
Miškuf, M., Zolotová, I., & Mocnej, J. (2018) · Cybernetics & Informatics (K&I) · doi:10.1109/cyberi.2018.8337557 · Cited by 6
Recruitment and Retention of Patients for Nonalcoholic Steatohepatitis Clinical Trials
DeLegge, M. H. (2020) · Gastroenterology Clinics of North America · doi:10.1016/j.gtc.2019.09.006 · Cited by 6
Shirai, C., Tsuda, S., Tarasawa, K., Fushimi, K., Fujimori, K., & Nakazawa, T. (2021) · BMC Ophthalmology · doi:10.1186/s12886-021-01897-4 · Cited by 6
Frailty and In-Hospital Mortality Risk Using EHR Nursing Data
Lekan, D. A., McCoy, T. P., Jenkins, M., Mohanty, S. D., & Manda, P. (2021) · Biological Research For Nursing · doi:10.1177/10998004211060541 · Cited by 6
“Threats to data quality such as missing, erroneous, uninterpretable, and inconsistent data require advanced data processing and analytical techniques (Hersh et al., 2013).”
BackgroundMethodData-Driven and Practice-Based Evidence
Osop, H., & Sahama, T. R. (2015) · Advances in healthcare information systems and administration book series · doi:10.4018/978-1-4666-9432-3.ch014 · Cited by 6
McDermott, S. P., & Wasan, A. D. (2023) · Journal of Pain Research · doi:10.2147/jpr.s389160 · Cited by 6
“Ideally, collection and storage of data should be carefully considered prospectively to avoid these problems and to complement research investigations.(4) However, this can be difficult to achieve prospectively and even more so for the large amounts of data that currently exist.”
BackgroundCharacterizing Secondary Use of Clinical Data.
Lee, E. S., Black, R. A., Harrington, R. D., Tarczy-Hornoch, P. Z., & Facmi (2015) · Europe PMC (PubMed Central) · Cited by 6
Efficiency and effectiveness evaluation of an automated multi-country patient count cohort system
Soto‐Rey, I., Trinczek, B., Girardeau, Y., Zapletal, É., Ammour, N., Doods, J., Dugas, M., & Fritz, F. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0035-9 · Cited by 8
“These reasons are similar to the ones found by Hersh [22], who also provides possible solutions to them.”
ResultMor Peleg, B., Leung, T. I., Desai, M., & Dumontier, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-59758-4_27 · Cited by 6
“However, Hersh et al. [4] note that “EHR data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research [e.g., found in textual notes], of insufficient granularity, and incompatible with research protocols” [i.e., treatment…”
BackgroundOpportunities for the use of health information technology to support research
Zayas‐Cabán, T., & Wald, J. S. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa037 · Cited by 7
Fatollahi, J. J., Bentley, S., Doran, N., & Brody, A. L. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182211923 · Cited by 7
“A known issue with multisite EMR-based research is that it can amplify challenges of using EMR data, such as cross-institutional differences in data capture and workflows, use of standards, and missingness [66].”
BackgroundOntological Foundations for Tracking Data Quality through the Internet of Things
Ceusters, W. M., & Bona, J. P. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-633-0-74 · Cited by 6
A Short Review of Ethical Challenges in Clinical Natural Language Processing
Suster, S., Tulkens, S., & Daelemans, W. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1610 · Cited by 6
“…patient story (e.g. due to time constraints or due to patient treatment in different hospitals), yet important decisions can be based on them.4 As language is situated, a lot of information may be implicit, such as the circumstances in which treatment decisions are made (Hersh et al., 2013).”
BackgroundKeim, S. K., & Bowles, K. H. (2017) · PubMed Central · Cited by 5
Bias reduction methods for propensity scores estimated from error-prone EHR-derived covariates
Harton, J. G., Mamtani, R., Mitra, N., & Hubbard, R. A. (2020) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-020-00219-3 · Cited by 5
Appenzeller, A., Bartholomäus, S., Breitschwerdt, R., Claussen, C., Geisler, S., Hartz, T., Kachel, P., Krempel, E., Robert, S., & Zeißig, S. R. (2021) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa53542.2021.9686918 · Cited by 9
Hammill, B. G., Leimberger, J. D., Lampron, Z. M., Raman, S. R., O'Brien, E. C., Wurst, K., Mountcastle, S. B., Cunnington, M. C., Janmohamed, S. G., & Curtis, L. H. (2022) · Clinical Trials · doi:10.1177/17407745221114298 · Cited by 7
Implementation of Personalized Medicine by Artificial Intelligence Platform
Yakimenko, Y., Stirenko, S., Koroliouk, D., Gordienko, Y. G., & Zanzotto, F. M. (2022) · Advances in intelligent systems and computing · doi:10.1007/978-981-19-3590-9_46 · Cited by 5
Kim, M. J., Kim, H. J., Kang, D., Ahn, H. K., Shin, S.-Y., Park, S., Cho, J., & Park, Y. H. (2023) · Journal of Medical Internet Research · doi:10.2196/43359 · Cited by 5
“As a result, local information system expertise and deep content knowledge are often required to understand the idiosyncratic manner in which data sources are captured and stored [17,30].”
Priou, S., Lamé, G., Janković, M., Châtellier, G., Bey, R., Tournigand, C., Daniel, C., & Kempf, E. (2023) · Studies in health technology and informatics · doi:10.3233/shti230103 · Cited by 5
“However, EHR data are not without flaws, and caveats and recommendations have been raised when reusing them for research [2,3].”
BackgroundDevelopment and Evaluation of a Similarity Measure for Medical Event Sequences
Mannino, M. V., Fredrickson, J., Banaei‐Kashani, F., Linck, I., & Alqurashi, R. (2017) · ACM Transactions on Management Information Systems · doi:10.1145/3070684 · Cited by 5
“Data warehouses can provide economies of scale to support comparative effectiveness research (CER), which aims to study populations and clinical outcomes of maximal pertinence to real-world clinical practice [Hersh et al. 2013].”
BackgroundVemulakonda, V. M., Bush, R. A., & Kahn, M. G. (2018) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2018.04.033 · Cited by 6
“Finally, working with data collected for clinical uses rather than research may lack the temporal relationship needed to determine disease cause [14].”
BackgroundMethodInfluentialMcDonald, L., Sammon, C., Carroll, R., Harish, A., Abhishek Shankar, C., Tyas, D., & Ramagopalan, S. V. (2019) · Future Oncology · doi:10.2217/fon-2019-0669 · Cited by 5
Crume, T., Duca, L. M., Ong, T. C., Kraus, E. M., Scott, K., Khanna, A. D., Kao, D. P., Rausch, C. M., McKenzie, L. M., Daley, M. F., et al. (2020) · American Heart Journal · doi:10.1016/j.ahj.2020.04.008 · Cited by 5
Gundler, C., Gottfried, K., Wiederhold, A. J., Ataian, M., Wurlitzer, M., Gewehr, J. E., & Ückert, F. (2024) · Interactive Journal of Medical Research · doi:10.2196/51563 · Cited by 8
“Filtering cohorts based on specific criteria may yield statistically inappropriate sizes, and requested data may be inadequately recorded [11].”
BackgroundUmberger, R. A., Indranoi, C., Simpson, M. H., Jensen, R., Shamiyeh, J. E., & Yende, S. P. (2019) · SAGE Open Nursing · doi:10.1177/2377960819850972 · Cited by 5
“Clinical EHR data, like administrative data sets, are not created for research purposes and present challenges (Hersh et al., 2013).”
BackgroundPrimary care risk stratification in COPD using routinely collected data: a secondary data analysis
Johnson, M. J., Rigge, L. A., Culliford, D., Josephs, L., Thomas, M., & Wilkinson, T. (2019) · npj Primary Care Respiratory Medicine · doi:10.1038/s41533-019-0154-6 · Cited by 5
Business Rules to Improve Secondary Data Use of Electronic Healthcare Systems
Blaisure, J. C., & Ceusters, W. M. (2017) · Studies in health technology and informatics · doi:10.3233/978-1-61499-753-5-303 · Cited by 5
Secondary Use of EHR: Interpreting Clinician Inter-Rater Reliability Through Qualitative Assessment
Mullin, S., Anand, E., Sinha, S., Song, B., Zhao, J. Y., & Elkin, P. (2017) · Context Sensitive Health Informatics · doi:10.3233/978-1-61499-794-8-165 · Cited by 3
“A systematic review showed that out of 98 studies, 55% of the studies used additional non-EHR sources of data and 40% of the studies supplemented EHR data with patient-reported data [6].”
BackgroundQuality Assessment of Healthcare Databases
Carle, F., Minco, L. D., Skrami, E., Gesuita, R., Palmieri, L., Giampaoli, S., & Corrao, G. (2022) · Epidemiology Biostatistics and Public Health · doi:10.2427/12901 · Cited by 6
Salvatore, M., Kundu, R., Du, J., Friese, C. R., Mondul, A. M., Hanauer, D. A., Lu, H., Pearce, C. L., & Mukherjee, B. (2024) · medRxiv · doi:10.1101/2024.10.28.24316286 · Cited by 8
Powers, J., Nandhakumar, S., Dard, S., Kovach, P. I., & Leese, P. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf060 · Cited by 4
Healthcare quality measures in implementation research: advantages, risks and lessons learned
Gustavson, A. M., Hagedorn, H., Jesser, L. E., Kenny, M. E., Clothier, B. A., Bounthavong, M., Ackland, P. E., Gordon, A. J., & Harris, A. H. S. (2022) · Health Research Policy and Systems · doi:10.1186/s12961-022-00934-y · Cited by 4
“Moreover, this process requires additional validation steps to ensure that the measures accurately represent the construct and are reliable throughout the study period [17].”
Pulia, M. S., Schwei, R. J., Patterson, B. W., Repplinger, M. D., Smith, M. A., & Shah, M. N. (2018) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2018.06.036 · Cited by 4
“To validate the extracted data, we employed multiple best practice strategies for the use of operational EHR data in comparative effectiveness research.[18,19] This includes comparing to a gold standard data source by creating a data subset of patients that had health care claims data available and compared each ED en…”
MethodZhang, Y., Tayarani, M., Alaref, S. J., Beecy, A., Liu, Y., Sholle, E., RoyChoudhury, A., Axsom, K., Gao, H. O., Pathak, J., et al. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa038 · Cited by 4
Pan, Y., Laber, E. B., Smith, M. A., & Zhao, Y. (2021) · Journal of the American Statistical Association · doi:10.1080/01621459.2021.1978467 · Cited by 4
“These issues are well-documented in the literature on EHR data (Hripcsak et al. 2011; Hersh et al. 2013; Flood et al. 2015; McVeigh et al. 2016).”
BackgroundDatabase quality assessment in research in paramedicine: a scoping review protocol
McDonald, N., Kriellaars, D. J., Doupe, M. B., Giesbrecht, G. G., & Pryce, R. T. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-063372 · Cited by 5
Bae, W. K., Cho, J.-H., Kim, S., Kim, B., Baek, H., Song, W., & Yoo, S. (2022) · JMIR Medical Informatics · doi:10.2196/41503 · Cited by 4
“Lastly, observational research that uses EHR data has the limitation that it cannot fully capture the entirety of a patient's health information [21].”
BackgroundKurniadi, A., & Pratiwi, R. (2017) · Jurnal Kesehatan Masyarakat · doi:10.15294/kemas.v13i2.8103 · Cited by 4
Carlson, J. T., & Laryea, J. A. (2019) · Clinics in Colon and Rectal Surgery · doi:10.1055/s-0038-1673358 · Cited by 5
“23,31”
BackgroundMethodInfluentialBig Data: Knowledge Discovery and Data Repositories
Velupillai, S. U., Davis, K. A. S., & Rozenblit, L. (2021) · Health informatics · doi:10.1007/978-3-030-70558-9_15 · Cited by 4
Appelbaum, L., Kaplan, I. D., Palchuk, M. B., Kundrot, S., Winer-Jones, J. P., & Rinard, M. (2022) · Exon Publications eBooks · doi:10.36255/exon-publications-digital-health-federated-databases · Cited by 4
“, in the physician’s office within the electronic patient medical record, deals upfront with some of the challenges inherent to this type of data source and helps overcome these challenges during the development and testing stages (12, 13).”
BackgroundSultan, S., Acharya, Y., Zayed, O., Elzomour, H., Parodi, J. C., Soliman, O. I. I., & Hynes, N. (2022) · Journal of Endovascular Therapy · doi:10.1177/15266028221102660 · Cited by 4
Davis, H., Tang, L. A., Picou, E. M., Bastarache, L. A., & Tharpe, A. M. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2023_jslhr-22-00610 · Cited by 4
Weiskopf, N. G., Khan, F. J., Woodcock, D., Dorr, D. A., Cigarroa, J. E., & Cohen, A. M. (2016) · PubMed Central · Cited by 4
Rojas, J. C., Lyons, P. G., Chhikara, K., Chaudhari, V., Bhavani, S. V., Nour, M., Buell, K. G., Smith, K. D., Gao, C. A., Amagai, S., et al. (2024) · medRxiv · doi:10.1101/2024.09.04.24313058 · Cited by 4
Predicting Self‐Reported Social Risk in Medically Complex Adults Using Electronic Health Data
Grant, R. W., McCloskey, J. K., Uratsu, C. S., Ranatunga, D., Ralston, J. D., Bayliss, E. A., & Sofrygin, O. (2024) · Medical Care · doi:10.1097/mlr.0000000000002021 · Cited by 4
“38 With these concerns in mind, we applied the following strategies 39 to minimize biased data and overreliance on automation: (1) we involved clinical stakeholders to help de fi ne the clinical application of the planned predictive model (eg, to support new work fl ows that use prediction results to…”
Cortina, J. M. A., Fatapour, Y., Brown, K. L., Gisladottir, U., Zietz, M., Walk, O. J. B. D., IV, Peter, D., Berkowitz, J., Friedrich, N., Kivelson, S., et al. (2024) · medRxiv · doi:10.2196/67591 · Cited by 3
“It includes both the presence and accuracy of essential data elements, such as diagnoses, treatments, laboratory results, and preventive care measures, ensuring a comprehensive and reliable foundation for clinical care, research, and quality assessment [12-14].”
Automation and Workflow Efficiency in Clinical Settings
Farrukh, N., Chandawarkar, A. R., Vundavalli, S., & Taxter, A. J. (2025) · Medical Clinics of North America · doi:10.1016/j.mcna.2025.07.001 · Cited by 3
Heesen, P., Feusi, D., Vogel, B., Studer, G., Fuchs, B., & on behalf of the Swiss Sarcoma Network (2025) · Cancers · doi:10.3390/cancers17182944 · Cited by 3
“Adequate monitoring and instant access to respective data are therefore critical [27–29].”
Janssen, A. L., Coggins, A., Tadros, J. H., Quinn, D., Shetty, A., & Shaw, T. J. (2025) · JMIR Medical Education · doi:10.2196/65287 · Cited by 4
“A considerable limitation is that there can be gaps in the data and that the data are not captured for the primary purpose of education [23], and there are limitations on how these data can be used to understand performance [19].”
Patterns and Treatment Timelines of Tuberculosis in Males From Ecuador’s Largest Prison
Romero-Sandoval, N. C., Torres, M., Solis, V., Orellana, F., Castells, P. À., Andrade, D., Sánchez‐Pérez, H. J., Jiménez, P., Millet, J.-P., Caylà, J. A., et al. (2025) · Journal of Correctional Health Care · doi:10.1177/10783458251380507 · Cited by 3
“Regarding the use of administrative records, Hersh et al. (2013) highlight the potential for comparative effectiveness research, but also caution about the accuracy and completeness of data.”
Validation of 4 Risk Stratification Tools for Delirium in the Emergency Department
Bartolacci, M., Carpenter, K. P., Jeffery, M. M., Mullan, A. F., Carpenter, C. R., & Bellolio, F. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.40920 · Cited by 3
Baierl, J., Hsiao, Y.-W., Jones, M. R., Peng, P.-C., & Pharoah, P. D. P. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf234 · Cited by 3
Burkhardt, H. A., Blach, S., Burugapalli, S. R., Cartwright, B. M. G., Martin, R., Simonov, M., Stewart, S. E., Turner, G., Winegar, A. L., & Stucky, N. L. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag142 · Cited by 3
Precision Oncology with Electronic Medical Records
Nayak, L., & De, R. K. (2019) · EMJ Innovation · doi:10.33590/emjinnov/10313168 · Cited by 3
Jha, J., Almagro, M., & Tissot, H. C. (2023) · Journal of Digital Health · doi:10.55976/jdh.22023119463-81 · Cited by 3
A Health Information Technology Protocol to Enhance Colorectal Cancer Screening
Baus, A., Boatman, D., Calkins, A., Pollard, C. R., Conn, M. E., Subramanian, S., & Kennedy-Rea, S. K. (2024) · JMIR Formative Research · doi:10.2196/55202 · Cited by 3
Chen, Y., Xiaoli, W., Chen, M., Song, Q., Jia, W. P., Li, X., Wen, Z., & Li, N. (2017) · doi:10.1145/3135954.3135962 · Cited by 3
“However, conducting studies on clinical data can be difficult, as under most circumstances, critical information is contained in free-texts and the vocabulary used is usually not standard nor designed with the needs of researchers in mind [2].”
BackgroundMediating knowledge across health ecosystems: a qualitative field study
Maženytė, B., & Petraitė, M. (2019) · Measuring Business Excellence · doi:10.1108/mbe-09-2019-0099 · Cited by 3
The Role of Machine Learning in the Healthcare Sector: A Roadmap to the Potential Prospects
Xhaferra, E., & Ismaili, F. (2022) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/hora55278.2022.9800065 · Cited by 3
Artificial intelligence in healthcare
Argenson, A., & Devi-Chou, V. (2023) · Principles of Gender-Specific Medicine · doi:10.1016/b978-0-323-88534-8.00025-0 · Cited by 3
Metadata
- Authors
- Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., & Saltz, J. H.
- Affiliations
- Oregon Health & Science University (US)
- AstraZeneca (United Kingdom) (GB)
- The Ohio State University Wexner Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Johns Hopkins University (US)
- Columbia University (US)
- Medical University of South Carolina (US)
- National Institutes of Health (US)
- Emory University (US)
- Corresponding author
- Hersh, William R.
- Logan, Judith R.
- Cimino, James J.
- ORCID
- Venue
- Medical Care
- Published
- 2013-08
- Volume
- 51
- Issue
- Supplement 8Suppl 3
- Pages
- S30-S37
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC3748381
- OpenAlex
- W2137767404
- Cited by
- 567
- References
- 68
- Citations by year
- 2026: 43 · 2025: 44 · 2024: 30 · 2023: 43 · 2022: 43 · 2021: 58 · 2020: 50 · 2019: 59 · 2018: 50 · 2017: 45 · 2016: 45 · 2015: 27 · 2014: 24 · 2013: 6
- FWCI
- 58.67
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000454, UL1TR000090)
- Centers for Disease Control and Prevention
- National Center for Research Resources (UL1RR025008, UL1RR024156, UL1RR025005, UL1RR024148, UL1RR024140)
- National Center for Advancing Translational Sciences
- H2020 European Research Council
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- electronic health records, comparative effectiveness research, data quality, data completeness, data provenance, data granularity, data reuse
- MeSH terms
- Data Collection, Data Interpretation, Statistical, Humans, Insurance Claim Review, Research Design, Comparative Effectiveness Research, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
