The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research
Open accessAbstract
The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, a major concern is that data not collected systematically for research will be poorer quality, which could have negative impacts on findings generated from these data.24–27”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 2,127
The “Meaningful Use” Regulation for Electronic Health Records
Blumenthal, D., & Tavenner, M. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006114 · Cited by 2,059
A review of uses of health care utilization databases for epidemiologic research on therapeutics
Schneeweiß, S., & Avorn, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.012 · Cited by 1,290
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 1×
“A review of the literature of studies evaluating the data quality of EHRs for clinical research identified 5 dimensions of data quality assessed: completeness, correctness, concordance, plausibility, and currency.(26) The authors identified many studies with a wide variety of techniques to assess these dimensions and,…”
Measuring Diagnoses: ICD Code Accuracy
O’Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,156
Assessing Quality Using Administrative Data
Iezzoni, L. I. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00048 · Cited by 981
Next-generation phenotyping of electronic health records
Hripcsak, G., & Albers, D. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001145 · Cited by 795
Self-citationSafran, C., Bloomrosen, M., Hammond, W. E., Labkoff, S., Markel-Fox, S., Tang, P., & Detmer, D. (2007) · JAMIA Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2273 · Cited by 714
the eMERGE Team, McCarty, C. A., Chisholm, R. L., Chute, C. G., Kullo, I. J., Jarvik, G. P., Larson, E. B., Li, R., Masys, D. R., Ritchie, M. D., et al. (2011) · BMC Medical Genomics · doi:10.1186/1755-8794-4-13 · Cited by 701
Desiderata for Controlled Medical Vocabularies in the Twenty-First Century
Cimino, J. J. (1998) · Methods of Information in Medicine · doi:10.1055/s-0038-1634558 · Cited by 656
Self-citation
Cited by
Sorted by citations
Items 501–564 of 564
Hummel, L. M., Lorenz-Artz, K., Bierbooms, J. J. P. A., & Bongers, I. M. B. (2025) · doi:10.2196/preprints.74240
Quality of primary total knee arthroplasty operative reports in a tertiary teaching hospital
Almousa, S. A. (2025) · World Journal of Orthopedics · doi:10.5312/wjo.v16.i5.104438
McLeod, D., Martins, I., Tinker, A. V., Selk, A., Brezden‐Masley, C., LeVasseur, N., & Altman, A. D. (2025) · Therapeutic Advances in Medical Oncology · doi:10.1177/17588359251339919
“97 Medical billing data are subject to biases and inaccuracies, 98–101 especially when conducted by support staff such as students and nurses, and such billing errors may be more frequent in systems that are rarely audited, such as OHIP.”
Lay, S., Nguyen, L. L., Yan, Y., Chan‐Golston, A. M., Santiago, A. D., Sidhu, R., & Yasaei, R. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-092122
The Development and Progress of Health Information Technology in Iran
Balaghafari, A., & Siamian, H. (2025) · Health Science Reports · doi:10.1002/hsr2.71106
“On the other hand, obstacles like clinician readiness [13] and clinical workflow integration [14] impede EHRs' full benefits.”
Baierl, J., Hsiao, Y.-W., Jones, M. R., Peng, P.-C., & Pharoah, P. D. P. (2025) · medRxiv · doi:10.1101/2025.07.30.25332455
Aktypi, F. D., Benros, M. E., & Rasmussen, S. (2026) · BMC Psychiatry · doi:10.1186/s12888-026-08373-7
“…study explores whether using large real-world EHRs and established explainable machine learning algorithms, can capture predictive signal for identifying individuals at higher risk for significant weight gain when treated with olanzapine while at the same time highlighting risk associations [18].”
Luo, J., Zhang, A., Gao, W., Bu, X., Li, R., Ye, Z., Zhang, X., Ren, L., Yin, J., Guan, Z., et al. (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70079
Cren, P., Leguillette, C., Craynest, F., Hammoudi, A., Barthoulot, M., Carton, M., Filleron, T., Chabaud, S., Drouet, Y., & Deley, M. L. (2025) · European Journal of Epidemiology · doi:10.1007/s10654-025-01278-x
Magoč, T., Tang, L. A., Nguyen, K. A., & Harle, C. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf188
Multimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Classifying polyneuropathy and myopathy patients on Electronic Health RecordsPreprint
Ahmed, M. M., Truong, N. D. K., Nyoungui, E. F., Zhao, J., Wedemeyer, H., Mayer, R. L., Schuster, V., Zschüntzsch, J., & Röttger, R. (2025) · medRxiv · doi:10.64898/2025.12.11.25342051
Schmaling, K. B., & Kaplan, R. M. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009391733.009
Additional file 2 of Database quality assessment in research in paramedicine: a scoping review
McDonald, N., Little, N., Kriellaars, D. J., Doupe, M. B., Giesbrecht, G. G., & Pryce, R. T. (2024) · Figshare · doi:10.6084/m9.figshare.26638662
Backenroth, D., Royce, T., Pinheiro, J., Samant, M. K., & Humblet, O. (2023) · Figshare · doi:10.6084/m9.figshare.24030852
John, L. H., Fridgeirsson, E. A., Kors, J. A., Reps, J. M., Williams, R. D., Ryan, P. B., & Rijnbeek, P. R. (2024) · Figshare · doi:10.6084/m9.figshare.26401840
Additional file 3 of Database quality assessment in research in paramedicine: a scoping review
McDonald, N., Little, N., Kriellaars, D. J., Doupe, M. B., Giesbrecht, G. G., & Pryce, R. T. (2024) · Figshare · doi:10.6084/m9.figshare.26638665
Shirai, C., Tsuda, S., Tarasawa, K., Fushimi, K., Fujimori, K., & Nakazawa, T. (2021) · Figshare · doi:10.6084/m9.figshare.14337640
O’Brien, E. C., Raman, S. R., Ellis, A. M., Hammill, B. G., Berdan, L. G., Rorick, T. L., Janmohamed, S. G., Lampron, Z. M., Hernandez, A. F., & Curtis, L. H. (2021) · Figshare · doi:10.6084/m9.figshare.15020753
Additional file 2 of Population segmentation based on healthcare needs: a systematic review
Chong, J. L., Lim, K. H., & Matchar, D. B. (2019) · Figshare · doi:10.6084/m9.figshare.9599996
Additional file 1 of Database quality assessment in research in paramedicine: a scoping review
McDonald, N., Little, N., Kriellaars, D. J., Doupe, M. B., Giesbrecht, G. G., & Pryce, R. T. (2024) · Figshare · doi:10.6084/m9.figshare.26638659
Additional file 1 of Population segmentation based on healthcare needs: a systematic review
Chong, J. L., Lim, K. H., & Matchar, D. B. (2019) · Figshare · doi:10.6084/m9.figshare.9599993
Meier, F., Probst-Hensch, N. M., Höglinger, M., Schwenkglenks, M., & Wieser, S. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13868-2
“The data could be influenced by misclassifications, recording bias, incorrect entries, incompleteness of records or even effects of billing incentives, as this is typical in real-world studies [37, 38].”
Early Adopters of AI and Gender Disparities in Health Outcomes
Singh, V. (2025) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83708-022-920261025
अथरववदसय ऐतहसकसवरपम- एक वशलषणम
मौर्य, ड. व. क. म. ड. व. क., Soetan, K. T., & Peter-Anyebe, A. C. (2025) · Shodhshauryam International Scientific Refereed Research Journal · doi:10.32628/shisrrj225185
Wiseman, B., Li, H., Mason, E., Lonergan, K., Mitchell, R. E., & Hayward, J. (2026) · medRxiv · doi:10.64898/2026.01.13.26344087
Meier, F., Probst-Hensch, N. M., Höglinger, M., Schwenkglenks, M., & Wieser, S. (2025) · Zürcher Hochschule für Angewandte Wissenschaften digital collection (Zurich University of Applied Sciences) · doi:10.21256/zhaw-35447
Phiri, J., Zimba, A., Njovu, C., & Lumpa, M. (2026) · Research Square · doi:10.21203/rs.3.rs-9227225/v1
Radiomics of metastatic cell-clear renal cancer
Bouchouicha, A. (2018) · theses.fr (ABES) · doi:10.70675/3a5aaffdz3530z4603z8f41zb379a525d0da
de l’Aulnoit, A. H. (2019) · theses.fr (ABES) · doi:10.70675/8d01e7f3z52b2z422bzafeez3f3e7bc7bd32
Leveraging Automated Methods to Reduce Bias in Retrospective Chart Reviews of Paracentesis Outcomes
Ono, S. (2026) · Critical Care Medicine · doi:10.1097/ccm.0000000000006954
Public Health Information Systems
Nam, K., & Feldman, S. S. (2026) · Health informatics · doi:10.1007/978-3-032-16657-9_16
Siegler, K., Smith, J., Walker, E., Bu, J., Robson, P., & Shack, L. (2026) · Health Informatics Journal · doi:10.1177/14604582261440410
“5,9,10 One approach to tackling many of these challenges is the creation of a data warehouse.”
Wang, T., Lei, Y., Li, L., Wang, Y., Tang, H., Wu, Y., Chen, S., Zhang, B., Hu, J., Wang, H., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-9613167/v1
Vergani, E., Iacomini, C., Giampietro, A., Milardi, D., Chiloiro, S., Mancini, A., Marinis, L. D., Bianchi, A., Patarnello, S., & Pontecorvi, A. (2026) · Journal of Endocrinological Investigation · doi:10.1007/s40618-026-02918-z
Awadallah, C., Atrash, S., Paul, B. A., Khan, A., Shaikh, H. G., Strouse, C. S., Chahine, Z., Mazloom, A., Faisal, M. S., Al-Juhaishi, T. M. S., et al. (2026) · Expert Review of Hematology · doi:10.1080/17474086.2026.2681764
“…anchoring where feasible (e.g. from leukapheresis or treatment intent rather than infusion), explicit reporting of attrition between intended treatment and observed outcomes, sensitivity analyses, and integration of community-inclusive RWD platforms to broaden the source population [28,46,47].”
Fang, Y., Shi, L., Huang, J., Liu, C., & Xu, Z. (2026) · doi:10.2196/preprints.101344
Liver Infection Detection Using Machine Learning
Kumar, S., Kundu, M., Kumari, K., Purushottam, P., Kumar, V., & Kumar, R. G. A. (2026) · International Conference on Recent Trends in Computer Science and Technology (ICRTCST) · doi:10.1109/icrtcst68392.2026.11545423
“It is the variability in data quality and standardization across different healthcare institutions [33].”
É., E. E., Raimi, O., Bademosi, A., Wegbom, A. I., & Ordu, S. (2026) · International Journal of Public Health and Pharmacology · doi:10.52589/ijphp-khuzlfqv
Hennessy, C., Swartz, A. Z., Harrell, F. E., & Silver, H. J. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02881-6
Ye, L., Cao, X., Ren, T., Huang, H., Deng, G., & Fu, L. (2026) · doi:10.2196/preprints.102302
Salvatore, M., Kundu, R., Du, J., Friese, C. R., Mondul, A. M., Hanauer, D., Lu, H., Pearce, C. L., & Mukherjee, B. (2026) · Genetic Epidemiology · doi:10.1002/gepi.70047
Hansen, B. B., Leth, K., Hansen, N. R., Abrahamsen, B., Fuggle, N. R., Brønd, J. C., & Rubin, K. H. (2026) · Calcified Tissue International · doi:10.1007/s00223-026-01563-1
“Furthermore, while the use of administrative data could enable high scalability, we must also recognize that such data are typically collected for billing purposes and not research [51].”
Pradhan, A. M., Shetty, V. A., Gregor, C. M., Graham, J. H., Tusing, L. D., Hirsch, A. G., Hall, E., Troiani, V., Davis, M. P., Beiler, D. L., et al. (2026) · Medical Cannabis and Cannabinoids · doi:10.1159/000553307
“Lastly, data captured in the EHR remains susceptible to the general limitations of EHR data, such as inaccuracy, incompleteness, and insufficient granularity [70].”
Xie, F., Xiong, X., Mai, X., Zhou, S. H. E. N. G. H., Liu, Y., Yu, K., Xu, X., Lin, M., Zhang, R., & Puskarich, M. A. (2026) · Research Square · doi:10.21203/rs.3.rs-10020548/v1
Akinbode, A. K., Sava, M. G., May, J. H., & Pirrallo, R. G. (2026) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2026.2692706
Alqahtani, Z. A., Almadfaa, R. O., Garami, E., Xiong, X., Wigle, P. R., Huang, B., Hincapie, A. L., & Guo, J. (2026) · Advances in Therapy · doi:10.1007/s12325-026-03657-6
“Such delays reflect incomplete visit histories or care received outside the TriNetX network, as well as missing data that could inflate the mean [20, 21].”
Machine learning on asynchronous clinical pages to predict clinical deterioration
Arvelo, I., Shipley, K., Wright, A., & Steitz, B. D. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag122
Natural Language Processing to Identify Substance Use in Electronic Health Records: A Scoping Review
Azubuike, C. D., Farrag, A. N., Zandbiglari, K., Soliman, A. S. A., Bagchi, C., Mehrabi, F., Adkins, L., Rouhizadeh, M., & Goodin, A. J. (2026) · Substance Use & Addiction Journal · doi:10.1177/29767342261463377
“4,11 Compared to structured fields, clinical narratives are far more likely to capture substance use information in EHRs.”
Marcus, R. L., Daley, K. N., French, M. A., Thackeray, A., Hoyer, E. H., Beck, D., & Young, D. L. (2026) · Learning Health Systems · doi:10.1002/lrh2.70110
Yang, E., Sun, J., Dong, Y., Chang, C., Zhong, L., Tse, G., Cheung, B. M. Y., García-Zamora, S., & Koh, A. S.-M. (2026) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00450.2026
Rigorous Female Breast Cancer Phenotyping Using the All of Us Research ProgramPreprint
Qi, Y., Lundy, K., Gee, D. A., & Chambwe, N. (2026) · medRxiv · doi:10.64898/2026.08.07.26359972
Heudel, P., & Blay, J. (2026) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2026.100773
Methods to validate structured electronic health record data from emergency departments (Preprint)Preprint
Brown, E. M., Oh, R. D.-E., Yeom, D. S., Niosco, C., Ho, V., Hau, J. P., Randhawa, L., Wang, J., Mah, C., Pavlovich, S., et al. (2026) · doi:10.2196/preprints.110624
Gasperi, M., Gerstenberger, A., Wild, M. G., Stein, M. B., & Afari, N. (2026) · Journal of Pain · doi:10.1016/j.jpain.2026.106435
Beyond Blind Trust in Hospital Data Warehouses: Evaluating Real‐World Data Accountability
Morohashi, A., Yamashita, S., Imaizumi, T., 佐野, 拓., Kuwatsuka, Y., Ando, M., & Shiratori, Y. (2026) · Learning Health Systems · doi:10.1002/lrh2.70118
“processes, clinical context, transformation logic, and provenance information may become partially obscured from secondary users [6].”
Big Problems, Big Data, Big Answers
Rider, N. L. (2026) · Immunology and Allergy Clinics of North America · doi:10.1016/j.iac.2026.08.010
Xin, W., Li, X., Qu, W., Nie, W., Zhao, Y., Wang, H., Yi, H., & Tan, S. (2026) · doi:10.2196/preprints.110943
Zikos, D., Chandrasekaran, A., & Dang, T. (2026) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2026.101815
Hussain, S., Raza, Z., Ahmad, S., & Kumar, N. (2026) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2026.111576
Ghafarian, A. M., Samani, A., Revankar, S., Charles, S., Oyibo, O., Alghamdi, F. S., Parray, K., Sutton, A., & Loggini, A. (2026) · Heart & Lung · doi:10.1016/j.hrtlng.2026.102967
Brunetta, E., Moro, L. D., Menon, V., & Selmi, C. F. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1952601
“…accumulated over more than a decade, we used a Python natural-language-processing (NLP) pipeline to calculate the Charlson Comorbidity Index (CCI) and assign principal diagnoses, in line with current methodological standards for the high-throughput generation of real-world evidence (19 – 22).”
Methods to validate structured electronic health record data from emergency departments (Preprint)
Brown, E. M., Oh, R. D.-E., Yeom, D. S., Niosco, C., Ho, V., Hau, J. P., Randhawa, L., Wang, J., Mah, C., Pavlovich, S., et al. (2026) · JMIR Medical Informatics · doi:10.2196/110624
Metadata
- Authors
- Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., & Saltz, J. H.
- Affiliations
- Oregon Health & Science University (US)
- AstraZeneca (United Kingdom) (GB)
- The Ohio State University Wexner Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Johns Hopkins University (US)
- Columbia University (US)
- Medical University of South Carolina (US)
- National Institutes of Health (US)
- Emory University (US)
- Corresponding author
- Hersh, William R.
- Logan, Judith R.
- Cimino, James J.
- ORCID
- Venue
- Medical Care
- Published
- 2013-08
- Volume
- 51
- Issue
- Supplement 8Suppl 3
- Pages
- S30-S37
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC3748381
- OpenAlex
- W2137767404
- Cited by
- 567
- References
- 68
- Citations by year
- 2026: 43 · 2025: 44 · 2024: 30 · 2023: 43 · 2022: 43 · 2021: 58 · 2020: 50 · 2019: 59 · 2018: 50 · 2017: 45 · 2016: 45 · 2015: 27 · 2014: 24 · 2013: 6
- FWCI
- 58.67
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000454, UL1TR000090)
- Centers for Disease Control and Prevention
- National Center for Research Resources (UL1RR025008, UL1RR024156, UL1RR025005, UL1RR024148, UL1RR024140)
- National Center for Advancing Translational Sciences
- H2020 European Research Council
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- electronic health records, comparative effectiveness research, data quality, data completeness, data provenance, data granularity, data reuse
- MeSH terms
- Data Collection, Data Interpretation, Statistical, Humans, Insurance Claim Review, Research Design, Comparative Effectiveness Research, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review