The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research
Open accessAbstract
The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, a major concern is that data not collected systematically for research will be poorer quality, which could have negative impacts on findings generated from these data.24–27”
From A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data (2016) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 2,127
The “Meaningful Use” Regulation for Electronic Health Records
Blumenthal, D., & Tavenner, M. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006114 · Cited by 2,059
A review of uses of health care utilization databases for epidemiologic research on therapeutics
Schneeweiß, S., & Avorn, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.012 · Cited by 1,290
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 1×
“A review of the literature of studies evaluating the data quality of EHRs for clinical research identified 5 dimensions of data quality assessed: completeness, correctness, concordance, plausibility, and currency.(26) The authors identified many studies with a wide variety of techniques to assess these dimensions and,…”
Measuring Diagnoses: ICD Code Accuracy
O’Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,156
Assessing Quality Using Administrative Data
Iezzoni, L. I. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-8_part_2-199710151-00048 · Cited by 981
Next-generation phenotyping of electronic health records
Hripcsak, G., & Albers, D. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001145 · Cited by 795
Self-citationSafran, C., Bloomrosen, M., Hammond, W. E., Labkoff, S., Markel-Fox, S., Tang, P., & Detmer, D. (2007) · JAMIA Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2273 · Cited by 714
the eMERGE Team, McCarty, C. A., Chisholm, R. L., Chute, C. G., Kullo, I. J., Jarvik, G. P., Larson, E. B., Li, R., Masys, D. R., Ritchie, M. D., et al. (2011) · BMC Medical Genomics · doi:10.1186/1755-8794-4-13 · Cited by 701
Desiderata for Controlled Medical Vocabularies in the Twenty-First Century
Cimino, J. J. (1998) · Methods of Information in Medicine · doi:10.1055/s-0038-1634558 · Cited by 656
Self-citationChildhood Obesity and Metabolic Imprinting
Hillier, T. A., Pedula, K. L., Schmidt, M. M., Mullen, J. A., Charles, M.-A., & Pettitt, D. J. (2007) · Diabetes Care · doi:10.2337/dc06-2361 · Cited by 656
Jollis, J. G., Ancukiewicz, M., DeLong, E. R., Pryor, D. B., Muhlbaier, L. H., & Mark, D. B. (1993) · Annals of Internal Medicine · doi:10.7326/0003-4819-119-8-199310150-00011 · Cited by 626
Achieving a Nationwide Learning Health System
Friedman, C. P., Wong, A. K., & Blumenthal, D. (2010) · Science Translational Medicine · doi:10.1126/scitranslmed.3001456 · Cited by 539
Reengineering Translational Science: The Time Is Right
Collins, F. S. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3002747 · Cited by 486
Chan, K. S., Fowles, J. B., & Weiner, J. P. (2010) · Medical Care Research and Review · doi:10.1177/1077558709359007 · Cited by 365
Lawson, E. H., Louie, R., Zingmond, D. S., Brook, R. H., Hall, B. L., Han, L. F., Rapp, M. T., & Ko, C. Y. (2012) · Annals of Surgery · doi:10.1097/sla.0b013e31826b4c4f · Cited by 343
Systematic review of scope and quality of electronic patient record data in primary care
Thiru, K., Hassey, A., & Sullivan, F. (2003) · BMJ · doi:10.1136/bmj.326.7398.1070 · Cited by 342
Unlocking Clinical Data from Narrative Reports: A Study of Natural Language Processing
Hripcsak, G. M., Friedman, C., Alderson, P. O., DuMouchel, W., Johnson, S. B., & Clayton, P. D. (1995) · Annals of Internal Medicine · doi:10.7326/0003-4819-122-9-199505010-00007 · Cited by 340
Self-citationSecondary Use of EHR: Data Quality Issues and Informatics Opportunities.
(2010) · PubMed Central · Cited by 332
de Lusignan, S., & van Weel, C. (2005) · Family Practice · doi:10.1093/fampra/cmi106 · Cited by 329
Lindenauer, P. K., Lagu, T. C., Shieh, M., Pekow, P. S., & Rothberg, M. B. (2012) · JAMA · doi:10.1001/jama.2012.384 · Cited by 295
Denny, J. C., Crawford, D. C., Ritchie, M. D., Bielinski, S. J., Basford, M., Bradford, Y., Chai, H. S., Bastarache, L. A., Zuvich, R. L., Peissig, P. L., et al. (2011) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2011.09.008 · Cited by 252
A systematic literature review of automated clinical coding and classification systems
Stanfill, M. H., Williams, M., Fenton, S. H., Jenders, R. A., & Hersh, W. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001024 · Cited by 224
Self-citationRea, S., Pathak, J., Savova, G., Oniki, T. A., Westberg, L., Beebe, C. E., Tao, C., Parker, C. G., Haug, P. J., Huff, S. M., et al. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2012.01.009 · Cited by 206
Zhang, Z., & Sun, J. (2009) · Statistical Methods in Medical Research · doi:10.1177/0962280209105023 · Cited by 190
The Methods of Comparative Effectiveness Research
Sox, H. C., & Goodman, S. N. (2012) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031811-124610 · Cited by 181
Review: Use of Electronic Medical Records for Health Outcomes Research
Dean, B. B., Lam, J. W. K., Natoli, J. L., Butler, Q. R., Aguilar, D., & Nordyke, R. J. (2009) · Medical Care Research and Review · doi:10.1177/1077558709332440 · Cited by 177
Ryan, P. B., Madigan, D., Stang, P., Overhage, J. M., Racoosin, J. A., & Hartzema, A. G. (2012) · Statistics in Medicine · doi:10.1002/sim.5620 · Cited by 177
Wahl, P. M., Rodgers, K., Schneeweiß, S., Gage, B. F., Butler, J., Wilmer, C. I., Nash, M., Esper, G. J., Gitlin, N., Osborn, N. K., et al. (2010) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1924 · Cited by 174
Chapter 13: Mining Electronic Health Records in the Genomics Era
Denny, J. C. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002823 · Cited by 170
Building a Virtual Cancer Research Organization
Hornbrook, M. C. (2005) · JNCI Monographs · doi:10.1093/jncimonographs/lgi033 · Cited by 154
Tannen, R. L., Weiner, M. G., & Xie, D. (2009) · BMJ · doi:10.1136/bmj.b81 · Cited by 149
Self-citationPatients Treated at Multiple Acute Health Care Facilities: Quantifying Information Fragmentation
Bourgeois, F. C. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.439 · Cited by 140
Validity of electronic health record-derived quality measurement for performance monitoring
Parsons, A., McCullough, C., Wang, J., & Shih, S. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000557 · Cited by 137
Toward Reuse of Clinical Data for Research and Quality Improvement: The End of the Beginning?
Weiner, M. G., & Embi, P. J. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-5-200909010-00141 · Cited by 129
Self-citationIdentification of Genomic Predictors of Atrioventricular Conduction
Denny, J. C., Ritchie, M. D., Crawford, D. C., Schildcrout, J. S., Ramirez, A. H., Pulley, J. M., Basford, M., Masys, D. R., Haines, J. L., & Roden, D. M. (2010) · Circulation · doi:10.1161/circulationaha.110.948828 · Cited by 124
Challenges in Using Electronic Health Record Data for CER
Bayley, K. B., Belnap, T., Savitz, L. A., Masica, A. L., Shah, N. D., & Fleming, N. S. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1d48 · Cited by 120
A Genome-Wide Association Study of Red Blood Cell Traits Using the Electronic Medical Record
Kullo, I. J., Ding, K., Jouni, H., Smith, C. Y., & Chute, C. G. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013011 · Cited by 118
Wei, W.-Q., Leibson, C. L., Ransom, J. E., Kho, A. N., Caraballo, P. J., Chai, H. S., Yawn, B. P., Pacheco, J. A., & Chute, C. G. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000597 · Cited by 112
Genetic variants associated with the white blood cell count in 13,923 subjects in the eMERGE Network
Crosslin, D. R., McDavid, A., Weston, N., Nelson, S. C., Zheng, X., Hart, E., de Andrade, M., Kullo, I. J., McCarty, C. A., Doheny, K. F., et al. (2011) · Human Genetics · doi:10.1007/s00439-011-1103-9 · Cited by 110
DeVoe, J. E., Gold, R. B., McIntire, P., Puro, J. E., Chauvie, S., & Gallia, C. A. (2011) · The Annals of Family Medicine · doi:10.1370/afm.1279 · Cited by 104
Association of Patient-Centered Outcomes With Patient-Reported and ICD-9-Based Morbidity Measures
Bayliss, E. A., Ellis, J. L., Shoup, J. A., Zeng, C. X., McQuillan, D. B., & Steiner, J. F. (2012) · The Annals of Family Medicine · doi:10.1370/afm.1364 · Cited by 92
Identification of metastatic cancer in claims data
Nordstrom, B. L., Whyte, J. L., Stolar, M. J., Mercaldi, C. J., & Kallich, J. D. (2012) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3247 · Cited by 84
Bias Associated with Mining Electronic Health Records
Hripcsak, G., Knirsch, C., Zhou, L., Wilcox, A., & Melton, G. (2011) · Journal of Biomedical Discovery and Collaboration · doi:10.5210/disco.v6i0.3581 · Cited by 83
Self-citationMowat, F. S., Lau, Whyte, Kelsh, Legg, Engel-Nitz, Watson, Collins, & Nordyke, R. J. (2011) · Clinical Epidemiology · doi:10.2147/clep.s23690 · Cited by 75
Effect of Nonpayment for Hospital-Acquired, Catheter-Associated Urinary Tract Infection
Meddings, J. A., Reichert, H., Rogers, M. A. M., Saint, S. K., Stephansky, J., & McMahon, L. F. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-5-201209040-00003 · Cited by 70
Finnell, J. T., Overhage, J. M., & Grannis, S. J. (2011) · PubMed · Cited by 69
Gorelick, M. H., Knight, S., Alessandrini, E. A., Stanley, R. M., Chamberlain, J. M., Kuppermann, N., & Alpern, E. R. (2007) · Academic Emergency Medicine · doi:10.1197/j.aem.2007.03.1357 · Cited by 68
Practices and perspectives on building integrated data repositories: results from a 2010 CTSA survey
MacKenzie, S. L., Wyatt, M., Schuff, R. A., Tenenbaum, J. D., & Anderson, N. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000508 · Cited by 62
Green, S. M. (2012) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2012.09.010 · Cited by 60
Evaluation of biomedical text-mining systems: Lessons learned from information retrieval
Hersh, W. (2005) · Briefings in Bioinformatics · doi:10.1093/bib/6.4.344 · Cited by 58
Self-citationThe importance of knowing context of hospital episode statistics when reconfiguring the NHS
Brennan, L., Watson, M., Klaber, R. E., & Charles, T. (2012) · BMJ · doi:10.1136/bmj.e2432 · Cited by 57
Harris, S. B., Glazier, R. H., Tompkins, J. W., Wilton, A. S., Chevendra, V., Stewart, M., & Thind, A. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-347 · Cited by 54
Benin, A. L., Vitkauskas, G., Thornquist, E., Shapiro, E. D., Concato, J., Aslan, M., & Krumholz, H. M. (2005) · Medical Care · doi:10.1097/01.mlr.0000167185.26058.8e · Cited by 52
Tannen, R. L., Weiner, M. G., Xie, D., & Barnhart, K. T. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.10.012 · Cited by 50
Self-citationHu, G., & Baker, S. P. (2012) · Public Health Reports · doi:10.1177/003335491212700307 · Cited by 50
Edinger, T., Cohen, A. M., Bedrick, S. D., Ambert, K. H., & Hersh, W. (2012) · PubMed Central · Cited by 50
Self-citationde Lusignan, S., Liaw, S., Krause, P., Ćurčin, V., Vicente, M., Michalakidis, G., Agréus, L., Leysen, P., Shaw, N. T., & Mendis, K. (2011) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638748 · Cited by 47
How the Continuity of Care Document Can Advance Medical Research and Public Health
D'Amore, J. D., Sittig, D. F., & Ness, R. B. (2012) · American Journal of Public Health · doi:10.2105/ajph.2011.300640 · Cited by 45
Tannen, R. L., Weiner, M. G., & Xie, D. (2008) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1584 · Cited by 45
Self-citationWei, W.-Q., Leibson, C. L., Ransom, J. E., Kho, A. N., & Chute, C. G. (2012) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2012.05.015 · Cited by 40
Sensible use of observational clinical data
Overhage, J., & Overhage, L. M. (2013) · Statistical Methods in Medical Research · doi:10.1177/0962280211403598 · Cited by 39
The impact of electronic medical records data sources on an adverse drug event quality measure
Kahn, M. G., & Ranade, D. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002451 · Cited by 31
Benin, A., Fenick, A., Herrin, J., Vitkauskas, G., Chen, J., & Brandt, C. (2011) · American Journal of Medical Quality · doi:10.1177/1062860611403136 · Cited by 31
Hormone therapy and coronary heart disease in young women
Weiner, M. G., Barnhart, K. T., Xie, D., & Tannen, R. L. (2007) · Menopause The Journal of The North American Menopause Society · doi:10.1097/gme.0b013e3180413e45 · Cited by 28
Self-citationYu, A. C., & Cimino, J. J. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.01.005 · Cited by 13
Self-citationMaster Data Management: Getting your House in Order
Seiler, K. P., Bodycombe, N. E., Hawkins, T., Shell, R., Lemon, A., Souza, A. G. D., & Clemons, P. A. (2011) · Combinatorial Chemistry & High Throughput Screening · doi:10.2174/138620711796957189 · Cited by 3
Yu, A. C., & Cimino, J. J. (2007) · PubMed · Cited by 3
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
“Adoption of unified data formats, such as Fast Healthcare Interoperability Resources [84], offer the potential for better aggregation of data, although improved interoperability does not necessarily fix the problem of inconsistent semantic coding in EHR data [85].”
Ahmed, Z., Mohamed, K., Zeeshan, S., & Dong, X. (2020) · Database · doi:10.1093/database/baaa010 · Cited by 1,035
“Use of EHR in comparative effectiveness research (46) Reporting caveats in existing healthcare systems.”
BackgroundMethodGoldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
“As others have noted, EHR data come with many challenges.(4) EHRs include all patients that touch a medical system, primarily capture data only when patients are ill, and collect metrics that clinicians deem to be necessary at each clinic visit.”
BackgroundAutomated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834
Metadata
- Authors
- Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., & Saltz, J. H.
- Affiliations
- Oregon Health & Science University (US)
- AstraZeneca (United Kingdom) (GB)
- The Ohio State University Wexner Medical Center (US)
- The University of Texas Health Science Center at Houston (US)
- Johns Hopkins University (US)
- Columbia University (US)
- Medical University of South Carolina (US)
- National Institutes of Health (US)
- Emory University (US)
- Corresponding author
- Hersh, William R.
- Logan, Judith R.
- Cimino, James J.
- ORCID
- Venue
- Medical Care
- Published
- 2013-08
- Volume
- 51
- Issue
- Supplement 8Suppl 3
- Pages
- S30-S37
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC3748381
- OpenAlex
- W2137767404
- Cited by
- 567
- References
- 68
- Citations by year
- 2026: 43 · 2025: 44 · 2024: 30 · 2023: 43 · 2022: 43 · 2021: 58 · 2020: 50 · 2019: 59 · 2018: 50 · 2017: 45 · 2016: 45 · 2015: 27 · 2014: 24 · 2013: 6
- FWCI
- 58.67
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (UL1TR000454, UL1TR000090)
- Centers for Disease Control and Prevention
- National Center for Research Resources (UL1RR025008, UL1RR024156, UL1RR025005, UL1RR024148, UL1RR024140)
- National Center for Advancing Translational Sciences
- H2020 European Research Council
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- electronic health records, comparative effectiveness research, data quality, data completeness, data provenance, data granularity, data reuse
- MeSH terms
- Data Collection, Data Interpretation, Statistical, Humans, Insurance Claim Review, Research Design, Comparative Effectiveness Research, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
