The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse
Open accessAbstract
BACKGROUND: Enormous amounts of data are recorded routinely in health care as part of the care process, primarily for managing individual patient care. There are significant opportunities to use these data for other purposes, many of which would contribute to establishing a learning health system. This is particularly true for data recorded in primary care settings, as in many countries, these are the first place patients turn to for most health problems. OBJECTIVE: In this paper, we discuss whether data that are recorded routinely as part of the health care process in primary care are actually fit to use for other purposes such as research and quality of health care indicators, how the original purpose may affect the extent to which the data are fit for another purpose, and the mechanisms behind these effects. In doing so, we want to identify possible sources of bias that are relevant for the use and reuse of these type of data. METHODS: This paper is based on the authors' experience as users of electronic health records data, as general practitioners, health informatics experts, and health services researchers. It is a product of the discussions they had during the Translational Research and Patient Safety in Europe (TRANSFoRm) project, which was funded by the European Commission and sought to develop, pilot, and evaluate a core information architecture for the learning health system in Europe, based on primary care electronic health records. RESULTS: We first describe the different stages in the processing of electronic health record data, as well as the different purposes for which these data are used. Given the different data processing steps and purposes, we then discuss the possible mechanisms for each individual data processing step that can generate biased outcomes. We identified 13 possible sources of bias. Four of them are related to the organization of a health care system, whereas some are of a more technical nature. CONCLUSIONS: There are a substantial number of possible sources of bias; very little is known about the size and direction of their impact. However, anyone that uses or reuses data that were recorded as part of the health care process (such as researchers and clinicians) should be aware of the associated data collection process and environmental influences that can affect the quality of the data. Our stepwise, actor- and purpose-oriented approach may help to identify these possible sources of bias. Unless data quality issues are better understood and unless adequate controls are embedded throughout the data lifecycle, data-driven health care will not live up to its expectations. We need a data quality research agenda to devise the appropriate instruments needed to assess the magnitude of each of the possible sources of bias, and then start measuring their impact. The possible sources of bias described in this paper serve as a starting point for this research agenda.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Morbidity is related to a green living environment
Maas, J., Verheij, R. A., de Vries, S., Spreeuwenberg, P., Schellevis, F. G., & Groenewegen, P. P. (2009) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2008.079038 · Cited by 1,133
Self-citationA Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126
Safran, C., Bloomrosen, M., Hammond, W. E., Labkoff, S., Markel-Fox, S., Tang, P. C., & Detmer, D. E. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2273 · Cited by 705
Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567 · cited 1×
“purpose may not be suitable for another and that there are serious issues to be considered in the use or reuse of EHR data [24-28].”
BackgroundICPC-2 : international classification of primary care
Lamberts, H., Wood, M., & Physicians, F. (1987) · Oxford University Press eBooks · Cited by 553
Achieving a Nationwide Learning Health System
Friedman, C. P., Wong, A. K., & Blumenthal, D. (2010) · Science Translational Medicine · doi:10.1126/scitranslmed.3001456 · Cited by 539
Friedman, C., Rubin, J., Brown, J., Buntin, M., Corn, M., Etheredge, L., Gunter, C., Musen, M., Platt, R., Stead, W., et al. (2015) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002977 · Cited by 509
What Is Real-World Data? A Review of Definitions Based on Literature and Stakeholder Interviews
Makady, A., de Boer, A., Hillege, H., Klungel, O., & Goettsch, W. (2017) · Value in Health · doi:10.1016/j.jval.2017.03.008 · Cited by 374
Defining and measuring completeness of electronic health records for secondary use
Weiskopf, N. G., Hripcsak, G., Swaminathan, S., & Weng, C. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.06.010 · Cited by 367 · cited 1×
“purpose may not be suitable for another and that there are serious issues to be considered in the use or reuse of EHR data [24-28].”
BackgroundChan, K. S., Fowles, J. B., & Weiner, J. P. (2010) · Medical Care Research and Review · doi:10.1177/1077558709359007 · Cited by 365
Prokosch, H. (2009) · doi:10.3414/me9132 · Cited by 262
The Learning Healthcare System: Where are we now? A systematic review
Budrionis, A., & Bellika, J. G. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.09.018 · Cited by 247
Diagnostic Difficulty and Error in Primary Care—A Systematic Review
Kostopoulou, O., Delaney, B. C., & Munro, C. W. (2008) · Family Practice · doi:10.1093/fampra/cmn071 · Cited by 200
Self-citationCombining Multiple Indicators of Clinical Quality: An Evaluation of Different Analytic Approaches
Reeves, D., Campbell, S. M., Adams, J., Shekelle, P. G., Kontopantelis, E., & Roland, M. O. (2007) · Medical Care · doi:10.1097/mlr.0b013e31803bb479 · Cited by 180
Validity of electronic health record-derived quality measurement for performance monitoring
Parsons, A., McCullough, C., Wang, J., & Shih, S. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000557 · Cited by 137
van Doormaal, J. E., van den Bemt, P. M. L. A., Zaal, R. J., Egberts, T. C. G., Lenderink, B. W., Kosterink, J. G. W., Haaijer‐Ruskamp, F. M., & Mol, P. G. M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3099 · Cited by 133
The UK's National Programme for IT: Why was it dismantled?
Justinia, T. (2016) · Health Services Management Research · doi:10.1177/0951484816662492 · Cited by 115
Köpcke, F., Trinczek, B., Majeed, R. W., Schreiweis, B., Wenk, J., Leusch, T., Ganslandt, T., Ohmann, C., Bergh, B., Röhrig, R., et al. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-37 · Cited by 107
Use and Abuse of Computer-Stored Medical Records
van der Lei, J. (1991) · Methods of Information in Medicine · doi:10.1055/s-0038-1634831 · Cited by 88
Improving the quality of EHR recording in primary care: a data quality feedback tool
van der Bij, S., Khan, N., ten Veen, P., de Bakker, D. H., & Verheij, R. A. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw054 · Cited by 85
Self-citationRyu, B., Kim, N., Heo, E., Yoo, S., Lee, K., Hwang, H. Y., Kim, J., Kim, Y., Lee, J., & Jung, S. Y. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8867 · Cited by 85
Patient-Powered Research Networks Aim To Improve Patient Care And Health Research
Fleurence, R., Beal, A. C., Sheridan, S. E., Johnson, L., & Selby, J. V. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0113 · Cited by 80
The denominator in general practice, a new approach from the Intego database
Bartholomeeusen, S., Kim, C., Mertens, R., Faes, C., & Buntinx, F. (2005) · Family Practice · doi:10.1093/fampra/cmi054 · Cited by 75
Zwaanswijk, M., Verheij, R. A., Wiesman, F. J., & Friele, R. D. (2011) · BMC Health Services Research · doi:10.1186/1472-6963-11-256 · Cited by 72
Self-citationAmarasingham, R., Velasco, F. T., Xie, B., Clark, C., Ma, Y., Zhang, S., Bhat, D., Lucena, B., Huesch, M. D., & Halm, E. A. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0162-6 · Cited by 72
Kostopoulou, O., Porat, T., Corrigan, D., Mahmoud, S., & Delaney, B. C. (2017) · British Journal of General Practice · doi:10.3399/bjgp16x688417 · Cited by 71
Self-citationde Vries, F., de Vries, C. S., Cooper, C. C., Leufkens, B., & van Staa, T. P. (2006) · International Journal of Epidemiology · doi:10.1093/ije/dyl147 · Cited by 67
Hersh, W., Cimino, J. J., Payne, P., Embí, P. J., Logan, J. R., Weiner, M. G., Bernstam, E. V., Lehmann, H. P., Hripcsak, G. M., Hartzog, T. H., et al. (2013) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1018 · Cited by 63
A Learning Health Care System Using Computer-Aided Diagnosis
Cahan, A., & Cimino, J. J. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6663 · Cited by 60
Schäfer, W. L. A., Boerma, W. G. W., Spreeuwenberg, P., et al. (2016) · Scandinavian Journal of Primary Health Care · doi:10.3109/02813432.2015.1132887 · Cited by 58
Validation of Syndromic Surveillance for Respiratory Pathogen Activity
van den Wijngaard, C., van Asten, L., van Pelt, W., Nagelkerke, N. J. D., Verheij, R. A., de Neeling, A. J., Dekkers, A. L. M., van der Sande, M. A. B., van Vliet, H., & Koopmans, M. (2008) · Emerging infectious diseases · doi:10.3201/eid1406.071467 · Cited by 56
Self-citationKuchinke, W., Ohmann, C., Verheij, R. A., et al. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.08.009 · Cited by 43
Self-citationSchweikardt, C., Verheij, R. A., Donker, G., & Coppieters, Y. (2016) · Journal of Public Health · doi:10.1007/s10389-016-0753-4 · Cited by 40
Self-citationWright, A. T., McCoy, A. B., Henkin, S., Kale, A., & Sittig, D. F. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001576 · Cited by 34
McGinn, T. G. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.4553 · Cited by 34
Liaw, S., Taggart, J., Yu, H., & de Lusignan, S. (2013) · PubMed · Cited by 33
Dorn, T., Yzermans, J. C., Spreeuwenberg, P. M. M., Schilder, A., & van der Zee, J. (2008) · Journal of Traumatic Stress · doi:10.1002/jts.20328 · Cited by 31
Barkhuysen, P. N., de Grauw, W. J. C., Akkermans, R., Donkers, J., Schers, H., Biermans, M. C. J., Tiersma, W., & Peters, H. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001479 · Cited by 31
The European Institute for Innovation through Health Data
Kalra, D., Stroetmann, V. N., Sundgren, M., Dupont, D., Schlünder, I., Thienpont, G., Coorevits, P. L., & Moor, G. D. (2016) · Learning Health Systems · doi:10.1002/lrh2.10008 · Cited by 28
Sollie, A., Sijmons, R. H., Helsper, C. W., & Numans, M. E. (2016) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2016.08.004 · Cited by 28
Li, P., Xie, C., Pollard, T., Johnson, A. E. W., Cao, D., Kang, H., Liang, H., Zhang, Y., Liu, X., Fan, Y., et al. (2017) · JMIR Medical Informatics · doi:10.2196/medinform.7380 · Cited by 28
de Lusignan, S., Ross, P., Shifrin, M. A., Hercigonja-Szekeres, M., & Brigitte Séroussi, A. (2013) · Studies in health technology and informatics · doi:10.3233/978-1-61499-289-9-397 · Cited by 27
Impact of remuneration on guideline adherence: Empirical evidence in general practice
van Dijk, C. E., Verheij, R. A., Spreeuwenberg, P. M. M., van den Berg, M. J., Groenewegen, P., Braspenning, J., & de Bakker, D. H. (2013) · Scandinavian Journal of Primary Health Care · doi:10.3109/02813432.2012.757078 · Cited by 25
Self-citationFassaert, T. J. L., Nielen, M. M. J., Verheij, R. A., Verhoeff, A. P., Dekker, J., Beekman, A. T. F., & de Wit, M. A. S. (2010) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2010.04.010 · Cited by 25
Self-citationThe Electronic Healthcare Record for Clinical Research (EHR4CR) information model and terminology
Ouagne, D., Hussain, S., Sadou, É., Jaulent, M., & Daniel, C. (2012) · Studies in health technology and informatics · doi:10.3233/978-1-61499-101-4-534 · Cited by 23
Uiters, E., Maurits, E. E. M., Droomers, M., Zwaanswijk, M., Verheij, R. A., & van der Lucht, F. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-1104 · Cited by 19
Self-citationWill the trilogue on the EU Data Protection Regulation recognise the importance of health research?
Coppen, R., van Veen, E.-B., Groenewegen, P., et al. (2015) · European Journal of Public Health · doi:10.1093/eurpub/ckv149 · Cited by 18
Opondo, D., Visscher, S., Eslami, S., Verheij, R. A., Korevaar, J. C., & Abu–Hanna, A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0129515 · Cited by 14
Self-citationData quality issues impede comparability of hospital treatment delay performance indicators
Verweij, L. M., Tra, J., Engel, J., Verheij, R. A., de Bruijne, M. C., & Wagner, C. (2015) · Netherlands Heart Journal · doi:10.1007/s12471-015-0708-3 · Cited by 10
Self-citationInzet variabiliseringsgelden 2013 leidt tot belangrijke verbetering EPD
van der Bij, S., & Verheij, R. A. (2013) · SynthesHis · doi:10.1007/s12494-013-0061-0 · Cited by 6
Self-citation
Cited by
Security and privacy of electronic health records: Concerns and challenges
Keshta, I., & Odeh, A. (2021) · Egyptian Informatics Journal · doi:10.1016/j.eij.2020.07.003 · Cited by 498
Koo, M. M., Swann, R., McPhail, S., Abel, G. A., Elliss-Brookes, L., Rubin, G. P., & Lyratzopoulos, G. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30595-9 · Cited by 258
From promise to practice: towards the realisation of AI-informed mental health care
Koutsouleris, N., Hauser, T. U., Skvortsova, V., & De Choudhury, M. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00153-4 · Cited by 251
Kohane, I. S., Aronow, B. J., Avillach, P., Beaulieu-Jones, B. K., Bellazzi, R., Bradford, R. L., Brat, G. A., Cannataro, M., Cimino, J. J., García-Barrio, N., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/22219 · Cited by 113
Development and validation of early warning score system: A systematic literature review
Fu, L.-H., Schwartz, J., Moy, A., Knaplund, C., Kang, M.-J., Schnock, K. O., Garcia, J. P., Jia, H., Dykes, P. C., Cato, K., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103410 · Cited by 102
Metadata
- Authors
- Verheij, R. A., Curcin, V., Delaney, B. C., & McGilchrist, M. M.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-05-29
- Volume
- 20
- Issue
- 5
- Pages
- e185
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W2805587876
- Cited by
- 290
- References
- 50
- FWCI
- 31.97
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.