A Natural Language Processing Model for COVID-19 Detection Based on Dutch General Practice Electronic Health Records by Using Bidirectional Encoder Representations From Transformers: Development and Validation Study
Open accessPublished version of a preprint
Abstract
Background: Natural language processing (NLP) models such as bidirectional encoder representations from transformers (BERT) hold promise in revolutionizing disease identification from electronic health records (EHRs) by potentially enhancing efficiency and accuracy. However, their practical application in practice settings demands a comprehensive and multidisciplinary approach to development and validation. The COVID-19 pandemic highlighted challenges in disease identification due to limited testing availability and challenges in handling unstructured data. In the Netherlands, where general practitioners (GPs) serve as the first point of contact for health care, EHRs generated by these primary care providers contain a wealth of potentially valuable information. Nonetheless, the unstructured nature of free-text entries in EHRs poses challenges in identifying trends, detecting disease outbreaks, or accurately pinpointing COVID-19 cases. Objective: This study aims to develop and validate a BERT model for detecting COVID-19 consultations in general practice EHRs in the Netherlands. Methods: The BERT model was initially pretrained on Dutch language data and fine-tuned using a comprehensive EHR data set comprising confirmed COVID-19 GP consultations and non–COVID-19–related consultations. The data set was partitioned into a training and development set, and the model’s performance was evaluated on an independent test set that served as the primary measure of its effectiveness in COVID-19 detection. To validate the final model, its performance was assessed through 3 approaches. First, external validation was applied on an EHR data set from a different geographic region in the Netherlands. Second, validation was conducted using results of polymerase chain reaction (PCR) test data obtained from municipal health services. Lastly, correlation between predicted outcomes and COVID-19–related hospitalizations in the Netherlands was assessed, encompassing the period around the outbreak of the pandemic in the Netherlands, that is, the period before widespread testing. Results: The model development used 300,359 GP consultations. We developed a highly accurate model for COVID-19 consultations (accuracy 0.97, F1-score 0.90, precision 0.85, recall 0.85, specificity 0.99). External validations showed comparable high performance. Validation on PCR test data showed high recall but low precision and specificity. Validation using hospital data showed significant correlation between COVID-19 predictions of the model and COVID-19–related hospitalizations (F1-score 96.8; P<.001; R2=0.69). Most importantly, the model was able to predict COVID-19 cases weeks before the first confirmed case in the Netherlands. Conclusions: The developed BERT model was able to accurately identify COVID-19 cases among GP consultations even preceding confirmed cases. The validated efficacy of our BERT model highlights the potential of NLP models to identify disease outbreaks early, exemplifying the power of multidisciplinary efforts in harnessing technology for disease identification. Moreover, the implications of this study extend beyond COVID-19 and offer a blueprint for the early recognition of various illnesses, revealing that such models could revolutionize disease surveillance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transformers, exemplified by the Bidirectional Encoder Representations from Transformers (BERT) model, have exhibited superior performance across a wide range of natural language processing tasks and have been successfully adapted for time series analysis.
“Transformers, exemplified by the Bidirectional Encoder Representations from Transformers (BERT) model, have exhibited superior performance across a wide range of natural language processing tasks and have been successfully adapted for time series analysis [45, 46].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837 · see citing works
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · see citing works
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 1,984 · see citing works
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · see citing works
Struyf, T., Deeks, J. J., Dinnes, J., Takwoingi, Y., Davenport, C. F., Leeflang, M., Spijker, R., Hooft, L., Emperador, D., Dittrich, S., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013665 · Cited by 584 · see citing works
Artificial intelligence in oncology
Shimizu, H., & Nakayama, K. I. (2020) · Cancer Science · doi:10.1111/cas.14377 · Cited by 388 · see citing works
BERTje: A Dutch BERT ModelPreprint
de Vries, W., van Cranenburgh, A., Bisazza, A., Caselli, T., van Noord, G., & Nissim, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09582 · Cited by 309 · see citing works
COVID-19 and primary care in six countries
Huston, P., Campbell, J., Russell, G. M., Goodyear‐Smith, F., Phillips, R. L., van Weel, C., & Hogg, W. E. (2020) · BJGP Open · doi:10.3399/bjgpopen20x101128 · Cited by 164 · see citing works
A survey on clinical natural language processing in the United Kingdom from 2007 to 2022
Wu, H., Wang, M., Wu, J., Francis, F., Chang, Y.-H., Shavick, A., Dong, H., Poon, M. T. C., Fitzpatrick, N., Levine, A. P., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00730-6 · Cited by 101 · see citing works
Al-Garadi, M. A., Yang, Y., & Sarker, A. (2022) · Healthcare · doi:10.3390/healthcare10112270 · Cited by 49 · see citing works
Malden, D. E., Tartof, S. Y., Ackerson, B. K., Hong, V. X., Skarbinski, J., Yau, V., Qian, L., Fischer, H. J., Shaw, S. F., Caparosa, S. L., et al. (2022) · JMIR Public Health and Surveillance · doi:10.2196/41529 · Cited by 26 · see citing works
Zhu, Y., Mahale, A., Peters, K., Mathew, L., Giuste, F., Anderson, B. J., & Wang, M. D. (2022) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3535508.3545555 · Cited by 23 · see citing works
Course of disease and risk factors for hospitalization in outpatients with a SARS-CoV-2 infection
Schäfer, E., Scheer, C. S., Saljé, K., Fritz, A., Kohlmann, T., Hübner, N., Napp, M., Fiedler-Lacombe, L., Stahl, D., Rauch, B. H., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-11103-0 · Cited by 19 · see citing works
Early detection of COVID-19 outbreaks using textual analysis of electronic medical records
Shapiro, M. J., Landau, R., Shay, S., Kaminsky, M., & Verhovsky, G. (2022) · Journal of Clinical Virology · doi:10.1016/j.jcv.2022.105251 · Cited by 8 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 1
- [10]Alsentzer EJ Murphy JR Boag W Weng WH Jin D Naumann T McDermott MBA Publicly available clinical BERT embeddings Aclanthology 2023-09-20 https://aclanthology.org/W19-1909.pdf
Cited by
Sorted by citations
Chen, S., Yu, J., Chamouni, S., Wang, Y., & Li, Y. (2024) · BMC Medicine · doi:10.1186/s12916-024-03566-x · Cited by 73 · see citing works
Alkalbani, A. M., Alrawahi, A. S., Salah, A., Haghighi, V., Zhang, Y., Alkindi, S. S. A., & Sheng, Q. Z. (2025) · Information · doi:10.3390/info16060489 · Cited by 27 · see citing works
Li, J., Tseng, Y., Chen, S.-H., & Chen, K. (2025) · Biomedical Journal · doi:10.1016/j.bj.2025.100929 · Cited by 14 · see citing works
Patient-Centered Research Through Artificial Intelligence to Identify Priorities in Cancer Care
Kim, J. Y., Chen, M. L., Rezaei, S. J., Ramírez‐Posada, M., Caswell-Jin, J. L., Kurian, A. W., Riaz, F., Sarin, K. Y., Tang, J. Y., Asch, S. M., et al. (2025) · JAMA Oncology · doi:10.1001/jamaoncol.2025.0694 · Cited by 14 · see citing works
Han, Y., Lam, J. C. K., Li, V. O. K., & Crowcroft, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04202-y · Cited by 11 · see citing works
Generative artificial intelligence for general practice; new potential ahead, but are we ready?
Geersing, G., de Wit, N. J., & Thompson, M. J. (2025) · European Journal of General Practice · doi:10.1080/13814788.2025.2511645 · Cited by 9 · see citing works
Alkalbani, A. M., Alrawahi, A. S., Salah, A., Haghighi, V., Zhang, Y., Alkindi, S. S. A., & Sheng, Q. Z. (2025) · Research Square · doi:10.21203/rs.3.rs-5128451/v2 · Cited by 6 · see citing works
van Essen, M., Twickler, R., Weesie, Y. M., Arslan, I. G., Groenhof, F., Peters, L. L., Bos, I., & Verheij, R. A. (2025) · Journal of Medical Internet Research · doi:10.2196/64628 · Cited by 4 · see citing works
Self-citationHunik, L., Uijen, A. A., Kueper, J. K., Terry, A. L., Hartman, T. O., van Laarhoven, T., & Schers, H. (2026) · European Journal of General Practice · doi:10.1080/13814788.2026.2620908 · Cited by 2 · see citing works
Hasselt Corona Impact Study: Impact of COVID-19 on healthcare seeking in a small Dutch town
Veldman, C., Gijssel, E. A. V., Rooij, A. H. V., Buitenhuis, L., van den Berg, J. W., & Blanker, M. H. (2025) · npj Primary Care Respiratory Medicine · doi:10.1038/s41533-025-00426-w · Cited by 2 · see citing works
AI-driven early infectious disease detection in Dutch primary care using BERT and ERNIE
Homburg, M., Danoe, G., Berger, M. Y., Hartman, T. O., Muris, J. W. M., Voss, A., Hamprecht, A. G., Brilman, M. F., Peters, L. L., & Berends, M. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02278-7 · Cited by 1 · see citing works
Self-citationAlkalbani, A. M., Alrawahi, A. S., Salah, A., Haghighi, V., Zhang, Y., Alkindi, S. S. A., & Sheng, Q. Z. (2024) · Research Square · doi:10.21203/rs.3.rs-5128451/v1 · Cited by 1 · see citing works
van Essen, M., Twickler, R., Weesie, Y. M., Arslan, I. G., Groenhof, F., Peters, L. L., Bos, I., & Verheij, R. A. (2024) · doi:10.2196/preprints.64628
Self-citationSpam Messages: Message Detection by BERT
Mandal, K., & Saxena, K. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-3939-7_24
Yoshihara, H., Maeda, H., Hagiwara, Y., Sato, D., Kitajima, K., Iwata, A., de Velde, N. V., Nakamura, Y., Yamagishi, Y., & Igarashi, A. (2025) · doi:10.2196/preprints.84974
Dowell, A. C., MacRae, J., Blanch, R., Hiroi, M., Castelino, L., & Turner, N. M. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-103176
Alam, M. A., Mehdi, S., Ahsan, S., Khan, A. A., Tahseen, Z., Eqbal, K., & Rahman, D. M. N. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44021-2.00013-0
Yoshihara, H., Maeda, H., Hagiwara, Y., Sato, D., Kitajima, K., Iwata, A., de Velde, N. V., Nakamura, Y., Yamagishi, Y., & Igarashi, A. (2025) · JMIR Formative Research · doi:10.2196/84974
- Authors
- Homburg, M., Meijer, E., Berends, M., Kupers, T., Olde Hartman, T., Muris, J., de Schepper, E., Velek, P., Kuiper, J., Berger, M., & Peters, L.
- Affiliations
- University Medical Center Groningen (NL)
- Maastricht University Medical Centre (NL)
- Erasmus MC (NL)
- Corresponding author
- Homburg, Maarten
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2023-10-04
- Volume
- 25
- Pages
- e49944
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC10563863
- OpenAlex
- W4387326738
- Cited by
- 18
- References
- 15
- Citations by year
- 2026: 4 · 2025: 10 · 2024: 4
- FWCI
- 1.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by)
- Radboud Repository (Radboud University) (submittedVersion, cc-by)
- Research Publications (Maastricht University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- Funders
- ZonMw (10430112110001, 10430022010006)
- Universitair Medisch Centrum Groningen
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Keywords
- COVID-19 detection, BERT, natural language processing, primary care visits, Dutch language, outbreak surveillance, clinical text, external validation, hospitalization prediction
- MeSH terms
- COVID-19, Humans, Natural Language Processing, Electronic Health Records, General Practice, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
