Development and Validation of a Deep Learning Model for Detection of Allergic Reactions Using Safety Event Reports Across Hospitals
Open accessAbstract
Importance: Although critical to patient safety, health care-related allergic reactions are challenging to identify and monitor. Objective: To develop a deep learning model to identify allergic reactions in the free-text narrative of hospital safety reports and evaluate its generalizability, efficiency, productivity, and interpretability. Design, Setting, and Participants: This cross-sectional study analyzed hospital safety reports filed between May 2004 and January 2019 at Brigham and Women's Hospital and between April 2006 and June 2018 at Massachusetts General Hospital in Boston. Training and validating a deep learning model involved extracting safety reports using 101 expert-curated keywords from Massachusetts General Hospital (data set I). The model was then evaluated on 3 data sets: reports without keywords (data set II), reports from a different time frame (data set III), and reports from a different hospital (Brigham and Women's Hospital; data set IV). Statistical analyses were performed between March 1, 2019, and July 18, 2020. Main Outcomes and Measures: The area under the receiver operating characteristic curve and area under the precision-recall curve were used on data set I. The precision at top-k was used on data sets II to IV. Results: A total of 299 028 safety reports with 172 854 patients were included. Of these patients, 86 544 were women (50.1%) and the median (interquartile range [IQR]) age was 59.7 (43.8-71.6) years. The deep learning model achieved an area under the receiver operating characteristic curve of 0.979 (95% CI, 0.973-0.985) and an area under the precision-recall curve of 0.809 (95% CI, 0.773-0.845). The model achieved precisions at the top 100 model-identified cases of 0.930 in data set II, 0.960 in data set III, and 0.990 in data set IV. Compared with the keyword-search approach, the deep learning model reduced the number of cases for manual review by 63.8% and identified 24.2% more cases of confirmed allergic reactions. The model highlighted important words (eg, rash, hives, and Benadryl) in prediction and extended the list of expert-curated keywords through an attention layer. Conclusions and Relevance: This study showed that a deep learning model can accurately and efficiently identify allergic reactions using free-text narratives written by a variety of health care professionals. This model could be used to improve allergy care, potentially enabling real-time event surveillance and guidance for medical errors and system improvement.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Medical datasets frequently suffer from imbalances in phenotypes, like rare diseases or serious clinical outcomes, and cohort identification.
“Medical datasets also frequently suffer from imbalances in phenotypes, like rare diseases or serious clinical outcomes37, and cohort identification7.”
From Probabilistic medical predictions of large language models (2024) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · see citing works
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · see citing works
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · see citing works
Incidence of Adverse Drug Events and Potential Adverse Drug Events: Implications for Prevention
Bates, D. W. (1995) · JAMA · doi:10.1001/jama.1995.03530010043033 · Cited by 2,127 · see citing works
Bates, D. W. (1998) · JAMA · doi:10.1001/jama.280.15.1311 · Cited by 2,107 · see citing works
Clinical Aspects of Immunology.
(1969) · Annals of Internal Medicine · doi:10.7326/0003-4819-71-1-229_2 · Cited by 1,526 · see citing works
Classen, D. C., Resar, R., Griffin, F., Federico, F., Frankel, T., Kimmel, N., Whittington, J. C., Frankel, A., Seger, A., & James, B. C. (2011) · Health Affairs · doi:10.1377/hlthaff.2011.0190 · Cited by 1,027 · see citing works
Osborne, N. J., Koplin, J. J., Martin, P. E., Gurrin, L. C., Lowe, A. J., Matheson, M. C., Ponsonby, A., Wake, M. A., Tang, M. L. K., Dharmage, S. C., et al. (2011) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2011.01.039 · Cited by 1,014 · see citing works
Drug Allergy: An Updated Practice Parameter
(2010) · Annals of Allergy Asthma & Immunology · doi:10.1016/j.anai.2010.08.002 · Cited by 954 · see citing works
The Clinical Aspects of Immunology.
(1964) · Annals of Internal Medicine · doi:10.7326/0003-4819-60-2-339_1 · Cited by 704 · see citing works
Critical incident reporting and learning
Mahajan, R. P. (2010) · British Journal of Anaesthesia · doi:10.1093/bja/aeq133 · Cited by 392 · see citing works
Feedback from incident reporting: information and action to improve patient safety
Benn, J. J., Koutantji, M., WALLACE, L. M., Spurgeon, P. C., Rejman, M., Healey, A., & Vincent, C. A. (2009) · BMJ Quality & Safety · doi:10.1136/qshc.2007.024166 · Cited by 339 · see citing works
Weingart, S. N., Pagovich, O. E., Sands, D. Z., Li, J. M., Aronson, M. D., Davis, R. B., Bates, D. W., & Phillips, R. S. (2005) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2005.0180.x · Cited by 322 · see citing works
Drug allergies documented in electronic health records of a large healthcare system
Zhou, L., Dhopeshwarkar, N., Blumenthal, K. G., Goss, F. R., Topaz, M., Slight, S. P., & Bates, D. W. (2016) · Allergy · doi:10.1111/all.12881 · Cited by 313 · see citing works
Self-citation2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records
Henry, S., Buchan, K., Filannino, M., Stubbs, A., & Uzuner, Ö. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz166 · Cited by 307 · see citing works
Cocos, A., Fiks, A. G., & Masino, A. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw180 · Cited by 210 · see citing works
Zhang, M., Holman, C. D. J., Price, S. D., Sanfilippo, F. M., Preen, D. B., & Bulsara, M. K. (2009) · BMJ · doi:10.1136/bmj.a2752 · Cited by 208 · see citing works
Institute of Medicine report: to err is human: building a safer health care system.
Homsted, L. (2000) · PubMed · Cited by 149 · see citing works
Young, I. J. B., Luz, S. F., & Lone, N. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103971 · Cited by 142 · see citing works
Prevalence of food allergies and intolerances documented in electronic health records
Acker, W. W., Plasek, J. M., Blumenthal, K. G., Lai, K. H., Topaz, M., Seger, D. L., Goss, F. R., Slight, S. P., Bates, D. W., & Zhou, L. (2017) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2017.04.006 · Cited by 123 · see citing works
Self-citationWang, L., Sha, L., Lakin, J. R., Bynum, J., Bates, D. W., Hong, P., & Zhou, L. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.6972 · Cited by 105 · see citing works
Self-citationBudnitz, D. S., Pollock, D. A., Mendelsohn, A. B., Weidenbach, K. N., McDonald, A. K., & Annest, J. L. (2004) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2004.09.020 · Cited by 83 · see citing works
Blumenthal, K. G., Lai, K. H., Huang, M., Wallace, Z. S., Wickner, P. G., & Zhou, L. (2017) · The Journal of Allergy and Clinical Immunology In Practice · doi:10.1016/j.jaip.2016.12.006 · Cited by 68 · see citing works
Self-citationA systematic review to evaluate the accuracy of electronic adverse drug event detection
Forster, A. J., Jennings, A., Chow, C., et al. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000454 · Cited by 52 · see citing works
WALLACE, L. M., Spurgeon, P. C., Benn, J. J., et al. (2009) · Health Services Management Research · doi:10.1258/hsmr.2008.008019 · Cited by 38 · see citing works
Kane‐Gill, S. L., Bos, J. V. D., & Handler, S. M. (2010) · Annals of Pharmacotherapy · doi:10.1345/aph.1m726 · Cited by 29 · see citing works
電子健康記録に記録された食物アレルギーと不耐性の有病率【Powered by NICT】
Acker, W. W., Joseph, M. P., Kimberly, G. B., Kenneth, H. L., Maxim, T., Diane, L. S., Foster, R. G., Sarah, P. S., David, W. B., & Li, Z. (2017) · Journal of Allergy and Clinical Immunology · Cited by 24 · see citing works
Okafor, N. G., Doshi, P., Miller, S. K., McCarthy, J. J., Hoot, N. R., Darger, B., Benitez, R. C., & Chathampally, Y. G. (2015) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2015.8.27390 · Cited by 23 · see citing works
Pathak, A. K. (2016) · JOURNAL OF CLINICAL AND DIAGNOSTIC RESEARCH · doi:10.7860/jcdr/2016/18826.8284 · Cited by 20 · see citing works
Jeffres, M. N., Hall-Lipsy, E. A., King, S. T., & Cleary, J. D. (2018) · Annals of Allergy Asthma & Immunology · doi:10.1016/j.anai.2018.03.010 · Cited by 20 · see citing works
Advancing the state of the art in automatic extraction of adverse drug events from narratives
Uzuner, Ö., Stubbs, A., & Lenert, L. A. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz206 · Cited by 14 · see citing works
Medical malpractice related to drug-induced anaphylaxis
Hyeon, C. W., Lee, J. Y., Jang, S. G., et al. (2019) · Medicine · doi:10.1097/md.0000000000015996 · Cited by 12 · see citing works
10,000 Good Catches: Increasing Safety Event Reporting In A Pediatric Health Care System
Crandall, K. M., Almuhanna, A., Cady, R. F., Fahey, L., Floyd, T. T., Freiburg, D., Hilliard, M. A., Kalburgi, S., Khan, N. I., Patrick, D., et al. (2018) · Pediatric Quality and Safety · doi:10.1097/pq9.0000000000000072 · Cited by 11 · see citing works
Allergic Reactions Captured by Voluntary Reporting
Blumenthal, K. G., Wolfson, A. R., Li, Y., et al. (2019) · Journal of Patient Safety · doi:10.1097/pts.0000000000000568 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- [23]Ginn R, Pimpalkhute P, Nikfarjam A, Mining Twitter for adverse drug reaction mentions: a corpus and classification benchmark. Paper presented at: Proceedings of the Fourth Workshop on Building and Evaluating Resources for Health and Biomedical Text Processing; May 21, 2014; Reykjavik, Iceland. Accessed January 1, 2019. http://www.nactem.ac.uk/biotxtm2014/papers/Ginnetal.pdf
- [28]Bahdanau D, Cho K, Bengio Y Neural machine translation by jointly learning to align and translate. Paper presented at: 3rd International Conference on Learning Representations; May 7-May 9, 2015; San Diego, California.
- [29]Yang Z, Yang D, Dyer C, He X, Smola AJ, Hovy EH Hierarchical attention networks for document classification. Paper presented at: 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; June 12-17, 2016; San Diego, California.
- [31]Paszke A, Gross S, Chintala S, Automatic differentiation in PyTorch. Paper presented at: NIPS Autodiff Workshop; December 9, 2017; Long Beach, California. Accessed January 1, 2019. https://openreview.net/pdf/25b8eee6c373d48b84e5e9c6e10e7cbbbce4ac73.pdf
Cited by
Sorted by citations
Probabilistic medical predictions of large language models
Gu, B., Desai, R. J., Lin, K. J., & Yang, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01366-4 · Cited by 50 · see citing works
Self-citationWu, J., Liu, X., Li, M., Li, W., Su, Z., Lin, S., Garay, L., Zhang, Z., Zhang, Y., Zeng, Q., et al. (2024) · NEJM AI · doi:10.1056/aira2400012 · Cited by 41 · see citing works
Self-citationWang, L., Novoa-Laurentiev, J., Cook, C., Srivatsan, S., Hua, Y., Yang, J., Miloslavsky, E., Choi, H. K., Zhou, L., & Wallace, Z. S. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105797 · Cited by 5 · see citing works
Self-citation
- Authors
- Yang, J., Wang, L., Phadke, N. A., Wickner, P. G., Mancini, C. M., Blumenthal, K. G., & Zhou, L.
- Affiliations
- Brigham and Women's Hospital (US)
- Harvard University (US)
- Corresponding author
- Yang, Jie
- ORCID
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2020-11-16
- Volume
- 3
- Issue
- 11
- Pages
- e2022836
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC7670315
- OpenAlex
- W3102840478
- Cited by
- 42
- References
- 38
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 8 · 2023: 8 · 2022: 6 · 2021: 5 · 2020: 2
- FWCI
- 3.68
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- American Academy of Allergy Asthma and Immunology
- Nvidia
- Agency for Healthcare Research and Quality (R01 HS025375)
- National Institute of Allergy and Infectious Diseases (R01AI150295)
- Topics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Pharmacology › Drug-Induced Adverse Reactions
- Keywords
- allergic reactions, adverse drug reactions, deep learning, natural language processing, automated detection, patient safety, generalizability, interpretability, adverse event monitoring, Brigham and Women's Hospital, Massachusetts General Hospital, Boston, United States
- MeSH terms
- Deep Learning, Adult, Aged, Algorithms, Boston, Cross-Sectional Studies, Diagnosis, Computer-Assisted, Female, Humans, Hypersensitivity, Male, Middle Aged, ROC Curve, Reproducibility of Results, Patient Safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
