CoChat

Enhancing Postmarketing Surveillance of Medical Products With Large Language Models

Open access
Matheny, Michael E., Yang, Jie, Smith, Joshua C., Walsh, Colin G., Al-Garadi, Mohammed A., Davis, Sharon E., Marsolo, Keith A., Fabbri, Daniel, Reeves, Ruth R., Johnson, Kevin B., Dal Pan, Gerald J., Ball, Robert, Desai, Rishi J. · 2024 · JAMA Network Open
Cited by 38

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

Abstract

Importance: The Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes. Observations: LLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost. Conclusions and Relevance: LLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2024.28276

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 27 references

Cited by

See all 38

Metadata

Authors
Matheny, M. E., Yang, J., Smith, J. C., Walsh, C. G., Al-Garadi, M. A., Davis, S. E., Marsolo, K. A., Fabbri, D., Reeves, R. R., Johnson, K. B., Dal Pan, G. J., Ball, R., & Desai, R. J.
Venue
JAMA Network Open
Published
2024-08-16
Volume
7
Issue
8
Pages
e2428276
ISSN
2574-3805 (ISSN-L)
OpenAlex
W4401635636
Cited by
38
References
27
FWCI
29.14
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Enhancing Postmarketing Surveillance of Medical Products With Large Language Models. https://app.cochat.ai/papers/10.1001/jamanetworkopen.2024.28276
The paper
Matheny, M. E., Yang, J., Smith, J. C., Walsh, C. G., Al-Garadi, M. A., Davis, S. E., Marsolo, K. A., Fabbri, D., Reeves, R. R., Johnson, K. B., Dal Pan, G. J., Ball, R., & Desai, R. J. (2024). Enhancing Postmarketing Surveillance of Medical Products With Large Language Models. JAMA Network Open. https://doi.org/10.1001/jamanetworkopen.2024.28276

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.