Zoo or Savannah? Choice of Training Ground for Evidence-Based Pharmacovigilance
Open accessAbstract
Pharmacovigilance seeks to detect and describe adverse drug reactions early. Ideally, we would like to see objective evidence that a chosen signal detection approach can be expected to be effective. The development and evaluation of evidence-based methods require benchmarks for signal detection performance, and recent years have seen unprecedented efforts to build such reference sets. Here, we argue that evaluation should be made against emerging and not established adverse drug reactions, and we present real-world examples that illustrate the relevance of this to pharmacovigilance methods development for both individual case reports and longitudinal health records. The establishment of broader reference sets of emerging safety signals must be made a top priority to achieve more effective pharmacovigilance methods development and evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To ensure relevance and avoid trivial classification, non-cases (controls) were limited to patients with more than one visit at Stanford, at least one visit within the last 2 years, and one comorbidity from hypertension, coronary atherosclerosis, dyslipidemia, or myocardial infarction.
“To ensure relevance of our classifier for use on patients at a higher risk for ASCVD for discerning between FH cases and similar non-FH cases “in the wild”,37,38 and to ensure that trivial classification (e.g., healthy 25-year olds classified as non-FH) does not produce optimistic results, we limited our non-cases (“controls”) to patients that: (1) had more than one visit at Stanford; (2) had at l”
From Finding missed cases of familial hypercholesterolemia in health systems using machine learning (2019) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Schneeweiss, S., Rassen, J. A., Glynn, R. J., Avorn, J., Mogun, H., & Brookhart, M. A. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181a663cc · Cited by 1,128 · see citing works
Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery
Norén, G. N., Hopstadius, J., & Bate, A. (2013) · Statistical Methods in Medical Research · doi:10.1177/0962280211403604 · Cited by 336 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 19
- Bate A, Evans SJW. Quantitative signal detection using sponta- neous ADR reporting. Pharmacoepidemiol Drug Saf. 2009;18(6): 427-36.
- Nore ´n GN, Hopstadius J, Bate A, Star K, Edwards IR. Temporal pattern discovery in longitudinal electronic patient records. Data Min Knowl Discov. 2010;20(3):361-87.
- White RW, Tatonetti NP, Shah NH, Altman RB, Horvitz E. Web- scale pharmacovigilance: listening to signals from the crowd. J Am Med Inform Assoc. 2013;20(3):404-8.
- Freifeld CC, Brownstein JS, Menone CM, et al. Digital drug safety surveillance: monitoring pharmaceutical products in Twitter. Drug Saf. 2014;37(5):343-50.
- Caster O, Nore ´n GN, Madigan D, Bate A. Large-scale regression- based pattern discovery: the example of screening the WHO global drug safety database. Stat Anal Data Min. 2010;3(4): 197-208.
- Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA Adverse Event Reporting System. Clin Pharmacol Ther. 2013;93(6):539-46.
- Tatonetti NP, Ye PP, Daneshjou R, Altman RB. Data-driven prediction of drug effects and interactions. Sci Transl Med. 2012;4(125):125-31.
- Farrington CP, Nash J, Miller E. Case series analysis of adverse reactions to vaccines: a comparative evaluation. Am J Epidemiol. 1996;143(11):1165-73.
- Ryan P, Schuemie M, Welebob E, Duke J, Valentine S, Hartzema A. Defining a reference set to support methodological research in drug safety. Drug Saf. 2013;36(1):33-47.
- Coloma P, Avillach P, Salvo F, et al. A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases. Drug Saf. 2013;36(1): 13-23.
- Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of sta- tistical signal detection procedures in Eudravigilance post- authorization data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475-87.
- Caster O, Nore ´n GN, Madigan D, Bate A. Logistic regression in signal detection: another piece added to the puzzle. Clin Phar- macol Ther. 2013;94(3):312.
- Hallas J. Evidence of depression provoked by cardiovascular medication: a prescription sequence symmetry analysis. Epide- miology. 1996;7:478-84.
- Nore ´n GN, Bergvall T, Ryan PB, Juhlin K, Schuemie MJ, Madigan D. Empirical performance of the calibrated self-con- trolled cohort analysis within temporal pattern discovery: lessons for developing a risk identification and analysis system. Drug Saf. 2013;36(1):107-21.
- Bailey S, Singh A, Azadian R, Huber P, Blum M. Prospective data mining of six products in the US FDA adverse event reporting system. Drug Saf. 2010;33(2):139-146.
- Hauben M, Nore ´n GN. A decade of data mining and still counting. Drug Saf. 2010;33(7):527-34.
- Lindquist M, Stahl M, Bate A, Edwards IR, Meyboom RH. A retrospective evaluation of a data mining approach to aid finding new adverse drug reaction signals in the WHO international database. Drug Saf. 2000;23(6):533-42.
- Strandell J, Caster O, Hopstadius J, Edwards IR, Nore ´n GN. The development and evaluation of triage algorithms for early discovery of adverse drug interactions. Drug Saf. 2013;36(5): 371-88.
- Boyce RD, Ryan PB, Nore ´n GN, et al. Bridging islands of information to establish an integrated knowledge base of drugs and health outcomes of interest. Drug Saf. doi:10.1007/s40264- 014-0189-0. · DOI
Cited by
Sorted by citations
Conducting and interpreting disproportionality analyses derived from spontaneous reporting systems
Cutroneo, P. M., Sartori, D., Tuccori, M., Crisafulli, S., Battini, V., Carnovale, C., Rafaniello, C., Capuano, A., Poluzzi, E., Moretti, U., et al. (2024) · Frontiers in Drug Safety and Regulation · doi:10.3389/fdsfr.2023.1323057 · Cited by 224 · see citing works
Schuemie, M. J., Hripcsak, G., Ryan, P. B., Madigan, D., & Suchard, M. A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708282114 · Cited by 163 · see citing works
Finding missed cases of familial hypercholesterolemia in health systems using machine learning
Banda, J. M., Sarraju, A., Abbasi, F., Parizo, J., Pariani, M., Ison, H., Briskin, E., Wand, H., Dubois, S., Jung, K., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0101-5 · Cited by 123 · see citing works
Improving reproducibility by using high-throughput observational studies with empirical calibration
Schuemie, M. J., Ryan, P. B., Hripcsak, G., Madigan, D., & Suchard, M. A. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0356 · Cited by 74 · see citing works
Osokogu, O. U., Fregonese, F., Ferrajolo, C., Verhamme, K., de Bie, S., Jong, G. ’., Catapano, M., Weibel, D., Kaguelidou, F., Bramer, W. M., et al. (2015) · Drug Safety · doi:10.1007/s40264-015-0265-0 · Cited by 31 · see citing works
- Authors
- Norén, G. N., Caster, O., Juhlin, K., & Lindquist, M.
- Affiliations
- Uppsala Monitoring Centre (SE)
- Corresponding author
- Norén, G. Niklas
- Venue
- Drug Safety
- Published
- 2014-07-09
- Volume
- 37
- Issue
- 9
- Pages
- 655-659
- ISSN
- 0114-5916 (ISSN-L)
- 1179-1942
- OpenAlex
- W2149676192
- Cited by
- 40
- References
- 2
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 1 · 2023: 3 · 2022: 1 · 2021: 1 · 2020: 4 · 2019: 3 · 2018: 5 · 2017: 3 · 2016: 3 · 2015: 6 · 2014: 5
- FWCI
- 4.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Drug Safety (publishedVersion)
- PubMed (publishedVersion)
- Funders
- European Commission (115004)
- Topics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
- Health Sciences › Medicine › Epidemiology › Data-Driven Disease Surveillance
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Keywords
- pharmacovigilance, signal detection, adverse drug reactions, reference set, benchmarking, longitudinal health records, evidence-based methods
- MeSH terms
- Humans, Global Health, Adverse Drug Reaction Reporting Systems, Evidence-Based Medicine, Benchmarking, Pharmacovigilance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
