CoChat

Development and Performance of the Pulmonary Embolism Result Forecast Model (PERFORM) for Computed Tomography Clinical Decision Support

Open access
Banerjee, Imon, Sofela, Miji, Yang, Jaden, Chen, Jonathan H., Shah, Nigam H., Ball, Robyn, Mushlin, Alvin I., Desai, Manisha, Bledsoe, Joseph, Amrhein, Timothy, Rubin, Daniel L., Zamanian, Roham, Lungren, Matthew P. · 2019 · JAMA Network Open
Cited by 80

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

Importance: Pulmonary embolism (PE) is a life-threatening clinical problem, and computed tomographic imaging is the standard for diagnosis. Clinical decision support rules based on PE risk-scoring models have been developed to compute pretest probability but are underused and tend to underperform in practice, leading to persistent overuse of CT imaging for PE. Objective: To develop a machine learning model to generate a patient-specific risk score for PE by analyzing longitudinal clinical data as clinical decision support for patients referred for CT imaging for PE. Design, Setting, and Participants: In this diagnostic study, the proposed workflow for the machine learning model, the Pulmonary Embolism Result Forecast Model (PERFORM), transforms raw electronic medical record (EMR) data into temporal feature vectors and develops a decision analytical model targeted toward adult patients referred for CT imaging for PE. The model was tested on holdout patient EMR data from 2 large, academic medical practices. A total of 3397 annotated CT imaging examinations for PE from 3214 unique patients seen at Stanford University hospitals and clinics were used for training and validation. The models were externally validated on 240 unique patients seen at Duke University Medical Center. The comparison with clinical scoring systems was done on randomly selected 100 outpatient samples from Stanford University hospitals and clinics and 101 outpatient samples from Duke University Medical Center. Main Outcomes and Measures: Prediction performance of diagnosing acute PE was evaluated using ElasticNet, artificial neural networks, and other machine learning approaches on holdout data sets from both institutions, and performance of models was measured by area under the receiver operating characteristic curve (AUROC). Results: Of the 3214 patients included in the study, 1704 (53.0%) were women from Stanford University hospitals and clinics; mean (SD) age was 60.53 (19.43) years. The 240 patients from Duke University Medical Center used for validation included 132 women (55.0%); mean (SD) age was 70.2 (14.2) years. In the samples for clinical scoring system comparisons, the 100 outpatients from Stanford University hospitals and clinics included 67 women (67.0%); mean (SD) age was 57.74 (19.87) years, and the 101 patients from Duke University Medical Center included 59 women (58.4%); mean (SD) age was 73.06 (15.3) years. The best-performing model achieved an AUROC performance of predicting a positive PE study of 0.90 (95% CI, 0.87-0.91) on intrainstitutional holdout data with an AUROC of 0.71 (95% CI, 0.69-0.72) on an external data set from Duke University Medical Center; superior AUROC performance and cross-institutional generalization of the model of 0.81 (95% CI, 0.77-0.87) and 0.81 (95% CI, 0.73-0.82), respectively, were noted on holdout outpatient populations from both intrainstitutional and extrainstitutional data. Conclusions and Relevance: The machine learning model, PERFORM, may consider multitudes of applicable patient-specific risk factors and dependencies to arrive at a PE risk prediction that generalizes to new population distributions. This approach might be used as an automated clinical decision-support tool for patients referred for CT PE imaging to improve CT use.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2019.8719

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 31

  1. Perrier, 2005, Multidetector-row computed tomography in suspected pulmonary embolism., N Engl J Med, 352, 1760, 10.1056/NEJMoa042905 · DOI · Paper page
  2. Albrecht, 2017, State-of-the-art pulmonary CT angiography for acute pulmonary embolism., AJR Am J Roentgenol, 208, 495, 10.2214/AJR.16.17202 · DOI
  3. Mountain, 2017, Correction: RESPECT-ED: rates of pulmonary emboli (PE) and sub-segmental PE with modern computed tomographic pulmonary angiograms in emergency departments: a multi-center observational study finds significant yield variation, uncorrelated with use or small PE rates., PLoS One, 12, e0184219, 10.1371/journal.pone.0184219 · doi:10.1371/journal.pone.0184219
  4. Dhakal, Overutilisation of imaging studies for diagnosis of pulmonary embolism: are we following the guidelines?, Postgrad Med J
  5. Oren, 2019, Curbing unnecessary and wasted diagnostic imaging., JAMA, 321, 245, 10.1001/jama.2018.20295 · DOI
  6. Venkatesh, 2012, Evaluation of pulmonary embolism in the emergency department and consistency with a national quality measure: quantifying the opportunity for improvement., Arch Intern Med, 172, 1028, 10.1001/archinternmed.2012.1804 · doi:10.1001/archinternmed.2012.1804
  7. Wiener, 2015, An official American Thoracic Society/American College of Chest Physicians policy statement: implementation of low-dose computed tomography lung cancer screening programs in clinical practice., Am J Respir Crit Care Med, 192, 881, 10.1164/rccm.201508-1671ST · DOI · Paper page
  8. Alhassan, 2016, Suboptimal implementation of diagnostic algorithms and overuse of computed tomography-pulmonary angiography in patients with suspected pulmonary embolism., Ann Thorac Med, 11, 254, 10.4103/1817-1737.191875 · doi:10.4103/1817-1737.191875
  9. Chong, 2018, Association of lower diagnostic yield with high users of CT pulmonary angiogram., JAMA Intern Med, 178, 412, 10.1001/jamainternmed.2017.7552 · doi:10.1001/jamainternmed.2017.7552
  10. Hentel, 2017, New CMS clinical decision support regulations: a potential opportunity with major challenges., Radiology, 283, 10, 10.1148/radiol.2017161560 · DOI
  11. Wang, 2016, The impact of clinical decision rules on computed tomography use and yield for pulmonary embolism: a systematic review and meta-analysis., Ann Emerg Med, 67, 693, 10.1016/j.annemergmed.2015.11.005 · DOI
  12. Bautista, 2009, Do clinicians use the American College of Radiology appropriateness criteria in the management of their patients?, AJR Am J Roentgenol, 192, 1581, 10.2214/AJR.08.1622 · DOI
  13. Gonsalves, 2008, Incidence of symptomatic venous thromboembolism following hematopoietic stem cell transplantation., J Thromb Haemost, 6, 1468, 10.1111/jth.2008.6.issue-9 · doi:10.1111/jth.2008.6.issue-9
  14. Yao, 2011, Electronic health records: implications for drug discovery., Drug Discov Today, 16, 594, 10.1016/j.drudis.2011.05.009 · DOI
  15. Banerjee, 2018, Radiology report annotation using intelligent word embeddings: applied to multi-institutional chest CT cohort., J Biomed Inform, 77, 11, 10.1016/j.jbi.2017.11.012 · DOI · Paper page
  16. Banerjee, 2019, Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification., Artif Intell Med, 97, 79, 10.1016/j.artmed.2018.11.004 · DOI · Paper page
  17. Zou, 2005, Regularization and variable selection via the elastic net., J R Stat Soc Series B Stat Methodol, 67, 301, 10.1111/rssb.2005.67.issue-2 · doi:10.1111/rssb.2005.67.issue-2
  18. Theunissen, 2016, A retrospective analysis of the combined use of PERC rule and Wells score to exclude pulmonary embolism in the emergency department., Emerg Med J, 33, 696, 10.1136/emermed-2016-205687 · DOI
  19. Crane, 2018, Retrospective validation of the pulmonary embolism rule-out criteria rule in “PE unlikely” patients with suspected pulmonary embolism., Eur J Emerg Med, 25, 185, 10.1097/MEJ.0000000000000442 · doi:10.1097/mej.0000000000000442
  20. Penaloza, 2013, Comparison of the unstructured clinician gestalt, the wells score, and the revised Geneva score to estimate pretest probability for suspected pulmonary embolism., Ann Emerg Med, 62, 117, 10.1016/j.annemergmed.2012.11.002 · DOI · Paper page
  21. Klok, 2008, Comparison of the revised Geneva score with the Wells rule for assessing clinical probability of pulmonary embolism., J Thromb Haemost, 6, 40, 10.1111/j.1538-7836.2007.02820.x · DOI · Paper page
  22. Penaloza, 2012, Performance of the Pulmonary Embolism Rule-out Criteria (the PERC rule) combined with low clinical probability in high prevalence population., Thromb Res, 129, e189, 10.1016/j.thromres.2012.02.016 · doi:10.1016/j.thromres.2012.02.016
  23. Wasson, 1996, Clinical prediction rules: have they come of age?, JAMA, 275, 641, 10.1001/jama.1996.03530320065037 · doi:10.1001/jama.1996.03530320065037
  24. Squizzato, 2012, Prognostic clinical prediction rules to identify a low-risk pulmonary embolism: a systematic review and meta-analysis., J Thromb Haemost, 10, 1276, 10.1111/j.1538-7836.2012.04739.x · DOI
  25. Chagnon, 2002, Comparison of two clinical prediction rules and implicit assessment among patients with suspected pulmonary embolism., Am J Med, 113, 269, 10.1016/S0002-9343(02)01212-3 · DOI · Paper page
  26. Le Gal, 2006, Prediction of pulmonary embolism in the emergency department: the revised Geneva score., Ann Intern Med, 144, 165, 10.7326/0003-4819-144-3-200602070-00004 · DOI · Paper page
  27. Blondon, 2012, Usefulness of preemptive anticoagulation in patients with suspected pulmonary embolism: a decision analysis., Chest, 142, 697, 10.1378/chest.11-2694 · DOI
  28. Fisher, 1992, The accuracy of Medicare’s hospital claims data: progress has been made, but problems remain., Am J Public Health, 82, 243, 10.2105/AJPH.82.2.243 · DOI · Paper page
  29. He, 2014, Mining high-dimensional administrative claims data to predict early hospital readmissions., J Am Med Inform Assoc, 21, 272, 10.1136/amiajnl-2013-002151 · DOI · Paper page
  30. Amrhein, 2014, Physician self-referral and imaging use appropriateness: negative cervical spine MRI frequency as an assessment metric., AJNR Am J Neuroradiol, 35, 2248, 10.3174/ajnr.A4076 · doi:10.3174/ajnr.a4076
  31. Penaloza, 2012, Performance of age-adjusted d-dimer cut-off to rule out pulmonary embolism., J Thromb Haemost, 10, 1291, 10.1111/j.1538-7836.2012.04769.x · DOI

Cited by

See all 80

Metadata

Authors
Banerjee, I., Sofela, M., Yang, J., Chen, J. H., Shah, N. H., Ball, R., Mushlin, A. I., Desai, M., Bledsoe, J., Amrhein, T., Rubin, D. L., Zamanian, R., & Lungren, M. P.
Venue
JAMA Network Open
Published
2019-08-07
Volume
2
Issue
8
Pages
e198719
ISSN
2574-3805 (ISSN-L)
OpenAlex
W2966167372
Cited by
80
References
4
FWCI
6.80
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.