Development and Performance of the Pulmonary Embolism Result Forecast Model (PERFORM) for Computed Tomography Clinical Decision Support
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Importance: Pulmonary embolism (PE) is a life-threatening clinical problem, and computed tomographic imaging is the standard for diagnosis. Clinical decision support rules based on PE risk-scoring models have been developed to compute pretest probability but are underused and tend to underperform in practice, leading to persistent overuse of CT imaging for PE. Objective: To develop a machine learning model to generate a patient-specific risk score for PE by analyzing longitudinal clinical data as clinical decision support for patients referred for CT imaging for PE. Design, Setting, and Participants: In this diagnostic study, the proposed workflow for the machine learning model, the Pulmonary Embolism Result Forecast Model (PERFORM), transforms raw electronic medical record (EMR) data into temporal feature vectors and develops a decision analytical model targeted toward adult patients referred for CT imaging for PE. The model was tested on holdout patient EMR data from 2 large, academic medical practices. A total of 3397 annotated CT imaging examinations for PE from 3214 unique patients seen at Stanford University hospitals and clinics were used for training and validation. The models were externally validated on 240 unique patients seen at Duke University Medical Center. The comparison with clinical scoring systems was done on randomly selected 100 outpatient samples from Stanford University hospitals and clinics and 101 outpatient samples from Duke University Medical Center. Main Outcomes and Measures: Prediction performance of diagnosing acute PE was evaluated using ElasticNet, artificial neural networks, and other machine learning approaches on holdout data sets from both institutions, and performance of models was measured by area under the receiver operating characteristic curve (AUROC). Results: Of the 3214 patients included in the study, 1704 (53.0%) were women from Stanford University hospitals and clinics; mean (SD) age was 60.53 (19.43) years. The 240 patients from Duke University Medical Center used for validation included 132 women (55.0%); mean (SD) age was 70.2 (14.2) years. In the samples for clinical scoring system comparisons, the 100 outpatients from Stanford University hospitals and clinics included 67 women (67.0%); mean (SD) age was 57.74 (19.87) years, and the 101 patients from Duke University Medical Center included 59 women (58.4%); mean (SD) age was 73.06 (15.3) years. The best-performing model achieved an AUROC performance of predicting a positive PE study of 0.90 (95% CI, 0.87-0.91) on intrainstitutional holdout data with an AUROC of 0.71 (95% CI, 0.69-0.72) on an external data set from Duke University Medical Center; superior AUROC performance and cross-institutional generalization of the model of 0.81 (95% CI, 0.77-0.87) and 0.81 (95% CI, 0.73-0.82), respectively, were noted on holdout outpatient populations from both intrainstitutional and extrainstitutional data. Conclusions and Relevance: The machine learning model, PERFORM, may consider multitudes of applicable patient-specific risk factors and dependencies to arrive at a PE risk prediction that generalizes to new population distributions. This approach might be used as an automated clinical decision-support tool for patients referred for CT PE imaging to improve CT use.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710
Mining electronic health records: towards better research applications and clinical care
Jensen, P. B., Jensen, L. J., & Brunak, S. (2012) · Nature Reviews Genetics · doi:10.1038/nrg3208 · Cited by 1,680
Deep Learning: A Primer for Radiologists
Chartrand, G., Cheng, P. M., Vorontsov, E., Drozdzal, M., Turcotte, S., Pal, C. J., Kadoury, S., & Tang, A. (2017) · RadioGraphics · doi:10.1148/rg.2017170077 · Cited by 1,115
Research works printed but not linked: 31
- Perrier, 2005, Multidetector-row computed tomography in suspected pulmonary embolism., N Engl J Med, 352, 1760, 10.1056/NEJMoa042905 · DOI · Paper page
- Albrecht, 2017, State-of-the-art pulmonary CT angiography for acute pulmonary embolism., AJR Am J Roentgenol, 208, 495, 10.2214/AJR.16.17202 · DOI
- Mountain, 2017, Correction: RESPECT-ED: rates of pulmonary emboli (PE) and sub-segmental PE with modern computed tomographic pulmonary angiograms in emergency departments: a multi-center observational study finds significant yield variation, uncorrelated with use or small PE rates., PLoS One, 12, e0184219, 10.1371/journal.pone.0184219 · doi:10.1371/journal.pone.0184219
- Dhakal, Overutilisation of imaging studies for diagnosis of pulmonary embolism: are we following the guidelines?, Postgrad Med J
- Oren, 2019, Curbing unnecessary and wasted diagnostic imaging., JAMA, 321, 245, 10.1001/jama.2018.20295 · DOI
- Venkatesh, 2012, Evaluation of pulmonary embolism in the emergency department and consistency with a national quality measure: quantifying the opportunity for improvement., Arch Intern Med, 172, 1028, 10.1001/archinternmed.2012.1804 · doi:10.1001/archinternmed.2012.1804
- Wiener, 2015, An official American Thoracic Society/American College of Chest Physicians policy statement: implementation of low-dose computed tomography lung cancer screening programs in clinical practice., Am J Respir Crit Care Med, 192, 881, 10.1164/rccm.201508-1671ST · DOI · Paper page
- Alhassan, 2016, Suboptimal implementation of diagnostic algorithms and overuse of computed tomography-pulmonary angiography in patients with suspected pulmonary embolism., Ann Thorac Med, 11, 254, 10.4103/1817-1737.191875 · doi:10.4103/1817-1737.191875
- Chong, 2018, Association of lower diagnostic yield with high users of CT pulmonary angiogram., JAMA Intern Med, 178, 412, 10.1001/jamainternmed.2017.7552 · doi:10.1001/jamainternmed.2017.7552
- Hentel, 2017, New CMS clinical decision support regulations: a potential opportunity with major challenges., Radiology, 283, 10, 10.1148/radiol.2017161560 · DOI
- Wang, 2016, The impact of clinical decision rules on computed tomography use and yield for pulmonary embolism: a systematic review and meta-analysis., Ann Emerg Med, 67, 693, 10.1016/j.annemergmed.2015.11.005 · DOI
- Bautista, 2009, Do clinicians use the American College of Radiology appropriateness criteria in the management of their patients?, AJR Am J Roentgenol, 192, 1581, 10.2214/AJR.08.1622 · DOI
- Gonsalves, 2008, Incidence of symptomatic venous thromboembolism following hematopoietic stem cell transplantation., J Thromb Haemost, 6, 1468, 10.1111/jth.2008.6.issue-9 · doi:10.1111/jth.2008.6.issue-9
- Yao, 2011, Electronic health records: implications for drug discovery., Drug Discov Today, 16, 594, 10.1016/j.drudis.2011.05.009 · DOI
- Banerjee, 2018, Radiology report annotation using intelligent word embeddings: applied to multi-institutional chest CT cohort., J Biomed Inform, 77, 11, 10.1016/j.jbi.2017.11.012 · DOI · Paper page
- Banerjee, 2019, Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification., Artif Intell Med, 97, 79, 10.1016/j.artmed.2018.11.004 · DOI · Paper page
- Zou, 2005, Regularization and variable selection via the elastic net., J R Stat Soc Series B Stat Methodol, 67, 301, 10.1111/rssb.2005.67.issue-2 · doi:10.1111/rssb.2005.67.issue-2
- Theunissen, 2016, A retrospective analysis of the combined use of PERC rule and Wells score to exclude pulmonary embolism in the emergency department., Emerg Med J, 33, 696, 10.1136/emermed-2016-205687 · DOI
- Crane, 2018, Retrospective validation of the pulmonary embolism rule-out criteria rule in “PE unlikely” patients with suspected pulmonary embolism., Eur J Emerg Med, 25, 185, 10.1097/MEJ.0000000000000442 · doi:10.1097/mej.0000000000000442
- Penaloza, 2013, Comparison of the unstructured clinician gestalt, the wells score, and the revised Geneva score to estimate pretest probability for suspected pulmonary embolism., Ann Emerg Med, 62, 117, 10.1016/j.annemergmed.2012.11.002 · DOI · Paper page
- Klok, 2008, Comparison of the revised Geneva score with the Wells rule for assessing clinical probability of pulmonary embolism., J Thromb Haemost, 6, 40, 10.1111/j.1538-7836.2007.02820.x · DOI · Paper page
- Penaloza, 2012, Performance of the Pulmonary Embolism Rule-out Criteria (the PERC rule) combined with low clinical probability in high prevalence population., Thromb Res, 129, e189, 10.1016/j.thromres.2012.02.016 · doi:10.1016/j.thromres.2012.02.016
- Wasson, 1996, Clinical prediction rules: have they come of age?, JAMA, 275, 641, 10.1001/jama.1996.03530320065037 · doi:10.1001/jama.1996.03530320065037
- Squizzato, 2012, Prognostic clinical prediction rules to identify a low-risk pulmonary embolism: a systematic review and meta-analysis., J Thromb Haemost, 10, 1276, 10.1111/j.1538-7836.2012.04739.x · DOI
- Chagnon, 2002, Comparison of two clinical prediction rules and implicit assessment among patients with suspected pulmonary embolism., Am J Med, 113, 269, 10.1016/S0002-9343(02)01212-3 · DOI · Paper page
- Le Gal, 2006, Prediction of pulmonary embolism in the emergency department: the revised Geneva score., Ann Intern Med, 144, 165, 10.7326/0003-4819-144-3-200602070-00004 · DOI · Paper page
- Blondon, 2012, Usefulness of preemptive anticoagulation in patients with suspected pulmonary embolism: a decision analysis., Chest, 142, 697, 10.1378/chest.11-2694 · DOI
- Fisher, 1992, The accuracy of Medicare’s hospital claims data: progress has been made, but problems remain., Am J Public Health, 82, 243, 10.2105/AJPH.82.2.243 · DOI · Paper page
- He, 2014, Mining high-dimensional administrative claims data to predict early hospital readmissions., J Am Med Inform Assoc, 21, 272, 10.1136/amiajnl-2013-002151 · DOI · Paper page
- Amrhein, 2014, Physician self-referral and imaging use appropriateness: negative cervical spine MRI frequency as an assessment metric., AJNR Am J Neuroradiol, 35, 2248, 10.3174/ajnr.A4076 · doi:10.3174/ajnr.a4076
- Penaloza, 2012, Performance of age-adjusted d-dimer cut-off to rule out pulmonary embolism., J Thromb Haemost, 10, 1291, 10.1111/j.1538-7836.2012.04769.x · DOI
Cited by
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Self-citationThe potential of artificial intelligence to improve patient safety: a scoping review
Bates, D. W., Levine, D., Syrowatka, A., Kuznetsova, M., Craig, K. J. T., Rui, A., Jackson, G. P., & Rhee, K. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00423-6 · Cited by 323
Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., & Lungren, M. P. (2020) · Scientific Reports · doi:10.1038/s41598-020-78888-w · Cited by 246
Self-citationDiprose, W. K., Buist, N., Hua, N., Thurier, Q., Shand, G., & Robinson, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz229 · Cited by 188
Metadata
- Authors
- Banerjee, I., Sofela, M., Yang, J., Chen, J. H., Shah, N. H., Ball, R., Mushlin, A. I., Desai, M., Bledsoe, J., Amrhein, T., Rubin, D. L., Zamanian, R., & Lungren, M. P.
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2019-08-07
- Volume
- 2
- Issue
- 8
- Pages
- e198719
- ISSN
- 2574-3805 (ISSN-L)
- OpenAlex
- W2966167372
- Cited by
- 80
- References
- 4
- FWCI
- 6.80
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development and Performance of the Pulmonary Embolism Result Forecast Model (PERFORM) for Computed Tomography Clinical Decision Support. https://app.cochat.ai/papers/10.1001/jamanetworkopen.2019.8719
- The paper
- Banerjee, I., Sofela, M., Yang, J., Chen, J. H., Shah, N. H., Ball, R., Mushlin, A. I., Desai, M., Bledsoe, J., Amrhein, T., Rubin, D. L., Zamanian, R., & Lungren, M. P. (2019). Development and Performance of the Pulmonary Embolism Result Forecast Model (PERFORM) for Computed Tomography Clinical Decision Support. JAMA Network Open. https://doi.org/10.1001/jamanetworkopen.2019.8719
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.