The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Comparing clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment: a pilot usability study.
Open accessAbstract
Background Major postoperative complications are associated with increased cost and mortality. The complexity of electronic health records overwhelms physicians' abilities to use the information for optimal and timely preoperative risk assessment. We hypothesized that data-driven, predictive-risk algorithms implemented in an intelligent decision-support platform simplify and augment physicians' risk assessments.Methods This prospective, nonrandomized pilot study of 20 physicians at a quaternary academic medical center compared the usability and accuracy of preoperative risk assessment between physicians and MySurgeryRisk, a validated, machine-learning algorithm, using a simulated workflow for the real-time, intelligent decision-support platform. We used area under the receiver operating characteristic curve to compare the accuracy of physicians' risk assessment for six postoperative complications before and after interaction with the algorithm for 150 clinical cases.Results The area under the receiver operating characteristic curve of the MySurgeryRisk algorithm ranged between 0.73 and 0.85 and was significantly better than physicians' initial risk assessments (area under the receiver operating characteristic curve between 0.47 and 0.69) for all postoperative complications except cardiovascular. After interaction with the algorithm, the physicians significantly improved their risk assessment for acute kidney injury and for an intensive care unit admission greater than 48 hours, resulting in a net improvement of reclassification of 12% and 16%, respectively. Physicians rated the algorithm as easy to use and useful.Conclusion Implementation of a validated, MySurgeryRisk computational algorithm for real-time predictive analytics with data derived from the electronic health records to augment physicians' decision-making is feasible and accepted by physicians. Early involvement of physicians as key stakeholders in both design and implementation of this technology will be crucial for its future success.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a prospective study, the algorithm predicted postoperative complications with greater accuracy than physicians.17”
From Artificial Intelligence and Surgical Decision-making. (2020) · cites this paper as [17]
Experimental — accuracy not yet measuredIn the citing paper
“Brennan et al [ 65 ] found that individuals who lean toward analytical thinking, which may be associated with a high need for cognition, exhibited a higher likelihood of changing their decisions after interacting with an algorithm.”
From Psychological Factors Influencing Appropriate Reliance on AI-enabled Clinical Decision Support Systems: Experimental Web-Based Study Among Dermatologists (2025) · cites this paper as [65]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Comorbidity measures for use with administrative data.
Elixhauser, A., Steiner, C., Harris, D. R., & Coffey, R. M. (1998) · Medical Care · doi:10.1097/00005650-199801000-00004 · Cited by 9,921
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,483
POSSUM: A scoring system for surgical audit
Copeland, G. P., Jones, D., & Walters, M. (1991) · British journal of surgery · doi:10.1002/bjs.1800780327 · Cited by 1,716
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688
Self-citationGeneral Performance on a Numeracy Scale among Highly Educated Samples
Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001) · Medical Decision Making · doi:10.1177/0272989x0102100105 · Cited by 1,501
Khuri, S. F., Henderson, W. G., DePalma, R. G., Mosca, C. L., Healey, N. A., & Kumbhani, D. J. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000179621.33268.83 · Cited by 1,451
The Role of Numeracy in Understanding the Benefit of Screening Mammography
Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-11-199712010-00003 · Cited by 1,132
Hospitalizations, costs, and outcomes of severe sepsis in the United States 2003 to 2007
Lagu, T., Rothberg, M. B., Shieh, M.-S., Pekow, P. S., Steingrub, J. S., & Lindenauer, P. K. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e318232db65 · Cited by 679
Gawande, A. A., Kwaan, M. R., Regenbogen, S. E., Lipsitz, S. R., & Zinner, M. J. (2006) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2006.11.011 · Cited by 504
Cost and Mortality Associated With Postoperative Acute Kidney Injury
Hobson, C., Ozrazgat‐Baslanti, T., Kuxhausen, A. N., Thottakkara, P., Efron, P. A., Moore, F. A., Moldawer, L. L., Segal, M. S., & Bihorac, A. (2014) · Annals of Surgery · doi:10.1097/sla.0000000000000732 · Cited by 402
Self-citationBihorac, A., Ozrazgat-Baslanti, T., Ebadi, A., Motaei, A., Madkour, M., Pardalos, P. M., Lipori, G., Hogan, W. R., Efron, P. A., Moore, F., et al. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000002706 · Cited by 377
Self-citationClinical Implications and Challenges of Artificial Intelligence and Deep Learning
Stead, W. W. (2018) · JAMA · doi:10.1001/jama.2018.11029 · Cited by 250
Thottakkara, P., Ozrazgat‐Baslanti, T., Hupf, B., Rashidi, P., Pardalos, P. M., Momčilović, P., & Bihorac, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0155705 · Cited by 202
Self-citationPostoperative Sepsis in the United States
Vogel, T. R., Dombrovskiy, V. Y., Carson, J. L., Graham, A. M., & Lowry, S. F. (2010) · Annals of Surgery · doi:10.1097/sla.0b013e3181dcf36e · Cited by 197
Detsky, M. E., Harhay, M. O., Bayard, D. F., Delman, A. M., Buehler, A. E., Kent, S. A., Ciuffetelli, I. V., Cooney, E. L., Gabler, N. B., Ratcliffe, S. J., et al. (2017) · JAMA · doi:10.1001/jama.2017.4078 · Cited by 191
Comparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145
Predicting postoperative morbidity by clinical assessment
Markus, P. M., Martell, J., Leister, I., Horstmann, O., Brinker, J., & Heinz Becker, A. (2005) · British journal of surgery · doi:10.1002/bjs.4608 · Cited by 143
Bihorac, A., Brennan, M., Ozrazgat‐Baslanti, T., Bozorgmehri, S., Efron, P. A., Moore, F. A., Segal, M. S., & Hobson, C. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31829860fc · Cited by 107
Self-citationChanges in Prognosis After the First Postoperative Complication
Silber, J. H., Rosenbaum, P. R., Trudeau, M. E., Chen, W., Zhang, X., Kelz, R. R., Mosher, R. E., & Even‐Shoshan, O. (2005) · Medical Care · doi:10.1097/00005650-200502000-00005 · Cited by 94
Diagnostic performance of a computer-based ECG rhythm algorithm
Poon, K., Okin, P. M., & Kligfield, P. (2005) · Journal of Electrocardiology · doi:10.1016/j.jelectrocard.2005.01.008 · Cited by 81
Liao, L., & Mark, D. B. (2003) · Journal of the American College of Cardiology · doi:10.1016/s0735-1097(03)00836-2 · Cited by 72
The surgeon's 'gut feeling' as a predictor of post-operative outcome.
Hartley, M., & Sagar, P. (1994) · PubMed · Cited by 69
Acute Kidney Injury in the Surgical Patient
Hobson, C., Singhania, G., & Bihorac, A. (2015) · Critical Care Clinics · doi:10.1016/j.ccc.2015.06.007 · Cited by 59
Self-citationIntelligent Perioperative System: Towards Real-Time Big Data Analytics in Surgery Risk Assessment
Zheng, F., Bhat, R. R., Yuan, X., Freeman, D., Baslanti, T. O., Bihorac, A., & Li, X. (2017) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/dasc-picom-datacom-cyberscitec.2017.201 · Cited by 52
Self-citationA Role for the Early Warning Score in Early Identification of Critical Postoperative Complications
Hollis, R. H., Graham, L. A., Lazenby, J. P., Brown, D. M., Taylor, B. B., Heslin, M. J., Rue, L. W., & Hawn, M. T. (2015) · Annals of Surgery · doi:10.1097/sla.0000000000001514 · Cited by 50
Acute Kidney Injury in the Surgical Patient: Recognition and Attribution
Bihorac, A. (2015) · The Nephron journals/Nephron journals · doi:10.1159/000439387 · Cited by 33
Self-citationIvanov, J., Borger, M. A., David, T. E., Cohen, G. N., Walton, N. A., & Naylor, C. D. (2000) · The Annals of Thoracic Surgery · doi:10.1016/s0003-4975(00)01387-4 · Cited by 31
Tepas, J. J., Rimar, J. M., Hsiao, A., & Nussbaum, M. S. (2013) · Surgery · doi:10.1016/j.surg.2013.07.014 · Cited by 28
The impact of web-based diabetes risk calculators on information processing and risk perceptions.
Harle, C. A., Padman, R., & Downs, J. S. (2008) · PubMed · Cited by 15
Hospitalizations, Costs, and Outcomes of Severe Sepsis in the United States 2003 to 2007
Taecker, M. (2012) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2012.06.011 · Cited by 14
Flagged first, then by citations
Cited by
Sorted by citations
Artificial Intelligence and Surgical Decision-making.
Loftus, T. J., Tighe, P. J., Filiberto, A. C., Efron, P. A., Brakenridge, S. C., Mohr, A. M., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2019.4917 · Cited by 650
Self-citationRole of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419
The Integration of Artificial Intelligence into Clinical Practice
Karalis, V. D. (2024) · Applied Biosciences · doi:10.3390/applbiosci3010002 · Cited by 255
Hassan, A. M., Rajesh, A., Asaad, M., Nelson, J. A., Coert, J. H., Mehrara, B. J., & Butler, C. E. (2022) · The American Surgeon · doi:10.1177/00031348221101488 · Cited by 156
Ren, Y., Loftus, T. J., Datta, S., Ruppert, M. M., Guan, Z., Miao, S., Shickel, B., Zheng, F., Giordano, C. R., Upchurch, G. R., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.11973 · Cited by 139
Self-citation
Metadata
- Authors
- Brennan, M., Puri, S., Ozrazgat-Baslanti, T., Feng, Z., Ruppert, M., Hashemighouchani, H., Momcilovic, P., Li, X., Wang, D. Z., & Bihorac, A.
- Affiliations
- University of Florida Health (US)
- University of Florida (US)
- Corresponding author
- Bihorac, Azra
- Venue
- Surgery
- Publisher
- Elsevier BV
- Published
- 2019-05
- Volume
- 165
- Issue
- 5
- Pages
- 1035-1045
- ISSN
- 0039-6060 (ISSN-L)
- PMC
- PMC6502657
- OpenAlex
- W2797775685
- Cited by
- 131
- References
- 32
- Citations by year
- 2026: 17 · 2025: 23 · 2024: 24 · 2023: 26 · 2022: 13 · 2021: 14 · 2020: 11 · 2019: 2
- FWCI
- 3.18
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Surgery (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Institutes of Health (UL1TR001427, UL1TR000064)
- National Institute of General Medical Sciences (P50GM111152, R01GM110240)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- preoperative risk assessment, clinical judgment, clinical decision support, postoperative complications, electronic health records, usability, physician acceptance, acute kidney injury, intensive care unit admission, surgical risk prediction
- MeSH terms
- Clinical Decision-Making, Machine Learning, Adult, Aged, Clinical Competence, Decision Support Techniques, Feasibility Studies, Female, Humans, Judgment, Male, Middle Aged, Pilot Projects, Postoperative Complications, Preoperative Care, Prospective Studies, ROC Curve, Surgical Procedures, Operative, Risk Assessment, Surgeons
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
