Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): A retrospective deep learning study
Open accessAbstract
BACKGROUND: Substance misuse is a heterogeneous and complex set of behavioural conditions that are highly prevalent in hospital settings and frequently co-occur. Few hospital-wide solutions exist to comprehensively and reliably identify these conditions to prioritise care and guide treatment. The aim of this study was to apply natural language processing (NLP) to clinical notes collected in the electronic health record (EHR) to accurately screen for substance misuse. METHODS: The model was trained and developed on a reference dataset derived from a hospital-wide programme at Rush University Medical Center (RUMC), Chicago, IL, USA, that used structured diagnostic interviews to manually screen admitted patients over 27 months (between Oct 1, 2017, and Dec 31, 2019; n=54 915). The Alcohol Use Disorder Identification Test and Drug Abuse Screening Tool served as reference standards. The first 24 h of notes in the EHR were mapped to standardised medical vocabulary and fed into single-label, multilabel, and multilabel with auxillary-task neural network models. Temporal validation of the model was done using data from the subsequent 12 months on a subset of RUMC patients (n=16 917). External validation was done using data from Loyola University Medical Center, Chicago, IL, USA between Jan 1, 2007, and Sept 30, 2017 (n=1991 adult patients). The primary outcome was discrimination for alcohol misuse, opioid misuse, or non-opioid drug misuse. Discrimination was assessed by the area under the receiver operating characteristic curve (AUROC). Calibration slope and intercept were measured with the unreliability index. Bias assessments were performed across demographic subgroups. FINDINGS: The model was trained on a cohort that had 3·5% misuse (n=1 921) with any type of substance. 220 (11%) of 1921 patients with substance misuse had more than one type of misuse. The multilabel convolutional neural network classifier had a mean AUROC of 0·97 (95% CI 0·96-0·98) during temporal validation for all types of substance misuse. The model was well calibrated and showed good face validity with model features containing explicit mentions of aberrant drug-taking behaviour. A false-negative rate of 0·18-0·19 and a false-positive rate of 0·03 between non-Hispanic Black and non-Hispanic White groups occurred. In external validation, the AUROCs for alcohol and opioid misuse were 0·88 (95% CI 0·86-0·90) and 0·94 (0·92-0·95), respectively. INTERPRETATION: We developed a novel and accurate approach to leveraging the first 24 h of EHR notes for screening multiple types of substance misuse. FUNDING: National Institute On Drug Abuse, National Institutes of Health.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the work by Afshar et al., clinical concepts were extracted from all available notes within each patient encounter using cTAKES.
“In the work by Afshar et al.50, clinical concepts, such as diseases, symptoms, anatomical sites, medications, and procedures, were extracted from all available notes within each patient encounter using the NLP engine, clinical Text And Knowledge Extraction System (cTAKES)86.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [50]
The Methods section (“Natural language processing of clinical notes”) says cTAKES extracted the listed clinical concepts from notes. It also says the study collected notes from only the first 24 hours of hospitalisation, so the source does not support extraction from all available notes across each patient’s entire encounter. No page number is provided in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThese extracted concepts were embedded and processed by a one-dimensional convolutional neural network (1D-CNN) for multi-task predictions of alcohol, opioid, and non-opioid misuse.
“These extracted concepts were then embedded and processed by a one-dimensional convolutional neural network (1D-CNN) for multi-task predictions of alcohol, opioid, and non-opioid misuse.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [50]
Methods (Natural language processing of clinical notes) says extracted clinical concepts were mapped to CUIs and could be passed through an embedding layer. Methods (Model architectures) and Results describe a multilabel convolutional neural network for jointly identifying alcohol, opioid, and non-opioid drug misuse. Our copy does not specify that the CNN was one-dimensional; it directs readers to the appendix (pp. 8–11) for architectural details. (LLM, conf=0.84)
Experimental — accuracy not yet measuredWe utilized the trained model from ref. 50 and reported the performance of its unhealthy alcohol use prediction head on our test set without any additional fine-tuning.
“We utilized the trained model from ref. 50 and reported the performance of its unhealthy alcohol use prediction head on our test set without any additional fine-tuning.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [50]
RNNs' proficiency in handling sequential information renders them indispensable in applications requiring deep understanding of temporal dynamics, such as natural language tasks and time-series analysis.
“This proficiency in handling sequential information renders them indispensable in applications that require a deep understanding of the temporal dynamics within data, such as natural language tasks and time-series analysis [155], [156].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [156]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267 · see citing works
Savova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967 · see citing works
Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
A comprehensive review of the psychometric properties of the Drug Abuse Screening Test
Yudko, E., Lozhkina, O., & Fouts, A. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.08.002 · Cited by 906 · see citing works
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
A review of approaches to identifying patient phenotype cohorts using electronic health records
Shivade, C., Raghavan, P., Fosler‐Lussier, E., Embí, P. J., Elhadad, N., Johnson, S. B., & Lai, A. M. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001935 · Cited by 533 · see citing works
US Preventive Services Task Force, Curry, S. J., Krist, A. H., Owens, D. K., Barry, M. J., Caughey, A. B., Davidson, K. W., Doubeni, C. A., Epling, J. W., Kemper, A. R., et al. (2018) · JAMA · doi:10.1001/jama.2018.16789 · Cited by 517 · see citing works
A tutorial on calibration measurements and calibration models for clinical prediction models
Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz228 · Cited by 486 · see citing works
Clinical text data in machine learning: systematic review
Spasic, I., & Nenadic, G. (2020) · JMIR Medical Informatics · doi:10.2196/17984 · Cited by 358 · see citing works
Velupillai, S. U., Suominen, H., Liakata, M., Roberts, A., Shah, A. D., Morley, K. I., Osborn, D., Hayes, J., Stewart, R. J., Downs, J. M., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.10.005 · Cited by 253 · see citing works
Lo-Ciganic, W.-H., Huang, J. L., Zhang, H. H., Weiss, J. C., Wu, Y., Kwoh, C. K., Donohue, J. M., Cochran, G., Gordon, A. J., Malone, D. C., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0968 · Cited by 227 · see citing works
Limitations of transformers on clinical text classification
Gao, S., Alawad, M., Young, M. T., Gounley, J., Schaefferkoetter, N., Yoon, H. J., Wu, X.-C., Durbin, E. B., Doherty, J., Stroup, A., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3062322 · Cited by 157 · see citing works
Neumann, T., Neuner, B., Gentilello, L. M., Weiß-Gerlach, E., Mentz, H., Rettig, J. S., Schröder, T., Wauer, H., Müller, C. A., Schütz, M., et al. (2004) · Alcoholism Clinical and Experimental Research · doi:10.1097/01.alc.0000145696.58084.08 · Cited by 117 · see citing works
MACKENZIE, D. M., Langa, A., & Brown, T. M. (1996) · Alcohol and Alcoholism · doi:10.1093/oxfordjournals.alcalc.a008195 · Cited by 99 · see citing works
Afshar, M., Phillips, A., Karnik, N., Mueller, J., To, D., Gonzalez, R., Price, R., Cooper, R., Joyce, C., & Dligach, D. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy166 · Cited by 94 · see citing works
Self-citationThompson, H. M., Sharma, B., Bhalla, S., Boley, R., McCluskey, C., Dligach, D., Churpek, M. M., Karnik, N. S., & Afshar, M. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab148 · Cited by 78 · see citing works
Self-citationChan, B., Freeman, M., Ayers, C., Korthuis, P. T., Paynter, R., Kondo, K., & Kansagara, D. (2020) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2020.108193 · Cited by 56 · see citing works
Alcohol Withdrawal Rates in Hospitalized Patients During the COVID-19 Pandemic
Sharma, R. A., Subedi, K., Gbadebo, B. M., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.0422 · Cited by 53 · see citing works
Thompson, H. M., Faig, W. G., VanKim, N. A., Sharma, B., Afshar, M., & Karnik, N. S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0239761 · Cited by 40 · see citing works
Self-citationIdentifying risk of opioid use disorder for patients taking opioid medications with deep learning
Dong, X., Deng, J., Rashidian, S., Abell-Hart, K., Hou, W., Rosenthal, R. N., Saltz, M. M., Saltz, J. H., & Wang, F. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab043 · Cited by 40 · see citing works
Sharma, B., Dligach, D., Swope, K., Salisbury‐Afshar, E., Karnik, N. S., Joyce, C., & Afshar, M. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1099-y · Cited by 39 · see citing works
Self-citationOpioid-related overdose deaths by race and neighborhood economic hardship in Chicago
Rushovich, T., Arwady, M. A., Salisbury‐Afshar, E., et al. (2020) · Journal of Ethnicity in Substance Abuse · doi:10.1080/15332640.2019.1704335 · Cited by 30 · see citing works
Verbakel, J. Y. J., Steyerberg, E. W., Uno, H., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.013 · Cited by 5 · see citing works
Aequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577
Flagged first, then by citations
Research works printed but not linked: 3
- [3]Owens PL, Fingar KR, McDermott KW, Inpatient stays involving mental and substance use disorders, 2016. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb249-Mental-Substance-Use-Disorder-Hospital-Stays-2016.jsp (accessed Aug 15, 2021). 31063293
- [4]Committee on the Recommended Social and Behavioral Domains and Measures for Electronic Health Records, Board on Population Health and Public Health Practice, Institute of Medicine. Capturing social and behavioral domains and measures in electronic health records: phase 2. Washington, DC: National Academies Press, 2015. 25590118
- [13]Freeman WJ, Weiss AJ, Heslin KC. Overview of US hospital stays in 2016: variation by geographic region. 2018. www.hcup-us.ahrq.gov/reports/statbriefs/sb246-Geographic-Variation-Hospital-Stays.pdf (accessed Sept 1, 2021). 30720972
Cited by
Sorted by citations
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“Additionally, the development of AI literacy programs will be crucial in helping society understand and responsibly interact with multimodal AI technologies [113], [135].”
BackgroundMoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
Gao, J., Rahman, M., Caskey, J., Oguss, M., O’Rourke, A., Brown, R., Stey, A., Mayampurath, A., Churpek, M. M., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02219-4 · Cited by 2 · see citing works
“50 andreportedtheperformanceofitsunhealthyalcoholuse prediction head on our test set without any additional fi ne-tuning.”
Self-citation
- Authors
- Afshar, M., Sharma, B., Dligach, D., Oguss, M., Brown, R., Chhabra, N., Thompson, H. M., Markossian, T., Joyce, C., Churpek, M. M., & Karnik, N. S.
- Affiliations
- University of Wisconsin–Madison (US)
- Rush University Medical Center (US)
- Loyola University Chicago (US)
- John H. Stroger, Jr. Hospital of Cook County (US)
- Corresponding author
- Afshar, Majid
- ORCID
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2022-06
- Volume
- 4
- Issue
- 6
- Pages
- e426-e435
- ISSN
- 2589-7500 (ISSN-L)
- PMC
- PMC9159760
- OpenAlex
- W4281643675
- Cited by
- 55
- References
- 26
- Citations by year
- 2026: 11 · 2025: 10 · 2024: 15 · 2023: 13 · 2022: 6
- FWCI
- 3.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- The Lancet Digital Health (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by-nc-nd)
- Loyola eCommons (Loyola University of Chicago) (submittedVersion, cc-by-nc-nd)
- Figshare (submittedVersion, cc-by-nc-nd)
- University of Illinois Chicago
- Funders
- National Institutes of Health (K23-AA024503, UL1-TR002389, R01-DA051464)
- National Institute on Drug Abuse
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Epidemiology › Substance Abuse Treatment and Outcomes
- Keywords
- hospitalization, electronic health records, natural language processing, deep learning, multi-label classification, convolutional neural networks, alcohol abuse, opioid misuse, referral to treatment, algorithmic bias, temporal validation, external validation, clinical notes, DAST
- MeSH terms
- Deep Learning, Adult, Alcoholism, Artificial Intelligence, Humans, Opioid-Related Disorders, Referral and Consultation, Retrospective Studies, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
