Illusory generalizability of clinical prediction models
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model’s success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further, such models based on data from specific populations may not be generalizable and thus may fail to properly assign the optimal treatment [26].”
From Personalized machine learning guided intervention for optimizing lifestyle behaviors in depression: a pilot study (2026) · cites this paper as [26]
In the citing paper
“AI agents show significant potential in the realm of personalized treatment by leveraging a variety of patient-specific data, including genetic information, medical history, lifestyle factors, and treatment responses.77,88,89,90,91,92”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
In the citing paper
“AI-driven real-time patient monitoring systems are revolutionizing clinical care by enabling continuous, data-driven tracking of patient health.90,104,105”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The Positive and Negative Syndrome Scale (PANSS) for schizophrenia.
Kay, S. R., Fiszbein, A., & Opler, L. A. (1987) · Schizophrenia Bulletin · doi:10.1093/schbul/13.2.261 · Cited by 21,378
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
(1976) · Medical Entomology and Zoology · Cited by 8,899
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
A New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202
The clinical global impressions scale: applying a research tool in clinical practice.
Busner, J., & Targum, S. D. (2007) · PubMed Central · Cited by 2,760
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816
Research works printed but not linked: 3
- 10.1176/appi.ajp.2017.16121358 · DOI · Paper page
- 10.1093/schbul/sbv167 · DOI · Paper page
- 10.1001/jamapsychiatry.2017.0025 · DOI · Paper page
Cited by
Torous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
“This reinforces the potential of roles such as that of digital navigators 11,13 in im proving engagement rates and therapeutic outcomes 198 .”
A review of model evaluation metrics for machine learning in genetics and genomics
Miller, C., Portlock, T., Nyaga, D. M., & O'Sullivan, J. M. (2024) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2024.1457619 · Cited by 199
Zantvoort, K., Nacke, B., Görlich, D., Hornstein, S., Jacobi, C., & Funk, B. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01360-w · Cited by 127
“Further, especially for complex features or ones with unknown or low information value, having a reasonablysizedtestsetisindispensable 18,46 .”
InfluentialFügener, A., Walzner, D. D., & Gupta, A. (2025) · Management Science · doi:10.1287/mnsc.2024.05684 · Cited by 161
Practical AI application in psychiatry: historical review and future directions
Sun, J., Lu, T., Shao, X., Han, Y., Xia, Y., Zheng, Y., Wang, Y., Li, X., Ravindran, A. V., Fan, L., et al. (2025) · Molecular Psychiatry · doi:10.1038/s41380-025-03072-3 · Cited by 100
“For example, Adam et al. analyzed the performance of AI models across multiple independent clinical trials that involved antipsychotic medication for schizophrenia [58].”
Botha, N. N., Segbedzi, C. E., Dumahasi, V. K., Maneen, S., Kodom, R. V., Tsedze, I. S., Akoto, L. A., Atsu, F. S., Lasim, O. U., & Ansah, E. W. (2024) · Archives of Public Health · doi:10.1186/s13690-024-01414-1 · Cited by 93
Hamidi, H., Şenbabaoğlu, Y., Beig, N. G., Roels, J., Manuel, C., Guan, X., Koeppen, H., Assaf, Z. J. F., Nabet, B. Y., Waddell, A., et al. (2024) · Cancer Cell · doi:10.1016/j.ccell.2024.10.016 · Cited by 63
Evaluating generalizability of artificial intelligence models for molecular datasets
Ektefaie, Y., Shen, A., Bykova, D., Marin, M. G., Zitnik, M., & Farhat, M. (2024) · bioRxiv · doi:10.1038/s42256-024-00931-6 · Cited by 80
“Despite the potential of these models, their generalizability is unknown as they have rarely been tested prospectively on non-overlapping datasets 55,56 .”
Starke, G., Gille, F., Termine, A., Aquino, Y. S. J., Chavarriaga, R., Ferrario, A., Hastings, J., Jongsma, K. R., Kellmeyer, P., Kulynych, B., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/56306 · Cited by 67
“Questionable improvements in real-life health care settings [15], a lack of regulatory frameworks [16], and unresolved questions of reimbursement [17] further complicate adoption of AI in health care.”
Biological markers and psychosocial factors predict chronic pain conditions
Fillingim, M., Tanguay-Sabourin, C., Parisien, M., Zare, A., Guglietti, G. V., Norman, J., Petre, B., Bortsov, A. V., Ware, M. A., Pérez, J., et al. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02156-y · Cited by 69
“This is especially important given that accurate predictions of clinical outcomes within trials in which models were developed often showed no better than chance level when tested in out-of-sample patients 19 .”
Generalizable and explainable deep learning for medical image computing: An overview
Chaddad, A., Hu, Y., Wu, Y., Wen, B., & Kateb, R. (2024) · Current Opinion in Biomedical Engineering · doi:10.1016/j.cobme.2024.100567 · Cited by 65
“Understanding the causes of these limitations can help improve modeling strategies and establish standards for model benchmarking before clinical use, ensuring robustness and generalizability [17].”
A scoping review of artificial intelligence applications in clinical trial risk assessment
Teodoro, D., Naderi, N., Yazdani, A., Zhang, B., & Bornet, A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01886-7 · Cited by 70
“…at least three phases ( n = 27 vs. n = 18 for phases I, II, and non-phase-speci fi caltogether),whichcouldleadtobettergeneralisabilityofsuch approaches as compared to the safety and ef fi cacy AI-based risk assessment 42,105 , which are either non-phase-speci fi c or focus on a single phase(Fig.4b).”
Bringing Imaging Biomarkers Into Clinical Reality in Psychiatry
Etkin, A., & Mathalon, D. H. (2024) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2024.2553 · Cited by 53
Berisha, V., & Liss, J. M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01199-1 · Cited by 41
“Unfortunately, supervisedmodelsbuiltonsmaller-scalecorporaoftenexhibitover-optimisticperformanceincontrolledenvironments 35 and fail to generalize in out-of-sample deployments 36,37 .”
BackgroundA foundational architecture for AI agents in healthcare
Liu, F., Niu, Y., Zhang, Q., Wang, K., Dong, Z., Wong, I. N., Cheng, L., Li, T., Duan, L., Li, K., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102374 · Cited by 41
“90,104,105 Through integration of wearable devices, biosensors, and remote monitoring tools, AI agents can collect and analyze a range of physiological data, including heart rate, blood pressure, and blood glucose levels.”
A generalizable data-driven model of atrophy heterogeneity and progression in memory clinic settings
Baumeister, H., Vogel, J. W., Insel, P. S., Kleineidam, L., Wolfsgruber, S., Stark, M., Gellersen, H. M., Yakupov, R., Schmid, M. C., Lüsebrink, F., et al. (2024) · Brain · doi:10.1093/brain/awae118 · Cited by 34
“27 demonstrated in various examples of disease progression models that, even when performance in internal validation procedures such as k -fold cross-validation was satisfactory, models performed poorly, at around chance level, in external testing data.”
BackgroundMethodInfluentialWiebe, A., Selaskowski, B., Paskin, M., Asché, L., Pakos, J., Aslan, B., Lux, S., Philipsen, A., & Braun, N. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-03217-y · Cited by 41
Generalizability of clinical prediction models in mental health
Richter, M., Emden, D., Leenings, R., Winter, N. R., Mikolajczyk, R., Massag, J., Zwiky, E., Borgers, T., Redlich, R., Koutsouleris, N., et al. (2025) · Molecular Psychiatry · doi:10.1038/s41380-025-02950-0 · Cited by 44
Self-citationPractical challenges for precision medicine
Petzschner, F. H. (2024) · Science · doi:10.1126/science.adm9218 · Cited by 30
Artificial intelligence in food and nutrition evidence: The challenges and opportunities
Bailey, R. L., MacFarlane, A. J., Field, M. S., Tagkopoulos, I., Baranzini, S. E., Edwards, K. M., Rose, C. J., Schork, N. J., Singhal, A., Wallace, B., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae461 · Cited by 36
“…AI prediction models of antipsychotic medications trained on RCT data failed to predict patient outcomes when applied to out-of-sample patients, indicating that treatment outcomes are not generalizable for schizophrenia, emphasizing the strong modifying effects of an individual’s contexts (41).”
Cuijpers, P. J., Harrer, M., & Furukawa, T. A. (2026) · World Psychiatry · doi:10.1002/wps.70002 · Cited by 36
“Psy chological interventions also depend heavily on contextual and setting specific factors, which further limits the generalizability of ML based prediction models 210 .”
Goerigk, S. A., Elsaeßer, M., Reinhard, M. A., Kriston, L., Härter, M. C., Hautzinger, M., Klein, J. P., McCullough, J. P., Schramm, E., & Padberg, F. (2024) · The Lancet Psychiatry · doi:10.1016/s2215-0366(24)00209-8 · Cited by 35
Artificial intelligence for the study of human ageing: a systematic literature review
Bernal, M. C., Batista, E., Martínez-Ballesté, A., & Solanas, A. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05817-z · Cited by 32
Ramakrishnaiah, Y., Maćešić, N., Webb, G. I., Peleg, A. Y., & Tyagi, S. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105816 · Cited by 39
Sweet, L., Athanasiadis, I. N., van Bree, R., Castellano, A., Martre, P., Paudel, D., Ruane, A. C., & Zscheischler, J. (2025) · One Earth · doi:10.1016/j.oneear.2025.101233 · Cited by 40
Morgan, G. J., Nau, A. N., Wong, S., Spencer, B., Shen, Y., Hua, A., Bullard, M. J., Sanchorawala, V., & Prokaeva, T. B. (2024) · Amyloid · doi:10.1080/13506129.2024.2434899 · Cited by 30
Zang, C., Hou, Y., Lyu, D., Jin, J., Sacco, S. J., Chen, K., Aseltine, R. H., & Wang, F. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-03034-3 · Cited by 30
“For example, the discussion of transportability in ML-based prediction models often assumes that complex prediction models with many variables outperform simpler models [19].”
BackgroundMethodQuality over quantity: powering neuroimaging samples in psychiatry
Makowski, C., Nichols, T. E., & Dale, A. M. (2024) · Neuropsychopharmacology · doi:10.1038/s41386-024-01893-4 · Cited by 25
Hartmann, S., Dwyer, D., Cavve, B. S., Byrne, E. M., Scott, I., Gao, C. X., Wannan, C., Yuen, H. P., Hartmann, J., Lin, A. P., et al. (2024) · World Psychiatry · doi:10.1002/wps.21240 · Cited by 30
“This finding highlights that, in addition to the evaluation of discrimination performance for calculators across cohorts and clinical services 72 , clinicians need to consider the age of a psychosis prediction model when interpreting its output.”
Personalized Machine Learning-Based Prediction of Wellbeing and Empathy in Healthcare Professionals
Nan, J., Herbert, M. S., Purpura, S., Henneken, A. N., Ramanathan, D., & Mishra, J. (2024) · Sensors · doi:10.3390/s24082640 · Cited by 22
Strodthoff, N., Lopez Alcaraz, J. M., & Haverkamp, W. (2024) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae039 · Cited by 22
“We acknowledge the importance of an external validation 24 and therefore validate the performance of our model on the CODE test set 15 across a set of six different cardiac conditions.”
MethodTransforming mental health research and care through artificial intelligence
Opel, N., & Breakspear, M. J. (2026) · Science · doi:10.1126/science.adz9193 · Cited by 37
Artificial intelligence and psychotherapy: A counterpoint
Richards, D. (2024) · Counselling and Psychotherapy Research · doi:10.1002/capr.12758 · Cited by 50
The neuroscience of mental illness: Building toward the future
Gordon, J. A., Dzirasa, K., & Petzschner, F. H. (2024) · Cell · doi:10.1016/j.cell.2024.09.028 · Cited by 25
Zhukovsky, P., Trivedi, M. H., Weissman, M. M., Parsey, R. V., Kennedy, S. H., & Pizzagalli, D. A. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.1310 · Cited by 23
“Given the moderate size of the imaging datasets analyzed here, and following recent studies of treatment outcome prediction 18 , we use elastic net logistic models aiming to maintain a higher feature-to-observation ratio 19 .”
AI for mental health: clinician expectations and priorities in computational psychiatry
Fischer, L., Mann, P. A., Nguyen, M.-H. H., Becker, S., Khodadadi, S. H. A., Schulz, A., Thanarajah, S. E., Repple, J., Hahn, T., Reif, A., et al. (2025) · BMC Psychiatry · doi:10.1186/s12888-025-06957-3 · Cited by 26
“Major studies even question the feasibility of accurate treatment predictions, citing significant limitations in current methodologies [23].”
Large Language Models for Psychological Assessment: A Comprehensive Overview
Brickman, J., Gupta, M., & Oltmanns, J. R. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251343582 · Cited by 48
Corponi, F., Li, B. M., Anmella, G., Mas, A., Pacchiarotti, I., Valentí, M., Grande, I., Benabarre, A., Garriga, M., Vieta, E., et al. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-02876-1 · Cited by 19
“Future work (i) The decline in performance over future time points stands out as the main challenge towards real-world implementations and suggests that the model struggles to adapt to changes in background (latent) variables, e.g., changes in activity patterns.”
Etkin, A., Powell, J. J., & Savitz, A. J. (2024) · Neuropsychopharmacology · doi:10.1038/s41386-024-01970-8 · Cited by 24
“It is also likewise well-documented that the kind of machine learning analyses increasingly used to develop candidate biomarker signals can over fi t on the data on which they are trained and not generalize to other samples [48, 49].”
BackgroundDeciphering sepsis: transforming diagnosis and treatment through systems immunology
Hancock, R. E. W., An, A., dos Santos, C. C., & Lee, A. H. (2025) · Frontiers in Science · doi:10.3389/fsci.2024.1469417 · Cited by 27
“Importantly, the aforementioned schizophrenia study (125) was limited in that it did not consider multiomics approaches.”
Ethnokinesiology: towards a neuromechanical understanding of cultural differences in movement
Ting, L. H., Gick, B., Kesar, T. M., & Xu, J. (2024) · Philosophical Transactions B · doi:10.1098/rstb.2023.0485 · Cited by 17
Fortea, L., Ortuño, M., Prisco, M. D., Oliva, V., Albajes‐Eizagirre, A., Fortea, A., Madero, S., Solanes, A., Vilajosana, E., Yao, Y., et al. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2024.10.020 · Cited by 26
“We fully acknowledge the low accuracy of MRI-based machine learning tools (45,46), which is expected considering that the diagnostic labels are a pragmatic but conventional classification.”
From multi-omics to predictive biomarker: AI in tumor microenvironment
Hai, L., Jiang, Z., Zhang, H., & Sun, Y. (2024) · Frontiers in Immunology · doi:10.3389/fimmu.2024.1514977 · Cited by 25
Global Health in the Age of AI: Charting a Course for Ethical Implementation and Societal Benefit
Morley, J., Hine, E., Roberts, H., Sirbu, R. A., Ashrafian, H., Blease, C., Boyd, M., Chen, J. L., Filho, A. D. P. C., Coiera, E. W., et al. (2025) · Minds and Machines · doi:10.1007/s11023-025-09730-3 · Cited by 24
“The resulting algorithms often performed poorly when deployed in the very communities whose data trained them (Chekroud et al., 2024), creating a troubling cycle where data flows from under-resourced communities to technology companies without returning to data contributors.”
Psychiatry in the age of AI: transforming theory, practice, and medical education
Zhang, X., & Zhang, X. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1660448 · Cited by 28
Gerster, M., Waterstraat, G., Binns, T. S., Darcy, N., Wiest, C., Köhler, R. M., Vanhoecke, J., West, T. O., Sure, M., Todorov, D., et al. (2025) · EBioMedicine · doi:10.1016/j.ebiom.2025.105988 · Cited by 25
“60 While previous studies linked STN beta power to motor symptom severity, inconsistencies across reports raised the need to study robustness and replicability.”
A simple but tough-to-beat baseline for fMRI time-series classification
Popov, P., Mahmood, U., Fu, Z., Yang, C., Calhoun, V. D., & Plis, S. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120909 · Cited by 28
“Interestingly, the introduction of the averaging step to LSTM and Transformer models improved their performance significantly, as we can see from the meanLSTM and meanTransformer results.”
Song, J., García-Balduz, J., Yang, T., Skirtach, A. G., & Parakhonskiy, B. V. (2025) · Journal of Colloid and Interface Science · doi:10.1016/j.jcis.2025.137495 · Cited by 17
Guerreiro, J., Garriga, R., Bagén, T. L., Sharma, B., Karnik, N. S., & Matic, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01203-8 · Cited by 19
“Moreover, clinical prediction models have been increasingly criticized for their illusory generalizability and for underperforming when applied out-of-sample 29 .”
Mkrtchian, A., Qiu, Z., Abir, Y., Erdmann, T., Dercon, Q., Sedlinská, T., Browning, M. D., Costello, H., & Huys, Q. J. M. (2025) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2025.0839 · Cited by 19
“This contrasts with symptom markers, which do not differentiate psychiatric medications 47,48 .”
Silva-Sousa, T., Usuda, J. N., Al-Arawe, N., Hinterseher, I., Catar, R. A., Luecht, C., Vallecillo-García, P., Riesner, K., Hackel, A. M., Schimke, L. F., et al. (2025) · Stem Cells Translational Medicine · doi:10.1093/stcltm/szaf037 · Cited by 19
Revisiting the use of adverse childhood experience screening in healthcare settings
Danese, A., Asmussen, K., MacLeod, J. G., Meehan, A. J., Sears, J., Slopen, N. B., Smith, P. A., & Sweeney, A. (2024) · Nature Reviews Psychology · doi:10.1038/s44159-024-00362-5 · Cited by 18
Libedinsky, I., Helwegen, K., Boonstra, J. T., Simón, L. G., Gruber, M., Repple, J., Kircher, T. T. J., Dannlowski, U., & van den Heuvel, M. P. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2024.10.007 · Cited by 18
Advances in personalization of psychological interventions
Lutz, W., Schwartz, B., Vehlen, A., Eberhardt, S. T., & Delgadillo, J. (2025) · World Psychiatry · doi:10.1002/wps.21342 · Cited by 14
Fu, B., Chen, J., Liu, X., Li, J., Feng, X., Wang, J., Li, Y., Zhang, Y., Ma, J., Wang, Y., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-68579-x · Cited by 15
“Two external validation cohorts ensured a robust evaluation of our models, circumventing the “ illusory generalizability ” common in machine learning based medical models 48 .”
Schurz, H., Solander, K., Åström, D., Cossío, F., Choi, T. Y., Dustler, M., Lundström, C. F., Gustafsson, H., Zackrisson, S., & Strand, F. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01623-0 · Cited by 14
Vos, G., van Eijk, L., Sarnyai, Z., & Azghadi, M. R. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108899 · Cited by 20
Ma, X., Pan, W., & Yu, X. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07706-6 · Cited by 17
“…items: while manual tests utilize progressive descriptions of real patient cases to construct dynamic distractors (such as temporally incorrect yet plausible options), AI-generated distractors tend to rely on superficial semantic differences, lacking deeper clinical reasoning simulations [19].”
Lenton‐Brym, A. P., Collins, A., Lane, J. E. M., Busso, C., Ouyang, J., Fitzpatrick, S., Kuo, J. R., & Monson, C. M. (2024) · British Journal of Clinical Psychology · doi:10.1111/bjc.12468 · Cited by 19
“Second, models that are developed on accessible datasets may demonstrate impressive predictive capacity within the datasets on which they are trained, but more modest ability when extrapolated to novel data and situations (Chekroud et al., 2024).”
BackgroundJahanshad, N., Lenzini, P., & Bijsterbosch, J. (2025) · Neuropsychopharmacology · doi:10.1038/s41386-024-01938-8 · Cited by 12
“Importantly, out-of-sample prediction using a truly independent dataset is important in addition to partitioning one dataset through cross validation to assess realistic model performance on prospective data (e.g., newly acquired patients) [61].”
BackgroundCalderon, A., Baik, S. Y., Ng, M. H. S., Fitzsimmons‐Craft, E. E., Eisenberg, D., Wilfley, D. E., Taylor, C. B., & Newman, M. G. (2024) · BMC Psychiatry · doi:10.1186/s12888-024-06074-7 · Cited by 18
“Unlike unregularized approaches (e.g., linear regression), supervised machine learning meth-ods allow the opportunity to aggregate disparate small variable effects to inform clinical outcomes while also accounting for complex, interactive, or non-linear effects [16].”
BackgroundMuñoz-Duque, L. A., Acuña, R. A. E., Munn, T., Muñoz, L. C. B., Johnson, S. C., Gilbert, M. R., & Conroy, A. H. (2024) · Social Science & Medicine · doi:10.1016/j.socscimed.2024.117331 · Cited by 11
Adkinson, B. D., Rosenblatt, M., Dadashkarimi, J., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2024.101464 · Cited by 16
Srivilaithon, W., & Thanasarnpaiboon, P. (2025) · BMC Emergency Medicine · doi:10.1186/s12873-025-01185-0 · Cited by 15
“By identifying subtle patterns, feature interactions, and high-dimensional dependencies, ML models provide automated feature selection and can adapt to heterogeneous patient populations without strict parametric assumptions [15, 26, 27].”
IUPHAR review: Moving beyond dopamine-based therapeutic strategies for schizophrenia
Moran, P. M., & Granger, K. T. (2025) · Pharmacological Research · doi:10.1016/j.phrs.2025.107727 · Cited by 13
Zhao, X., Sun, B., Chu, X., Ye, J., & Chu, X. (2025) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2025.109491 · Cited by 11
The Application of AI to Ecological Momentary Assessment Data in Suicide Research: Systematic Review
Melia, R., Schafer, K. M., Rogers, M. L., Wilson, E., & Joiner, T. E. (2025) · Journal of Medical Internet Research · doi:10.2196/63192 · Cited by 14
“Chekroud et al [57] noted that this generally results in predictive performance measures that are lower but allows a more realistic indication of the potential of the model to inform clinical practice [61].”
Application of Explainable Artificial Intelligence in Autism Spectrum Disorder Detection
Viswan, V., Shaffi, N., Sadiq, M. A. K., Sirasanagandla, S. R., Aradhya, V. N. M., Kaiser, M. S., Wang, S., & Mahmud, M. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10462-w · Cited by 16
Hong, M., Kang, R.-R., Yang, J. H., Rhee, S. J., Lee, H., Kim, Y.-G., Lee, K. Y., Kim, H., Lee, Y. S., Youn, T., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/65994 · Cited by 13
Prediction of Treatment Outcome in Bipolar Disorder: When Can We Expect Clinical Relevance?
Scott, K., Khayachi, A., Alda, M., & Nunes, A. (2025) · Biological Psychiatry · doi:10.1016/j.biopsych.2025.02.005 · Cited by 10
The Promise of Investigating Neural Variability in Psychiatric Disorders
Tsikonofilos, K., Kumar, A., Ampatzis, K., Garrett, D. D., & Månsson, K. N. T. (2025) · Biological Psychiatry · doi:10.1016/j.biopsych.2025.02.004 · Cited by 12
“It has been argued that clinically useful predictive models require large sample sizes (3) and that neuroimaging-based features do not improve their predictive capacity (2).”
Hazenbroek, M., Pengel, L. H. M., Sassen, S. D. T., Massey, E. K., Reinders, M. E. J., de Winter, B. C. M., & Hesselink, D. A. (2024) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.16121 · Cited by 9
“30,34 Collaboration between experts from the clinical and technological fields is necessary to determine the role of AI in precision dosing.”
Bertin, D., Bongrand, P., & Bardin, N. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25063270 · Cited by 10
“Indeed, patients’ demographic properties might exhibit temporal or local [42,43] changes, thereby hampering model validity through a so-called dataset shift.”
BackgroundNeural activity during inhibitory control predicts suicidal ideation with machine learning
Nan, J., Grennan, G., Ravichandran, S., Ramanathan, D., & Mishra, J. (2024) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-024-00012-x · Cited by 8
“In a recent study [85], ML models aimed at predicting treatment success for schizophrenia found that models built on one dataset (including both single-trial and multi-trial data sets) did not generalize to other independent studies.”
ResultBurghoorn, D. G., Booij, S. H., Schoevers, R. A., & Riese, H. (2024) · BJPsych Open · doi:10.1192/bjo.2024.789 · Cited by 10
The diagnostic potential of resting state functional MRI: Statistical concerns
Doubovikov, E. D., & Aksenov, D. P. (2025) · NeuroImage · doi:10.1016/j.neuroimage.2025.121334 · Cited by 11
Sarhan, O. M., Gebril, M. I., & Elsegaie, D. (2025) · Journal of Pharmacy and Pharmacology · doi:10.1093/jpp/rgaf113 · Cited by 8
Wolpe, N., Perrottelli, A., Giuliani, L., Yang, Z., Rekhi, G., Jones, P. B., Bernardo, M., García‐Portilla, M. P., Kaiser, S., Robert, G. H., et al. (2025) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2024.12.016 · Cited by 9
“Although some found high correlations between these PANSS factors and BNSS MAP and EXP (Kaliuzhna et al., 2020), generalisability across heterogeneous samples and cultures, which is likely to affect predictive performance (Chekroud et al., 2024) and hence external validity has been limited.”
Genomic and Developmental Models to Predict Cognitive and Adaptive Outcomes in Autistic Children
Bourque, V.-R., Schmilovich, Z., Huguet, G., England, J., Okewole, A., Poulain, C., Renne, T., Jean‐Louis, M., Saci, Z., Zhang, X., et al. (2025) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2025.0205 · Cited by 9
Four questions to predict cognitive decline in de novo Parkinson’s disease
Hlavnička, J., Mana, J., Bezdíček, O., Čihák, M., Havlík, F., Skrabal, D., Bartošová, T., Šonka, K., Růžička, E., & Dušek, P. (2025) · npj Parkinson s Disease · doi:10.1038/s41531-025-00958-5 · Cited by 8
“Sincethemodelunderwentdevelopmentandcross-validationtesting onthe PPMIdatasetandblindvalidationonindependentBIO-PDdata,weassume suf fi cient generalizability of our fi ndings 52 .”
Advancements and challenges in using AI for biomarker detection in early Alzheimer’s disease
Beheshti, I., Albensi, B. C., Freitas, A. A., & Ghafourian, T. (2025) · Drug Discovery Today · doi:10.1016/j.drudis.2025.104415 · Cited by 8
Designing Clinically Useful AI: A Blueprint for Impact
Parsa, S., Keyes, T., Dash, D., Char, D. S., Mello, M. M., Callahan, A., Smith, M. A., Lee, S., Wang, T., Salisbury, H., et al. (2026) · NEJM AI · doi:10.1056/aip2500808 · Cited by 9
A cautionary tale for AI and machine learning in psychiatry
Chen, Z. S., Schultebraucks, K., & Wu, W. (2026) · Translational Psychiatry · doi:10.1038/s41398-026-03930-w · Cited by 11
“Surprisingly, the model accuracy signi fi cantly declined to a chance level when applied to independent clinical trials, suggesting that there is a critical need for validating clinical prediction models across diverse clinical samples [20].”
Why is it so difficult to implement precision psychiatry into clinical care?
Falkai, P., & Koutsouleris, N. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.100952 · Cited by 10
Self-citationFields, C. T., Black, C. G., Thind, J. K., Jegede, O. O., Aksen, D. E., Rosenblatt, M., Assari, S., Bellamy, C. D., Anderson, E., Holmes, A. J., et al. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1492736 · Cited by 12
Personalised and precision mental health in eating disorders: why routine outcome measurement is key
Austin, A., & Allen, K. L. (2025) · Journal of Eating Disorders · doi:10.1186/s40337-025-01290-2 · Cited by 7
“Additionally, generalizability of precision models is a significant statistical challenge, as many models are developed using data from specific settings, with reduced performance when applied to other settings [12].”
Bull, C. F., Kisely, S. R., Betts, K. S., & Hu, Y. (2025) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2025.107630 · Cited by 13
Lohner, V., Badhwar, A., Detcheverry, F. E., García, C. L., Gellersen, H. M., Khodakarami, Z., Lattmann, R., Li, R., Low, A., Mazo, C., et al. (2025) · Alzheimer s Research & Therapy · doi:10.1186/s13195-025-01815-6 · Cited by 11
“A recent study by Chekround and colleagues found that, even with adequately cross-validated algorithms, the performance of these ML models was consistently lower in external datasets [128].”
Parola, A., Jessen, E. T., Rybner, A. S., Mortensen, M. D., Larsen, S. N., Simonsen, A., Lin, J. M., Zhou, Y., Wang, H., Koelkebeck, K., et al. (2025) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaf124 · Cited by 12
Overview of Challenges in Brain-Based Predictive Modeling: Toward Meaningful Predictive Insights
Komeyer, V., Nieto, N., Eickhoff, S. B., Raimondo, F., & Patil, K. R. (2025) · Biological Psychiatry · doi:10.1016/j.biopsych.2025.09.003 · Cited by 13
SERS Meets Artificial Intelligence: A New Frontier in Cancer Diagnosis and Prognosis
Huda, N. U., Bari, R. Z. A., Javed, M., Kiani, M. N., & Jin, Y. (2026) · Analytical Chemistry · doi:10.1021/acs.analchem.6c00874 · Cited by 7
Creating diverse and inclusive scientific practices for research datasets and dissemination
Kam, J. W. Y., Badhwar, A., Borghesani, V., Lee, K., Noble, S., Raamana, P. R., Ratnanather, J. T., Tan, D. G. H., Oestreich, L., Lee, H. W., et al. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00216 · Cited by 9
“…disparity of available data for certain groups may lead to discrepancies between training and test datasets, which may result in biased or unsatisfactory outcomes due to dataset shifts that are limited in its generalizability ( Chekroud et al., 2024; Park et al., 2021; Qui ñ onero-Candela, 2009).”
BackgroundMyers, C. E., Dave, C. V., Chesin, M. S., Marx, B. P., Hill, L. M. S., Reddy, V., Miller, R. B., King, A. R., & Interian, A. (2024) · Behaviour Research and Therapy · doi:10.1016/j.brat.2024.104637 · Cited by 8
Slot, M. I. E., Urquijo-Castro, M. F., Rossum, I. W.-V., van Hell, H. H., Dwyer, D., Dazzan, P., Maat, A., de Haan, L., Crespo‐Facorro, B., Glenthoj, B. Y., et al. (2024) · Schizophrenia · doi:10.1038/s41537-024-00505-w · Cited by 6
“In line with recent observations 28 , our results indicate that models based on single data sets provide limited insight into performance in future patients; hence, external validation of prediction models in an unrelated and carefully controlled clinical environment is essential in assessing the…”
BackgroundResultHornstein, S., Lueken, U., Wundrack, R., & Hilbert, K. (2024) · JMIR AI · doi:10.2196/63701 · Cited by 7
Rösler, W., Roiss, M., & Widmer, C. (2024) · healthbook TIMES Oncology Hematology · doi:10.36000/hbt.oh.2024.20.146 · Cited by 7
Mangalam, M., Seckler, H., & Kelty‐Stephen, D. G. (2024) · Physical Review Research · doi:10.1103/physrevresearch.6.033276 · Cited by 8
Piccin, J., Buchweitz, C., Manfro, P. H., Pereira, R. B., Cunegatto, F. R., Souza, L., Viduani, A. C., Caye, A., Kohrt, B. A., Mondelli, V., et al. (2025) · BMJ Mental Health · doi:10.1136/bmjment-2024-301207 · Cited by 9
“5 7 Low, evenly distributed attrition rates ensured balanced longitudinal data.”
Precision oncology in the age of AI: lessons from AI-driven drug discovery and clinical translation
Yoo, W. (2026) · BJC Reports · doi:10.1038/s44276-026-00221-1 · Cited by 9
“Without prospective, multi-center validation, candidates optimized within development datasets may fail to yield consistent ef fi cacy or safety outcomes across broader clinical environments [27, 28].”
Rosenblatt, M., Foster, M. L., Adkinson, B. D., Tejavibulya, L., Khaitova, M., Ye, J., Sun, H., Rodriguez, R. X., Camp, C., Chinta, A., et al. (2026) · Nature Methods · doi:10.1038/s41592-026-03115-9 · Cited by 11
Metadata
- Authors
- Chekroud, A. M., Hawrilenko, M., Loho, H., Bondar, J., Gueorguieva, R., Hasan, A., Kambeitz, J., Corlett, P. R., Koutsouleris, N., Krumholz, H. M., Krystal, J. H., & Paulus, M.
- Affiliations
- Yale University (US)
- New York City Council (US)
- University of Augsburg (DE)
- University of Cologne (DE)
- Ludwig-Maximilians-Universität München (DE)
- Yale New Haven Hospital (US)
- Laureate Institute for Brain Research (US)
- Corresponding author
- Chekroud, Adam M.
- ORCID
- Venue
- Science
- Published
- 2024-01-12
- Volume
- 383
- Issue
- 6679
- Pages
- 164-167
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4390734451
- Cited by
- 251
- References
- 37
- Citations by year
- 2026: 76 · 2025: 96 · 2024: 78
- FWCI
- 36.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical prediction model, generalizability, out-of-sample forecasting, antipsychotic drugs, schizophrenia, machine learning, context sensitivity, treatment outcome prediction, clinical decision-making
- MeSH terms
- Machine Learning, Adolescent, Adult, Aged, Aged, 80 and over, Child, Female, Humans, Male, Middle Aged, Prognosis, Schizophrenia, Antipsychotic Agents, Models, Statistical, Treatment Outcome, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.