The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Personalized Prediction of Psychosis: External Validation of the NAPLS-2 Psychosis Risk Calculator With the EDIPPP Project
Open accessAbstract
OBJECTIVE: As part of the second phase of the North American Prodrome Longitudinal Study (NAPLS-2), Cannon and colleagues report, concurrently with the present article, on a risk calculator for the individualized prediction of a psychotic disorder in a 2-year period. The present study represents an external validation of the NAPLS-2 psychosis risk calculator using an independent sample of patients at clinical high risk for psychosis collected as part of the Early Detection, Intervention, and Prevention of Psychosis Program (EDIPPP). METHOD: Of the total EDIPPP sample of 210 subjects rated as being at clinical high risk based on the Structured Interview for Prodromal Syndromes, 176 had at least one follow-up assessment and were included in the construction of a new prediction model with six predictor variables in the NAPLS-2 psychosis risk calculator (unusual thoughts and suspiciousness, symbol coding test performance, verbal learning test performance, decline in social functioning, baseline age, and family history). Discrimination performance was assessed with the area under the receiver operating characteristic curve (AUC). The NAPLS-2 risk calculator was then used to generate a psychosis risk estimate for each case in the external validation sample. RESULTS: The external validation model showed good discrimination, with an AUC of 0.790 (95% CI=0.644-0.937). In addition, the personalized risk generated by the risk calculator provided a solid estimation of the actual conversion outcome in the validation sample. CONCLUSIONS: Two independent samples of clinical high-risk patients converge to validate the NAPLS-2 psychosis risk calculator. This prediction calculator represents a meaningful step toward early intervention and the personalized treatment of psychotic disorders.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“176 (16.6, US); 0.79 (AUC) 292 199 (19.1, China); 0.63 (AUC) 293 98 (18.59, US); 0.71 (AUC) 294”
From Preventive psychiatry: a blueprint for improving the mental health of young people (2021) · cites this paper as [292]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Stone, N. J., Robinson, J. G., Lichtenstein, A. H., Bairey Merz, C. N., Blum, C. B., Eckel, R. H., Goldberg, A. C., Gordon, D., Levy, D., Lloyd-Jones, D. M., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437738.63853.7a · Cited by 7,081
Non-Parametric Statistics for the Behavioral Sciences.
Stuart, A., & Siegel, S. (1957) · Economica · doi:10.2307/2551606 · Cited by 3,403
Lindström, J., & Tuomilehto, J. O. (2003) · Diabetes Care · doi:10.2337/diacare.26.3.725 · Cited by 1,937 · cited 1×
“A number of risk calculators are now freely available to the public and can estimate the risk, for example, of prostate, ovarian, breast, pancreatic, and colorectal cancer(4–6), type 2 diabetes(7, 8), and cardiovascular disease (9–11).”
BackgroundRidker, P. M., Buring, J. E., Rifai, N., & Cook, N. R. (2007) · JAMA · doi:10.1001/jama.297.6.611 · Cited by 1,814
Miller, T. J., McGlashan, T. H., Rosen, J. L., Cadenhead, K. S., Ventura, J., McFarlane, W. R., Perkins, D. O., Pearlson, G. D., & Woods, S. W. (2003) · Schizophrenia Bulletin · doi:10.1093/oxfordjournals.schbul.a007040 · Cited by 1,800
Self-citationSome suggestions for measuring predictive performance
Sheiner, L. B., & Beal, S. L. (1981) · Journal of Pharmacokinetics and Biopharmaceutics · doi:10.1007/bf01060893 · Cited by 1,618 · cited 1×
“MPE was calculated as the difference between the risk estimates generated by the external validation model and the NAPLS2 psychosis risk calculator, providing a metric for the tendency for overestimation/underestimation of risk(41).”
MethodBenedict, R. H. B., Schretlen, D., Groninger, L., & Brandt, J. (1998) · The Clinical Neuropsychologist · doi:10.1076/clin.12.1.43.1726 · Cited by 1,597
The Psychosis High-Risk State: A Comprehensive State-of-the-Art Review
Fusar-Poli, P., Borgwardt, S., Bechdolf, A., Addington, J., Riecher-Rössler, A., Schultze-Lutter, F., Keshavan, M., Wood, S., Ruhrmann, S., Seidman, L. J., et al. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.269 · Cited by 1,483 · cited 1×
“Advances include operationalizing and validating clinical criteria that identify individuals considered to be prodromal or at clinical high risk (CHR) for psychosis, approximately 29% of whom will develop a psychotic illness over two years(13).”
BackgroundSelf-citationPrognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337 · cited 1×
“Over time, prevention has progressed from identifying populations at risk to personalizing risk estimates(1, 2).”
BackgroundCannon, T. D., Cadenhead, K., Cornblatt, B., Woods, S. W., Addington, J., Walker, E., Seidman, L. J., Perkins, D., Tsuang, M., McGlashan, T., et al. (2008) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2007.3 · Cited by 1,269
Self-citationUsers' Guides to the Medical Literature
McGinn, T. G., Guyatt, G., Wyer, P. C., Naylor, C. D., Stiell, I. G., Richardson, W. S., & for the Evidence-Based Medicine Working Group (2000) · JAMA · doi:10.1001/jama.284.1.79 · Cited by 1,009
Kapur, S., Phillips, A. G., & Insel, T. R. (2012) · Molecular Psychiatry · doi:10.1038/mp.2012.105 · Cited by 980 · cited 1×
“Lack of replications of well-performing models and complex biological findings with limited clinical applicability may have contributed to this delay(43).”
BackgroundGreen, M. F., Nuechterlein, K. H., Gold, J. M., Barch, D. M., Cohen, J., Essock, S., Fenton, W. S., Frese, F., Goldberg, T. E., Heaton, R. K., et al. (2004) · Biological Psychiatry · doi:10.1016/j.biopsych.2004.06.023 · Cited by 919 · cited 1×
“In addition to several other clinical measures, the baseline assessment included the Measurement and Treatment Research to Improve Cognition in Schizophrenia Consensus Cognitive Battery (MCCB)(32).”
MethodPredicting the 30-Year Risk of Cardiovascular Disease
Pencina, M. J., D’Agostino, R. B., Larson, M. G., Massaro, J. M., & Vasan, R. S. (2009) · Circulation · doi:10.1161/circulationaha.108.816694 · Cited by 885
Miller, T. J., McGlashan, T. H., Rosen, J. L., Somjee, L., Markovich, P. J., Stein, K., & Woods, S. W. (2002) · American Journal of Psychiatry · doi:10.1176/appi.ajp.159.5.863 · Cited by 872
Ridker, P. M., Paynter, N. P., Rifai, N., Gaziano, J. M., & Cook, N. R. (2008) · Circulation · doi:10.1161/circulationaha.108.814251 · Cited by 841
Symptom Assessment in Schizophrenic Prodromal States
Miller, T. J., McGlashan, T. H., Woods, S. W., Stein, K., Driesen, N. R., Corcoran, C. M., Hoffman, R. E., & Davidson, L. (1999) · Psychiatric Quarterly · doi:10.1023/a:1022034115078 · Cited by 641
Ruhrmann, S., Schultze-Lutter, F., Salokangas, R. K. R., Heinimaa, M., Linszen, D., Dingemans, P., Birchwood, M., Patterson, P., Juckel, G., Heinz, A., et al. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2009.206 · Cited by 632
Cornblatt, B., Auther, A. M., Niendam, T. A., Smith, C. W., Zinberg, J. L., Bearden, C. E., & Cannon, T. D. (2007) · Schizophrenia Bulletin · doi:10.1093/schbul/sbm029 · Cited by 631 · cited 2×
“Decline in social functioning in the year prior to the baseline assessment, measured using the GF:S scale (31, 35, 36); and 6.”
MethodSelf-citationClinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 595 · cited 1×
“For both measures, a lower score indicates higher precision and less bias and ≤15% was considered to be an acceptable level(42).”
An Individualized Risk Calculator for Research in Prodromal Psychosis
Cannon, T. D., Yu, C., Addington, J., Bearden, C. E., Cadenhead, K. S., Cornblatt, B. A., Heinssen, R., Jeffries, C. D., Mathalon, D. H., McGlashan, T. H., et al. (2016) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2016.15070890 · Cited by 574
Self-citationDeFilippis, A. P., Young, R. L., Carrubba, C. J., McEvoy, J. W., Budoff, M. J., Blumenthal, R. S., Kronmal, R. A., McClelland, R. L., Nasir, K., & Blaha, M. J. (2015) · Annals of Internal Medicine · doi:10.7326/m14-1281 · Cited by 534
Assessing the economic costs of serious mental illness
Insel, T. R. (2008) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2008.08030366 · Cited by 487 · cited 1×
“2 billion in lost earnings per year(12).”
Long-term Follow-up of a Group at Ultra High Risk (“Prodromal”) for Psychosis
Nelson, B., Yuen, H. P., Wood, S. J., Lin, A. P., Spiliotacopoulos, D., Bruxner, A., Broussard, C., Simmons, M. B., Foley, D. L., Brewer, W. J., et al. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.1270 · Cited by 434
Hippisley‐Cox, J., Coupland, C. A. C., Robson, J. P., Sheikh, A., & Brindle, P. M. (2009) · BMJ · doi:10.1136/bmj.b880 · Cited by 373 · cited 1×
“A number of risk calculators are now freely available to the public and can estimate the risk, for example, of prostate, ovarian, breast, pancreatic, and colorectal cancer(4–6), type 2 diabetes(7, 8), and cardiovascular disease (9–11).”
BackgroundCardiovascular Disease Risk Assessment: Insights from Framingham
D’Agostino, R. B., Pencina, M. J., Massaro, J. M., & Coady, S. (2013) · Global Heart · doi:10.1016/j.gheart.2013.01.001 · Cited by 350 · cited 2×
“Genetic Risk and Deterioration Syndrome (GRD), genetic risk for psychosis coupled with deterioration in functioning, and (3) Brief Intermittent Psychotic Syndrome (BIPS), intermittent psychotic symptoms that are recent, brief in duration, and not seriously disorganizing or dangerous.”
BackgroundNorth American Prodrome Longitudinal Study (NAPLS 2): Overview and recruitment
Addington, J., Cadenhead, K. S., Cornblatt, B. A., Mathalon, D. H., McGlashan, T. H., Perkins, D. O., Seidman, L. J., Tsuang, M. T., Walker, E. F., Woods, S. W., et al. (2012) · Schizophrenia Research · doi:10.1016/j.schres.2012.09.012 · Cited by 272 · cited 1×
“In the current issue of the American Journal of Psychiatry, Cannon and colleagues, as part of the second phase of the North American Prodrome Longitudinal Study (NAPLS2)(20), report on a risk calculator for the individualized prediction of developing a psychotic disorder in a 2-year period.”
BackgroundSelf-citationRisk Factors for Psychosis: Impaired Social and Role Functioning
Cornblatt, B. A., Carrión, R. E., Addington, J., Seidman, L. J., Walker, E. F., Cannon, T. D., Cadenhead, K. S., McGlashan, T. H., Perkins, D. O., Tsuang, M. T., et al. (2011) · Schizophrenia Bulletin · doi:10.1093/schbul/sbr136 · Cited by 265 · cited 1×
“Decline in social functioning in the year prior to the baseline assessment, measured using the GF:S scale (31, 35, 36); and 6.”
MethodSelf-citationPrediction of Functional Outcome in Individuals at Clinical High Risk for Psychosis
Carrión, R. E., McLaughlin, D., Goldberg, T. E., Auther, A. M., Olsen, R. H., Olvet, D. M., Correll, C. U., & Cornblatt, B. A. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.1909 · Cited by 250 · cited 1×
“Decline in social functioning in the year prior to the baseline assessment, measured using the GF:S scale (31, 35, 36); and 6.”
MethodSelf-citationHippisley‐Cox, J., & Coupland, C. A. C. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-007825 · Cited by 177
Assessing Individual Risk for Prostate Cancer
Nam, R. K., Toi, A., Klotz, L. H., Trachtenberg, J., Jewett, M. A. S., Appu, S., Loblaw, D. A., Sugar, L., Narod, S. A., & Kattan, M. W. (2007) · Journal of Clinical Oncology · doi:10.1200/jco.2007.10.6450 · Cited by 175
Cornblatt, B. A., Carrión, R. E., Auther, A. M., McLaughlin, D., Olsen, R. H., John, M., & Correll, C. U. (2015) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2015.13121686 · Cited by 155
Self-citationMcFarlane, W. R., Levin, B. L., Travis, L., Lucas, F. L., Lynch, S., Verdi, M., Williams, D. L., Adelsheim, S. N., Calkins, R., Carter, C. S., et al. (2014) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu108 · Cited by 117 · cited 4×
“8%) transitioned to psychosis over 2 years.(24) Conversion to psychosis Carrión et al.”
BackgroundMethodInfluentialSelf-citationLoewy, R. L., Fisher, M., Schlosser, D. A., Biagianti, B., Stuart, B. K., Mathalon, D. H., & Vinogradov, S. (2016) · Schizophrenia Bulletin · doi:10.1093/schbul/sbw009 · Cited by 95 · cited 1×
“Individuals with substantial neurocognitive difficulties could be offered, for example, cognitive training.(46) On the other hand, high-risk individuals and/or individuals with more severe positive symptoms would potentially be offered more aggressive intervention, possibly involving medications.”
BackgroundThe Role of Physicians in the Era of Predictive Analytics
Sniderman, A. D., D’Agostino, R. B., & Pencina, M. J. (2015) · JAMA · doi:10.1001/jama.2015.6177 · Cited by 79 · cited 2×
“Finally, since the field of psychosis prevention is constantly evolving, an accurate assessment of risk requires constant updating to take into account other factors not included in the prediction model that may alter risks/ benefits(1).”
BackgroundWells, B. J., Kattan, M. W., Cooper, G. S., Jackson, L. W., & Koroukian, S. M. (2014) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2014.01.130040 · Cited by 64
A Stratified Model for Psychosis Prediction in Clinical Practice
Michel, C., Ruhrmann, S., SCHIMMELMANN, B. G., Klosterkötter, J., & Schultze‐Lutter, F. (2014) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu025 · Cited by 63
McFarlane, W. R., Stastny, P., & Deakins, S. M. (1992) · New Directions for Mental Health Services · doi:10.1002/yd.23319925306 · Cited by 61 · cited 2×
“The EDIPPP project was a large nationwide clinical trial designed to examine the effectiveness of Family-aided Assertive Community Treatment (FACT)(22, 23) in preventing the onset of psychosis(24).”
BackgroundSelf-citationSeverity of Thought Disorder Predicts Psychosis in Persons at Clinical High-Risk
Perkins, D., Jeffries, C. D., Cornblatt, B., Woods, S., Addington, J., Bearden, C., Cadenhead, K., Cannon, T. D., Heinssen, R., Mathalon, D., et al. (2015) · Schizophrenia Research · doi:10.1016/j.schres.2015.09.008 · Cited by 52 · cited 1×
“Severity of SIPS items P1+P2 (Unusual Thought Content and Suspiciousness) recoded(18); 3.”
BackgroundSelf-citationMcFarlane, W. R., Cook, W. L., Downing, D., Verdi, M., Woodberry, K. A., & Ruff, A. (2010) · Psychiatric Services · doi:10.1176/appi.ps.61.5.512 · Cited by 48
Self-citationMcFarlane, W. R., Cook, W. L., Downing, D., et al. (2010) · Psychiatric Services · doi:10.1176/ps.2010.61.5.512 · Cited by 38
Self-citationFjell, A., Thorsen, G. R. B., Friis, S., Johannessen, J. O., Larsen, T. K., Lie, K., Lyse, H., Melle, I. S., Simonsen, E. H., Smeby, N. A., et al. (2007) · Psychiatric Services · doi:10.1176/ps.2007.58.2.171 · Cited by 35
Self-citationLynch, S. B., McFarlane, W. R., Joly, B. M., Adelsheim, S. N., Auther, A. M., Cornblatt, B. A., Migliorati, M., Ragland, J. D., Sale, T. G., Spring, E., et al. (2016) · Psychiatric Services · doi:10.1176/appi.ps.201300236 · Cited by 29
Self-citationMcFarlane, W. R., Cook, W. L., Downing, D., Ruff, A., Lynch, S. B., Adelsheim, S. N., Calkins, R., Carter, C. S., Cornblatt, B. A., & Milner, K. K. (2012) · Adolescent Psychiatry · doi:10.2174/2210676611202020112 · Cited by 24 · cited 2×
“with the goals of raising awareness of the early warning signs of psychosis and demonstrating the effectiveness of community outreach, education and early referral, combined with the FACT intervention, to reduce the incidence of psychosis(25).”
BackgroundMethodSelf-citationRisk Prediction for Individuals
CALSTER, B. V., Steyerberg, E. W., & Harrell, F. H. (2015) · JAMA · doi:10.1001/jama.2015.12215 · Cited by 16 · cited 1×
“25% risk should meaningfully impact treatment decisions(45).”
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“The Brier score is the mean squared difference between the observed outcome and predicted risk score(40).”
Background
Research works printed but not linked: 3
- Keefe R, 1999, Brief Assessment of Cognition in Schizophrenia (BACS) Manual–A: Version 2.1
- Siegel S, 1988, Nonparametric Statistics for the Behavioral Sciences
- Loewy R, Schizophr Bull
Cited by
Sorted by citations
Machine Learning Approaches for Clinical Psychology and Psychiatry
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966
An Individualized Risk Calculator for Research in Prodromal Psychosis
Cannon, T. D., Yu, C., Addington, J., Bearden, C. E., Cadenhead, K. S., Cornblatt, B. A., Heinssen, R., Jeffries, C. D., Mathalon, D. H., McGlashan, T. H., et al. (2016) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2016.15070890 · Cited by 574
“In a companion article (30), Carrión et al.”
BackgroundMethodSelf-citationPreventive psychiatry: a blueprint for improving the mental health of young people
Fusar‐Poli, P., Correll, C. U., Arango, C., Berk, M., Patel, V., & Ioannidis, J. P. A. (2021) · World Psychiatry · doi:10.1002/wps.20869 · Cited by 497
Candidate biomarkers in psychiatric disorders: state of the field
Abi‐Dargham, A., Moeller, S. J., Ali, F., DeLorenzo, C., Domschke, K., Horga, G., Jutla, A., Kotov, R., Paulus, M. P., Rubio, J. M., et al. (2023) · World Psychiatry · doi:10.1002/wps.21078 · Cited by 421
What is the risk‐benefit ratio of long‐term antipsychotic treatment in people with schizophrenia?
Correll, C. U., Rubio, J. M., & Kane, J. M. (2018) · World Psychiatry · doi:10.1002/wps.20516 · Cited by 324
Metadata
- Authors
- Carrión, R. E., Cornblatt, B. A., Burton, C. Z., Tso, I. F., Auther, A. M., Adelsheim, S., Calkins, R., Carter, C. S., Niendam, T., Sale, T. G., Taylor, S. F., & McFarlane, W. R.
- Affiliations
- Northwell Health (US)
- ORCID
- Venue
- American Journal of Psychiatry
- Published
- 2016-10-01
- Volume
- 173
- Issue
- 10
- Pages
- 989-996
- ISSN
- 0002-953X (ISSN-L)
- 1535-7228
- PMC
- PMC5048503
- OpenAlex
- W2462155653
- Cited by
- 170
- References
- 47
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 7 · 2023: 13 · 2022: 12 · 2021: 16 · 2020: 24 · 2019: 30 · 2018: 21 · 2017: 21 · 2016: 7
- FWCI
- 23.39
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- American Journal of Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- PDXScholar (Portland State University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Institutes of Health (KL2TR000434)
- National Institute of Mental Health (R01MH061523, P30MH074543, R01MH059883, K23MH087708, R21MH101676, K23MH108823, U01MH081857)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Social Sciences › Psychology › Clinical Psychology › Psychopathy, Forensic Psychiatry, Sexual Offending
- Health Sciences › Medicine › Psychiatry and Mental health › Bipolar Disorder and Treatment
- Keywords
- psychosis prediction, clinical high risk for psychosis, external validation, early detection and intervention, prodromal psychosis, transition to psychosis, suspicion, symbol coding, verbal learning, family history (medicine), personalized treatment
- MeSH terms
- Adolescent, Adult, Child, Female, Humans, Longitudinal Studies, Male, Models, Psychological, Predictive Value of Tests, Psychiatric Status Rating Scales, Psychotic Disorders, Risk Factors, Early Diagnosis, Young Adult, Prodromal Symptoms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
