CoChat
Sign Up
Cited by 341Schizophrenia research and treatmentBipolar Disorder and TreatmentPsychopathy, Forensic Psychiatry, Sexual Offending

Prediction Models of Functional Outcomes for Individuals in the Clinical High-Risk State for Psychosis or With Recent-Onset Depression: A Multimodal, Multisite Machine Learning Analysis

Open access
Koutsouleris, Nikolaos, Kambeitz-Ilankovic, Lana, Ruhrmann, Stephan, Rosen, Marlene, Ruef, Anne, Dwyer, Dominic B., Paolini, Marco, Chisholm, Katharine +18 more·2018·JAMA Psychiatry
DOI 10.1001/jamapsychiatry.2018.2165PMC PMC6248111
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Social and occupational impairments contribute to the burden of psychosis and depression. There is a need for risk stratification tools to inform personalized functional-disability preventive strategies for individuals in at-risk and early phases of these illnesses. Objective: To determine whether predictors associated with social and role functioning can be identified in patients in clinical high-risk (CHR) states for psychosis or with recent-onset depression (ROD) using clinical, imaging-based, and combined machine learning; assess the geographic, transdiagnostic, and prognostic generalizability of machine learning and compare it with human prognostication; and explore sequential prognosis encompassing clinical and combined machine learning. Design, Setting, and Participants: This multisite naturalistic study followed up patients in CHR states, with ROD, and with recent-onset psychosis, and healthy control participants for 18 months in 7 academic early-recognition services in 5 European countries. Participants were recruited between February 2014 and May 2016, and data were analyzed from April 2017 to January 2018. ain Outcomes and Measures: Performance and generalizability of prognostic models. Results: A total of 116 individuals in CHR states (mean [SD] age, 24.0 [5.1] years; 58 [50.0%] female) and 120 patients with ROD (mean [SD] age, 26.1 [6.1] years; 65 [54.2%] female) were followed up for a mean (SD) of 329 (142) days. Machine learning predicted the 1-year social-functioning outcomes with a balanced accuracy of 76.9% of patients in CHR states and 66.2% of patients with ROD using clinical baseline data. Balanced accuracy in models using structural neuroimaging was 76.2% in patients in CHR states and 65.0% in patients with ROD, and in combined models, it was 82.7% for CHR states and 70.3% for ROD. Lower functioning before study entry was a transdiagnostic predictor. Medial prefrontal and temporo-parieto-occipital gray matter volume (GMV) reductions and cerebellar and dorsolateral prefrontal GMV increments had predictive value in the CHR group; reduced mediotemporal and increased prefrontal-perisylvian GMV had predictive value in patients with ROD. Poor prognoses were associated with increased risk of psychotic, depressive, and anxiety disorders at follow-up in patients in the CHR state but not ones with ROD. Machine learning outperformed expert prognostication. Adding neuroimaging machine learning to clinical machine learning provided a 1.9-fold increase of prognostic certainty in uncertain cases of patients in CHR states, and a 10.5-fold increase of prognostic certainty for patients with ROD. Conclusions and Relevance: Precision medicine tools could augment effective therapeutic strategies aiming at the prevention of social functioning impairments in patients with CHR states or with ROD.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamapsychiatry.2018.2165

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Incorporating disparate data lends itself to seemingly better predictions.

    “While incorporating disparate data does lend itself to seemingly better predictions139, as knowledge around certain diseases accumulates, data fusion in healthcare is an evolving target that warrants proactively adapting to the dynamic landscape190.”

    From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [139]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef, Dwyer, Paolini, Chisholm, Kambeitz, Haidl, Schmidt, Schultze-Lutter, Falkai, Reiser, Riecher-Rossler, Upthegrove, Hietala, Salokangas, Pantelis, Meisenzahl, Wood, Beque, Brambilla, Borgwardt. Statistical analysis: Koutsouleris, Kambeitz, Upthegrove. Obtained funding: Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Riecher-Rossler, Salokangas, Pantelis, Brambilla, Borgwardt, Wood. Administrative, technical, or material support: Koutsouleris, Rosen, Ruef, Paolini, Chisholm, Haidl, Schmidt, Gillam, Reiser, Upthegrove, Hietala, Meisenzahl, Beque, Brambilla, Borgwardt. Supervision: Koutsouleris, Ruhrmann, Rosen, Schultze-Lutter, Falkai, Reiser, Riecher-Rossler, Hietala, Wood, Brambilla, Borgwardt. REFERENCES 1. Addington J, Cornblatt BA, Cadenhead KS, et al. At clinical high risk for psychosis: outcome for nonconverters. Am J Psychiatry. 2011;168(8): 800-805. doi:10.1176/appi.ajp.2011.10081191 · DOI
  2. Salokangas RKR, Nieman DH, Heinimaa M, et al; EPOS group. Psychosocial outcome in patients at clinical high risk of psychosis: a prospective follow-up. Soc Psychiatry Psychiatr Epidemiol. 2013; 48(2):303-311. doi:10.1007/s00127-012-0545-2 · DOI
  3. McGorry P, Keshavan M, Goldstone S, et al. Biomarkers and clinical staging in psychiatry. World Psychiatry. 2014;13(3):211-223. doi:10.1002/wps .20144 · DOI
  4. Schmidt SJ, Schultze-Lutter F, Schimmelmann BG, et al. EPA guidance on the early intervention in clinical high risk states of psychoses. Eur Psychiatry. 2015;30(3):388-404. doi:10.1016/j.eurpsy.2015.01 .013 · DOI · Paper page
  5. Fusar-Poli P, Borgwardt S, Bechdolf A, et al. The psychosis high-risk state: a comprehensive state-of-the-art review. JAMA Psychiatry. 2013;70 (1):107-120. doi:10.1001/jamapsychiatry.2013.269 · DOI · Paper page
  6. Simon AE, Velthorst E, Nieman DH, Linszen D, Umbricht D, de Haan L. Ultra high-risk state for psychosis and non-transition: a systematic review. Schizophr Res. 2011;132(1):8-17. doi:10.1016/j.schres .2011.07.002 · DOI
  7. Michel C, Ruhrmann S, Schimmelmann BG, Klosterkötter J, Schultze-Lutter F. Course of clinical high-risk states for psychosis beyond conversion. Eur Arch Psychiatry Clin Neurosci. 2018;268(1): 39-48. doi:10.1007/s00406-016-0764-8 · DOI
  8. Cornblatt BA, Auther AM, Niendam T, et al. Preliminary findings for two new measures of social and role functioning in the prodromal phase of schizophrenia. Schizophr Bull. 2007;33(3):688-702. doi:10.1093/schbul/sbm029 · DOI
  9. Fusar-Poli P, Byrne M, Valmaggia L, et al; OASIS Team. Social dysfunction predicts two years clinical outcome in people at ultra high risk for psychosis. J Psychiatr Res. 2010;44(5):294-301. doi:10.1016/j .jpsychires.2009.08.016 · DOI
  10. Salokangas RK, Heinimaa M, From T, et al; EPOS group. Short-term functional outcome and premorbid adjustment in clinical high-risk patients: results of the EPOS project. Eur Psychiatry. 2014;29 (6):371-380. doi:10.1016/j.eurpsy.2013.10.003 · DOI
  11. Fusar-Poli P, Nelson B, Valmaggia L, Yung AR, McGuire PK. Comorbid depressive and anxiety disorders in 509 individuals with an at-risk mental state: impact on psychopathology and transition to psychosis. Schizophr Bull. 2014;40(1):120-131. doi:10.1093/schbul/sbs136 · DOI · Paper page
  12. Bodatsch M, Ruhrmann S, Wagner M, et al. Prediction of psychosis by mismatch negativity. Biol Psychiatry. 2011;69(10):959-966. doi:10.1016/j .biopsych.2010.09.057 · DOI
  13. Koutsouleris N, Riecher-Rössler A, Meisenzahl EM, et al. Detecting the psychosis prodrome across high-risk populations using neuroanatomical biomarkers. Schizophr Bull. 2015;41(2):471-482. doi:10.1093/schbul/sbu078 · DOI
  14. Michel C, Ruhrmann S, Schimmelmann BG, Klosterkötter J, Schultze-Lutter F. A stratified model for psychosis prediction in clinical practice. Schizophr Bull. 2014;40(6):1533-1542. doi:10.1093 /schbul/sbu025 · DOI
  15. Nieman DH, Ruhrmann S, Dragt S, et al. Psychosis prediction: stratification of risk estimation with information-processing and premorbid functioning variables. Schizophr Bull. 2014;40(6):1482-1490. doi:10.1093/schbul/sbt145 · DOI
  16. Fusar-Poli P, Rutigliano G, Stahl D, et al. Development and validation of a clinically based risk calculator for the transdiagnostic prediction of psychosis. JAMA Psychiatry. 2017;74(5):493-500. doi:10.1001/jamapsychiatry.2017.0284 · DOI · Paper page
  17. Kessler RC, van Loo HM, Wardenaar KJ, et al. Testing a machine-learning algorithm to predict the persistence and severity of major depressive disorder from baseline self-reports. Mol Psychiatry. 2016;21(10):1366-1371. doi:10.1038/mp.2015.198 · DOI
  18. Koutsouleris N, Kahn RS, Chekroud AM, et al. Multisite prediction of 4-week and 52-week treatment outcomes in patients with first-episode psychosis: a machine learning approach. Lancet Psychiatry. 2016;3(10):935-946. doi:10.1016/S2215 -0366(16)30171-7 · DOI
  19. Kambeitz-Ilankovic L, Meisenzahl EM, Cabral C, et al. Prediction of outcome in the psychosis prodrome using neuroanatomical pattern classification. Schizophr Res. 2016;173(3):159-165. doi:10.1016/j.schres.2015.03.005 · DOI
  20. de Wit S, Ziermans TB, Nieuwenhuis M, et al. Individual prediction of long-term outcome in adolescents at ultra-high risk for psychosis: applying machine learning techniques to brain imaging data. Hum Brain Mapp. 2017;38(2):704-714. doi:10.1002/hbm.23410 · DOI
  21. Velthorst E, Nieman DH, Linszen D, et al. Disability in people clinically at high risk of psychosis. Br J Psychiatry. 2010;197(4):278-284. doi:10.1192/bjp.bp.109.075036 · DOI
  22. Cornblatt BA, Carrión RE, Addington J, et al. Risk factors for psychosis: impaired social and role functioning. Schizophr Bull. 2012;38(6):1247-1257. doi:10.1093/schbul/sbr136 · DOI
  23. Studerus E, Ramyead A, Riecher-Rössler A. Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting. Psychol Med. 2017; 47(7):1163-1178. doi:10.1017/S0033291716003494 · DOI
  24. Schmidt A, Cappucciati M, Radua J, et al. Improving prognostic accuracy in subjects at clinical high risk for psychosis: systematic review of predictive models and meta-analytical sequential testing simulation. Schizophr Bull. 2017;43(2): 375-388.
  25. Hahn T, Nierenberg AA, Whitfield-Gabrieli S. Predictive analytics in mental health: applications, guidelines, challenges and perspectives. Mol Psychiatry. 2017;22(1):37-43. doi:10.1038/mp.2016 .201 · DOI
  26. Chekroud AM, Koutsouleris N. The perilous path from publication to practice. Mol Psychiatry. 2018;23(1):24-25. doi:10.1038/mp.2017.227 · DOI
  27. McFarlane WR, Levin B, Travis L, et al. Clinical and functional outcomes after 2 years in the early detection and intervention for the prevention of psychosis multisite effectiveness trial. Schizophr Bull. 2015;41(1):30-43. doi:10.1093/schbul/sbu108 · DOI
  28. Gaonkar B, T Shinohara R, Davatzikos C; Alzheimers Disease Neuroimaging Initiative. Interpreting support vector machine models for multivariate group wise analysis in neuroimaging. Med Image Anal. 2015;24(1):190-204. doi:10.1016/j .media.2015.06.008 · DOI
  29. Ding C, Li T, Peng W, Park H. Orthogonal Nonnegative Matrix T-factorizations for Clustering. In: Proceedings of the 12th Association for Computing Machinery's Special Interest Group on Knowledge Discovery and Data Mining International Conference, Philadelphia, PA; August 20-23, 2006; 126-135.
  30. Huys QJ, Maia TV, Frank MJ. Computational psychiatry as a bridge from neuroscience to clinical applications. Nat Neurosci. 2016;19(3):404-413. doi:10.1038/nn.4238 · DOI · Paper page
  31. Paulus MP, Huys QJ, Maia TV. A roadmap for the development of applied computational psychiatry. Biol Psychiatry Cogn Neurosci Neuroimaging. 2016;1(5):386-392. doi:10.1016/j.bpsc .2016.05.001 · DOI
  32. Eack SM, Newhill CE, Keshavan MS. Cognitive enhancement therapy improves resting-state functional connectivity in early course schizophrenia. J Soc Social Work Res. 2016;7(2): 211-230. doi:10.1086/686538 · DOI · Paper page
  33. Eack SM, Hogarty GE, Cho RY, et al. Neuroprotective effects of cognitive enhancement therapy against gray matter loss in early schizophrenia: results from a 2-year randomized controlled trial. Arch Gen Psychiatry. 2010;67(7): 674-682. doi:10.1001/archgenpsychiatry.2010.63 · DOI · Paper page
  34. Keshavan MS, Eack SM, Wojtalik JA, et al. A broad cortical reserve accelerates response to cognitive enhancement therapy in early course schizophrenia. Schizophr Res. 2011;130(1-3):123-129. doi:10.1016/j.schres.2011.05.001 · DOI
  35. Burcusa SL, Iacono WG. Risk for recurrence in depression. Clin Psychol Rev. 2007;27(8):959-985. doi:10.1016/j.cpr.2007.02.005 · DOI · Paper page
  36. Evert H, Harvey C, Trauer T, Herrman H. The relationship between social networks and occupational and self-care functioning in people with psychosis. Soc Psychiatry Psychiatr Epidemiol. 2003;38(4):180-188. doi:10.1007/s00127-003 -0617-4 · DOI
  37. Lidwall U, Bill S, Palmer E, Olsson Bohlin C. Mental disorder sick leave in Sweden: a population study. Work. 2018;59(2):259-272. doi:10.3233 /WOR-172672 · DOI
  38. Glenthøj LB, Fagerlund B, Hjorthøj C, et al. Social cognition in patients at ultra-high risk for psychosis: what is the relation to social skills and functioning? Schizophr Res Cogn. 2016;5:21-27. doi:10.1016/j.scog.2016.06.004 · DOI
  39. Simons CJP, Bartels-Velthuis AA, Pijnenborg GHM; Genetic Risk and Outcome of Psychosis (GROUP) Investigators. Cognitive performance and long-term social functioning in psychotic disorder: a three-year follow-up study. PLoS One. 2016;11(4): e0151299. doi:10.1371/journal.pone.0151299 · DOI
  40. Hua LL, Wilens TE, Martelon M, Wong P, Wozniak J, Biederman J. Psychosocial functioning, familiality, and psychiatric comorbidity in bipolar youth with and without psychotic features. J Clin Psychiatry. 2011;72(3):397-405. doi:10.4088/JCP .10m06025yel · DOI
  41. Koutsouleris N, Davatzikos C, Bottlender R, et al. Early recognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification. Schizophr Bull. 2012;38(6):1200-1215. doi:10.1093/schbul/sbr037 · DOI
  42. Bedi G, Carrillo F, Cecchi GA, et al. Automated analysis of free speech predicts psychosis onset in high-risk youths. NPJ Schizophr. 2015;1:15030. doi: 10.1038/npjschz.2015.30 · DOI · Paper page
  43. Frank E, Maier D, Pajula J, et al. Platform for systems medicine research and diagnostic applications in psychotic disorders-The METSY project. Eur Psychiatry. 2018;50:40-46. doi:10.1016 /j.eurpsy.2017.12.001 · DOI
  44. Palaniyappan L, Liddle PF. Does the salience network play a cardinal role in psychosis? an emerging hypothesis of insular dysfunction. J Psychiatry Neurosci. 2012;37(1):17-27. doi:10.1503 /jpn.100176 · DOI · Paper page
  45. Wang H, Zeng L-L, Chen Y, Yin H, Tan Q, Hu D. Evidence of a dissociation pattern in default mode subnetwork functional connectivity in schizophrenia. Sci Rep. 2015;5:14655. doi:10.1038 /srep14655 · DOI
  46. Fusar-Poli P, Radua J, McGuire P, Borgwardt S. Neuroanatomical maps of psychosis onset: voxel-wise meta-analysis of antipsychotic-naive VBM studies. Schizophr Bull. 2012;38(6):1297-1307. doi:10.1093/schbul/sbr134 · DOI
  47. Wood SJ, Yung AR, McGorry PD, Pantelis C. Neuroimaging and treatment evidence for clinical staging in psychotic disorders: from the at-risk mental state to chronic schizophrenia. Biol Psychiatry. 2011;70(7):619-625. doi:10.1016/j.biopsych.2011.05 .034 · DOI
  48. Douaud G, Mackay C, Andersson J, et al. Schizophrenia delays and alters maturation of the brain in adolescence. Brain. 2009;132(pt 9): 2437-2448. doi:10.1093/brain/awp126 · DOI
  49. Falkai P, Rossner MJ, Schulze TG, et al. Kraepelin revisited: schizophrenia from degeneration to failed regeneration. Mol Psychiatry. 2015;20(6):671-676. doi:10.1038/mp.2015.35 · DOI
  50. Bartholomeusz CF, Cropley VL, Wannan C, Di Biase M, McGorry PD, Pantelis C. Structural neuroimaging across early-stage psychosis: Aberrations in neurobiological trajectories and implications for the staging model. Aust N Z J Psychiatry. 2017;51(5):455-476. doi:10.1177 /0004867416670522 · DOI
  51. Torres US, Duran FL, Schaufelberger MS, et al. Patterns of regional gray matter loss at different stages of schizophrenia: a multisite, cross-sectional VBM study in first-episode and chronic illness. Neuroimage Clin. 2016;12:1-15. doi:10.1016/j.nicl .2016.06.002 · DOI
  52. Palaniyappan L. Progressive cortical reorganisation: A framework for investigating structural changes in schizophrenia. Neurosci Biobehav Rev. 2017;79:1-13. doi:10.1016/j.neubiorev .2017.04.028 · DOI
  53. Ryali S, Supekar K, Chen T, et al. Temporal dynamics and developmental maturation of salience, default and central-executive network interactions revealed by variational Bayes hidden Markov modeling. PLoS Comput Biol. 2016;12(12): e1005138. doi:10.1371/journal.pcbi.1005138 · DOI
  54. Sherman LE, Rudie JD, Pfeifer JH, Masten CL, McNealy K, Dapretto M. Development of the default mode and central executive networks across early adolescence: a longitudinal study. Dev Cogn Neurosci. 2014;10:148-159. doi:10.1016/j.dcn .2014.08.002 · DOI
  55. Wang W, Zhao Y, Hu X, et al. Conjoint and dissociated structural and functional abnormalities in first-episode drug-naive patients with major depressive disorder: a multimodal meta-analysis. Sci Rep. 2017;7(1):10401. doi:10.1038/s41598-017 -08944-5 · DOI
  56. Schmaal L, Hibar DP, Sämann PG, et al. Cortical abnormalities in adults and adolescents with major depression based on brain scans from 20 cohorts worldwide in the ENIGMA Major Depressive Disorder Working Group. Mol Psychiatry. 2017;22 (6):900-909. doi:10.1038/mp.2016.60 · DOI · Paper page
  57. Insel T, Cuthbert B, Garvey M, et al. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry. 2010;167(7):748-751. doi:10.1176/appi .ajp.2010.09091379 · DOI · Paper page
  58. Dwyer DB, Cabral C, Kambeitz-Ilankovic L, et al. Brain subtyping enhances the neuroanatomical discrimination of schizophrenia [published online February 26, 2018]. Schizophr Bull. doi:10.1093 /schbul/ · DOI

Cited by

Sorted by citations

Authors
Koutsouleris, N., Kambeitz-Ilankovic, L., Ruhrmann, S., Rosen, M., Ruef, A., Dwyer, D. B., Paolini, M., Chisholm, K., Kambeitz, J., Haidl, T., Schmidt, A., Gillam, J., Schultze-Lutter, F., Falkai, P., Reiser, M., Riecher-Rössler, A., Upthegrove, R., Hietala, J., Salokangas, R. K. R., Pantelis, C., Meisenzahl, E., Wood, S. J., Beque, D., Brambilla, P., Borgwardt, S., & for the PRONIA Consortium
Affiliations
  • Ludwig-Maximilians-Universität München (DE)
  • Department of Psychiatry and Psychotherapy, Ludwig-Maximilian-University, Munich, Germany
  • University of Cologne (DE)
  • University of Birmingham (GB)
  • University of Basel (CH)
  • The University of Melbourne (AU)
  • Department of Psychiatry and Psychotherapy, Medical Faculty, Heinrich-Heine University, Düsseldorf, Germany
  • Department of Psychiatry, University Psychiatric Clinic, Psychiatric University Hospital, University of Basel, Basel, Switzerland
  • University of Turku (FI)
  • Heinrich Heine University Düsseldorf (DE)
  • Corporate Global Research, GE Corporation, Munich, Germany
  • University of Milan (IT)
Corresponding author
Koutsouleris, Nikolaos
ORCID
Venue
JAMA Psychiatry
Publisher
American Medical Association (AMA)
Published
2018-11-01
Volume
75
Issue
11
Pages
1156
ISSN
2168-622X (ISSN-L)
DOI
10.1001/jamapsychiatry.2018.2165
PMC
PMC6248111
OpenAlex
W2893799782
Cited by
341
References
20
Citations by year
2026: 32 · 2025: 39 · 2024: 36 · 2023: 39 · 2022: 40 · 2021: 59 · 2020: 51 · 2019: 42 · 2018: 3
FWCI
30.49
Type
journal article
Language
en
Open access
Locations
  • JAMA Psychiatry (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Kölner Universitäts PublikationsServer (Universität zu Köln) (publishedVersion)
  • Research Portal (King's College London) (publishedVersion, other-oa)
  • Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) (submittedVersion, other-oa)
  • edoc (University of Basel) (submittedVersion)
  • Aston Publications Explorer (Aston University) (submittedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
Funders
  • European Commission
  • Medical Research Council (MR/K013599/1)
  • National Institute of Mental Health
Topics
  • Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
  • Health Sciences › Medicine › Psychiatry and Mental health › Bipolar Disorder and Treatment
  • Social Sciences › Psychology › Clinical Psychology › Psychopathy, Forensic Psychiatry, Sexual Offending
Keywords
clinical high risk for psychosis, recent-onset depression, functional outcome prediction, social functioning, role functioning, machine learning prognostic models, structural neuroimaging, gray matter volume, medial prefrontal cortex, cerebellum, dorsolateral prefrontal cortex, risk stratification, early psychosis
MeSH terms
Machine Learning, Adult, Depression, Depressive Disorder, Female, Humans, Male, Neuropsychological Tests, Psychotic Disorders, Social Adjustment, Case-Control Studies, Young Adult, Neuroimaging, Gray Matter

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review