Automatic Detection of Incoherent Speech for Diagnosing Schizophrenia
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Schizophrenia is a mental disorder which afflicts an estimated 0.7% of adults worldwide (Saha et al., 2005).It affects many areas of mental function, often evident from incoherent speech.Diagnosing schizophrenia relies on subjective judgments resulting in disagreements even among trained clinicians.Recent studies have proposed the use of natural language processing for diagnosis by drawing on automatically-extracted linguistic features, and particularly the use of discourse coherence.Here, we present the first benchmark comparison of previously proposed coherence models for detecting symptoms of schizophrenia and evaluate their performance on a new dataset of recorded interviews between subjects and clinicians.We also present two improved coherence metrics based on modern sentence embedding techniques that outperform the previous methods on our dataset.Finally, we propose a novel computational model for reference incoherence based on ambiguous pronoun usage and show that it is a highly predictive feature on our data.While the number of subjects is limited in this pilot study, our results suggest new directions for diagnosing common symptoms of schizophrenia.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Computational analysis of psychotherapy is encouraging researchers and companies to identify patterns of patient symptomology and therapist intervention (36, 37).”
From Key Considerations for Incorporating Conversational AI in Psychotherapy (2019) · cites this paper as [37]
Experimental — accuracy not yet measuredIn the citing paper
“Previous work has shown that ASR accuracy is reduced for both elderly patients and those with dementia [99], [100], [101].”
From A Review of Automated Speech and Language Features for Assessment of Cognitive and Thought Disorders (2020) · cites this paper as [99]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Second, we draw on recent advances in word embeddings (Mikolov et al., 2013; Pennington et al., 2014) and sentence embeddings (Arora et al., 2016; Pagliardini et al., 2018).”
MethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×
“Various other computational models have shown features of text and speech that can be automatically extracted and are associated with schizophrenia, including lexical features drawn from lexicons (Hong et al., 2012, 2015; Mitchell et al., 2015) and acoustic features (Covington et al., 2012).”
BackgroundHalliday, M. A. K., & Hasan, R. (2014) · Routledge eBooks · doi:10.4324/9781315836010 · Cited by 6,082
Rhetorical Structure Theory: Toward a functional theory of text organization
MANN, W. C., & THOMPSON, S. A. (1988) · Text - Interdisciplinary Journal for the Study of Discourse · doi:10.1515/text.1.1988.8.3.243 · Cited by 3,853 · cited 2×
“Relational models like rhetorical structure theory define discourse relations that hierarchically structure texts (Mann and Thompson, 1988; Lascaridest and Asher, 1991).”
BackgroundAttention, intentions, and the structure of discourse
Grosz, B. J., & Sidner, C. L. (1986) · doi:10.5555/12457.12458 · Cited by 2,580
A Systematic Review of the Prevalence of Schizophrenia
Saha, S., Chant, D., Welham, J., & McGrath, J. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020141 · Cited by 2,168
(2017) · International Conference on Learning Representations · Cited by 1,075
Lexical cohesion computed by thesaural relations as an indicator of the structure of text
Morris, J., & Hirst, G. (1991) · OLAC - Open Language Archives Community · doi:10.5555/971738.971740 · Cited by 950
End-to-end Neural Coreference Resolution
Lee, K., He, L., Lewis, M., & Zettlemoyer, L. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1018 · Cited by 869 · cited 1×
“Co-references are extracted from the corpus with a pretrained co-reference resolver (Lee et al., 2017).”
MethodThe measurement of textual coherence with latent semantic analysis
Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998) · Discourse Processes · doi:10.1080/01638539809545029 · Cited by 749 · cited 2×
“From the beginning it was clear that this method relied on simplifying assumptions that might be inappropriate for schizophrenia; for example Foltz et al. (1998) notes that a discourse that simply repeated a sentence would be judged as highly coherent, problematic since repetition or perseveration…”
MethodUnsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features
Pagliardini, M., Gupta, P., & Jaggi, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1049 · Cited by 678 · cited 5×
“Sent2Vec learns a new word embedding similar to Word2Vec but extends the training objective such that each sentence embedding is predictive of the sentences around it.”
MethodInfluentialAutomated analysis of free speech predicts psychosis onset in high-risk youths
Bedi, G., Carrillo, F., Cecchi, G. A., Slezak, D. F., Sigman, M., Mota, N. B., Ribeiro, S., Javitt, D. C., Copelli, M., & Corcoran, C. M. (2015) · npj Schizophrenia · doi:10.1038/npjschz.2015.30 · Cited by 673 · cited 12×
“What we will call the Incoherence Model (Bedi et al., 2015) measures the coherence of a speaker by computing the semantic coherence of each ad-”
BackgroundMethodInfluentialModeling Local Coherence: An Entity-Based Approach
Barzilay, R., & Lapata, M. (2008) · Computational Linguistics · doi:10.1162/coli.2008.34.1.1 · Cited by 671 · cited 1×
“Referen-tial coherence focuses on the coherence of entities moving in and out of focus across a text (Grosz and Sidner, 1986; Barzilay and Lapata, 2008).”
BackgroundQuantifying incoherence in speech: An automated methodology and novel application to schizophrenia
Elvevåg, B., Foltz, P. W., Weinberger, D. R., & Goldberg, T. E. (2007) · Schizophrenia Research · doi:10.1016/j.schres.2007.03.001 · Cited by 363 · cited 16×
“What we will call the Tangentiality Model (Elvevåg et al., 2007) uses the coherence metric to compare fixed-sized word windows of responses to their corresponding questions.”
BackgroundMethodInfluentialAdler, C. M., Malhotra, A. K., Elman, I., Goldberg, T., Egan, M., Pickar, D., & Breier, A. (1999) · American Journal of Psychiatry · doi:10.1176/ajp.156.10.1646 · Cited by 337 · cited 13×
“Symptoms of FTD can manifest as speech irregularities, generally perceived as a lack of coherence.”
BackgroundInfluentialCoh-Metrix: Capturing Linguistic Features of Cohesion
McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010) · Discourse Processes · doi:10.1080/01638530902959943 · Cited by 309 · cited 1×
“…applies SVD (Singular value decomposition) to a matrix of word-document co-occurrences, and was applied early on as a model of discourse coherence, using cosines between embeddings for text regions as a measure of concept overlap or lexical cohesion (Foltz et al., 1998; McNamara et al., 2010).”
MethodAutomatic evaluation of text coherence: models and representations
Lapata, M., & Barzilay, R. (2005) · Cited by 218
Quantifying the Language of Schizophrenia in Social Media
Mitchell, M., Hollingshead, K., & Coppersmith, G. (2015) · Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality · doi:10.3115/v1/w15-1202 · Cited by 172 · cited 2×
“Various other computational models have shown features of text and speech that can be automatically extracted and are associated with schizophrenia, including lexical features drawn from lexicons (Hong et al., 2012, 2015; Mitchell et al., 2015) and acoustic features (Covington et al., 2012).”
BackgroundDiscourse relations and defeasible knowledge
Lascarides, A., & Asher, N. M. (1991) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981344.981352 · Cited by 153 · cited 1×
“Relational models like rhetorical structure theory define discourse relations that hierarchically structure texts (Mann and Thompson, 1988; Lascaridest and Asher, 1991).”
BackgroundElvevåg, B., Foltz, P. W., Rosenstein, M., & DeLisi, L. E. (2010) · Journal of Neurolinguistics · doi:10.1016/j.jneuroling.2009.05.002 · Cited by 133 · cited 1×
“These proof-of-concept studies proposed using a coherence measure based on Latent Semantic Analysis (LSA) to quantitatively measure the presence or onset of FTD in subjects (Bedi et al., 2015; Elvev˚ag et al., 2007, 2010).”
MethodThe linguistics of schizophrenia: thought disturbance as language pathology across positive symptoms
Hinzen, W. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00971 · Cited by 131 · cited 3×
“FTD has been reported to coincide with anomalies in de-ictic noun phrase usage, including various unusual uses of pronouns (Hinzen and Rossell´o, 2015).”
BackgroundMethodHong, K., Nenkova, A., March, M. E., Parker, A. P., Verma, R., & Kohler, C. G. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.10.002 · Cited by 64 · cited 6×
“Since repetition itself is a symptom of schizophrenia (Hong et al., 2015), we need a more powerful model of what constitutes abnormal repetition as opposed to nat-ural lexical cohesion, presumably a model that will need to draw on other linguistic markers.”
BackgroundMethodInfluentialCovington, M. A., Lunden, S. L. A., Cristofaro, S. L., Wan, C. R., Bailey, C., Broussard, B., Fogarty, R., Johnson, S., Zhang, S., & Compton, M. T. (2012) · Schizophrenia Research · doi:10.1016/j.schres.2012.10.005 · Cited by 57 · cited 1×
“Various other computational models have shown features of text and speech that can be automatically extracted and are associated with schizophrenia, including lexical features drawn from lexicons (Hong et al., 2012, 2015; Mitchell et al., 2015) and acoustic features (Covington et al., 2012).”
BackgroundAmbiguous pronoun use in narratives of children with Autism Spectrum Disorders
Novogrodsky, R., & Edelson, L. R. (2015) · Child Language Teaching and Therapy · doi:10.1177/0265659015602935 · Cited by 53
Lintean, M. C., Moldovan, C., Rus, V., & McNamara, D. S. (2010) · The Florida AI Research Society · Cited by 40
Lexical Differences in Autobiographical Narratives from Schizophrenic Patients and Healthy Controls
Hong, K., Kohler, C. G., March, M. E., Parker, A. P., & Nenkova, A. (2013) · Cited by 17
Research works printed but not linked: 2
- Nancy C Andreasen. 1979. Thought, language, and communication disorders: Ii. diagnostic significance. Archives of general Psychiatry, 36(12):1325-1330.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119.
Cited by
Detecting formal thought disorder by deep contextualized word representations
Sarzynska-Wawer, J., Wawer, A., Pawlak, A., Szymanowska, J., Stefaniak, I., Jarkiewicz, M., & Okruszek, L. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.114135 · Cited by 335
Key Considerations for Incorporating Conversational AI in Psychotherapy
Miner, A. S., Shah, N., Bullock, K. D., Arnow, B. A., Bailenson, J., & Hancock, J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00746 · Cited by 166
“Computational analysis of psychotherapy is encouraging researchers and companies to identify patterns of patient symptomology and therapist intervention (36, 37).”
BackgroundTang, S. X., Kriz, R., Cho, S., Park, S. J., Harowitz, J., Gur, R. E., Bhati, M. T., Wolf, D. H., Sedoc, J., & Liberman, M. (2021) · Schizophrenia · doi:10.1038/s41537-021-00154-3 · Cited by 172
Parola, A., Lin, J. M., Simonsen, A., Bliksted, V., Zhou, Y., Wang, H., Inoue, L., Koelkebeck, K., & Fusaroli, R. (2022) · Schizophrenia Research · doi:10.1016/j.schres.2022.07.002 · Cited by 121
Grodniewicz, J. P., & Hohol, M. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1190084 · Cited by 100
Metadata
- Authors
- Iter, D., Yoon, J., & Jurafsky, D.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic
- Published
- 2018
- Pages
- 136-146
- OpenAlex
- W2806168354
- Cited by
- 115
- References
- 26
- Citations by year
- 2026: 9 · 2025: 10 · 2024: 17 · 2023: 17 · 2022: 24 · 2021: 14 · 2020: 13 · 2019: 11
- FWCI
- 17.03
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Computational Linguistics and Clinical Psychology (CLPsych) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1514268)
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Keywords
- schizophrenia diagnosis, discourse coherence, coherence measures, sentence embeddings, ambiguous pronouns, clinical interview, benchmark comparison, computational psychiatry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.