CoChat
Sign Up
Cited by 19Emotion and Mood RecognitionMental Health Research TopicsDigital Mental Health Interventions

The power of speech in the wild: discriminative power of daily voice diaries in understanding auditory verbal hallucinations using deep learning

Open access
Wang, Weichen, Xu, Weizhe, Chander, Ayesha, Nepal, Subigya, Buck, Benjamin, Pakhomov, Serguei, Cohen, Trevor, Ben-Zeev, Dror +1 more·2023·Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
DOI 10.1145/3610890
Open this paper in CoChat — free accountOpen PDF

Abstract

Mobile phone sensing is increasingly being used in clinical research studies to assess a variety of mental health conditions (e.g., depression, psychosis). However, in-the-wild speech analysis -- beyond conversation detecting -- is a missing component of these mobile sensing platforms and studies. We augment an existing mobile sensing platform with a daily voice diary to assess and predict the severity of auditory verbal hallucinations (i.e., hearing sounds or voices in the absence of any speaker), a condition that affects people with and without psychiatric or neurological diagnoses. We collect 4809 audio diaries from N=384 subjects over a one-month-long study period. We investigate the performance of various deep-learning architectures using different combinations of sensor behavioral streams (e.g., voice, sleep, mobility, phone usage, etc.) and show the discriminative power of solely using audio recordings of speech as well as automatically generated transcripts of the recordings; specifically, our deep learning model achieves a weighted f-1 score of 0.78 solely from daily voice diaries. Our results surprisingly indicate that a simple periodic voice diary combined with deep learning is sufficient enough of a signal to assess complex psychiatric symptoms (e.g., auditory verbal hallucinations) collected from people in the wild as they go about their daily lives.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3610890

Cites

82 of 82 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. Daniel A Adler, Dror Ben-Zeev, Vincent WS Tseng, John M Kane, Rachel Brian, Andrew T Campbell, Marta Hauser, Emily A Scherer, and Tanzeem Choudhury. 2020. Predicting early warning signs of psychotic relapse from passive sensing data: an approach using encoder-decoder neural networks. JMIR mHealth and uHealth 8, 8 (2020), e19962.
  2. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning. PMLR, 173-182.
  3. Nancy C Andreasen and Michael Flaum. 1991. Schizophrenia: the characteristic symptoms. Schizophrenia bulletin 17, 1 (1991), 27-49.
  4. Nancy C Andreasen and William M Grove. 1986. Thought, language, and communication in schizophrenia: diagnosis and prognosis. Schizophrenia bulletin 12, 3 (1986), 348-359.
  5. Min Hane Aung, Mark Matthews, and Tanzeem Choudhury. 2017. Sensing behavioral symptoms of mental health and delivering personalized interventions using mobile technologies. Depression and anxiety 34, 7 (2017), 603-609.
  6. Dror Ben-Zeev, Rachel Brian, Rui Wang, Weichen Wang, Andrew T Campbell, Min SH Aung, Michael Merrill, Vincent WS Tseng, Tanzeem Choudhury, Marta Hauser, et al. 2017. CrossCheck: Integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatric rehabilitation journal 40, 3 (2017), 266.
  7. Dror Ben-Zeev, Emily A Scherer, Rui Wang, Haiyi Xie, and Andrew T Campbell. 2015. Next-generation psychiatric assessment: Using smartphone sensors to monitor behavior and mental health. Psychiatric rehabilitation journal 38, 3 (2015), 218.
  8. Dror Ben-Zeev, Rui Wang, Saeed Abdullah, Rachel Brian, Emily A Scherer, Lisa A Mistler, Marta Hauser, John M Kane, Andrew Campbell, and Tanzeem Choudhury. 2016. Mobile behavioral sensing for outpatients and inpatients with schizophrenia. Psychiatric services 67, 5 (2016), 558-561.
  9. Josef Bless, Runar Smelror, Ingrid Agartz, and Kenneth Hugdahl. 2017. SA110. Using a Smartphone App to Assess Auditory Hallucinations in Adolescent Schizophrenia: Is This the Way to go for Better Control Over Voices? Schizophrenia bulletin 43, Suppl 1 (2017), S152.
  10. Vera Brink, Catheleine van Driel, Saliha El Bouhaddani, Klaas J Wardenaar, Lieke van Domburgh, Barbara Schaefer, Marije van Beilen, Agna A Bartels-Velthuis, and Wim Veling. 2020. Spontaneous discontinuation of distressing auditory verbal hallucinations in a school-based sample of adolescents: a longitudinal study. European child & adolescent psychiatry 29 (2020), 777-790.
  11. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014). · arXiv
  12. Michael A Cohn, Matthias R Mehl, and James W Pennebaker. 2004. Linguistic markers of psychological change surrounding September 11, 2001. Psychological science 15, 10 (2004), 687-693.
  13. Saskia de Leede-Smith and Emma Barkus. 2013. A comprehensive review of auditory verbal hallucinations: lifetime prevalence, correlates and mechanisms in healthy and clinical individuals. Frontiers in human neuroscience 7 (2013), 367.
  14. Philippe Delespaul, Marten devries, and Jim van Os. 2002. Determinants of occurrence and recovery from hallucinations in daily life. Social psychiatry and psychiatric epidemiology 37, 3 (2002), 97-104.
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). · arXiv
  16. Daniel Freeman and Philippa A Garety. 2003. Connecting neurosis and psychosis: the direct influence of emotion on delusions and hallucinations. Behaviour research and therapy 41, 8 (2003), 923-947.
  17. Kelvin MT Fung, Hector WH Tsang, and Patrick W Corrigan. 2008. Self-stigma of people with schizophrenia as predictor of their adherence to psychosocial treatment. Psychiatric rehabilitation journal 32, 2 (2008), 95.
  18. Gillian Haddock, J McCarron, N Tarrier, and EB Faragher. 1999. Scales to measure dimensions of hallucinations and delusions: the psychotic symptom rating scales (PSYRATS). Psychological medicine 29, 4 (1999), 879-889.
  19. RE Hoffman, M Varanko, J Gilmore, and AL Mishara. 2008. Experiential features used by patients with schizophrenia to differentiate 'voices' from ordinary verbal thought. Psychological medicine 38, 8 (2008), 1167-1176.
  20. Ralph E Hoffman. 2007. A social deafferentation hypothesis for induction of active schizophrenia. Schizophrenia bulletin 33, 5 (2007), 1066-1070.
  21. Daniel C Javitt. 2009. When doors of perception close: bottom-up models of disrupted cognition in schizophrenia. Annual review of clinical psychology 5 (2009), 249-275.
  22. Ewa Kacewicz, James W Pennebaker, Matthew Davis, Moongee Jeon, and Arthur C Graesser. 2013. Pronoun use reflects standings in social hierarchies. Journal of Language and Social Psychology (2013), 0261927X13502654.
  23. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv
  24. Frank Larøi, Iris E Sommer, Jan Dirk Blom, Charles Fernyhough, Dominic H Ffytche, Kenneth Hugdahl, Louise C Johns, Simon McCarthy-Jones, Antonio Preti, Andrea Raballo, et al. 2012. The characteristic features of auditory verbal hallucinations in clinical and nonclinical groups: state-of-the-art overview and future directions. Schizophrenia bulletin 38, 4 (2012), 724-733.
  25. Belinda R Lennox, S Bert, G Park, Peter B Jones, and Peter G Morris. 1999. Spatial and temporal mapping of neural activity associated with auditory hallucinations. The Lancet 353, 9153 (1999), 644.
  26. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130 (2017). · arXiv
  27. Hong Lu, Denise Frauendorfer, Mashfiqui Rabbi, Marianne Schmid Mast, Gokul T Chittaranjan, Andrew T Campbell, Daniel Gatica- Perez, and Tanzeem Choudhury. 2012. Stresssense: Detecting stress in unconstrained acoustic environments using smartphones. In Proceedings of the 2012 ACM conference on ubiquitous computing. 351-360.
  28. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765-4774. http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
  29. John McGrath, Sukanta Saha, Joy Welham, Ossama El Saadi, Clare MacCauley, and David Chant. 2004. A systematic review of the incidence of schizophrenia: the distribution of rates and the influence of sex, urbanicity, migrant status and methodology. BMC medicine 2, 1 (2004), 1-22.
  30. Neil M McLachlan, Dougal S Phillips, Susan L Rossell, and Sarah J Wilson. 2013. Auditory processing and hallucinations in schizophrenia. Schizophrenia research 150, 2-3 (2013), 380-385.
  31. David C Mohr, Mi Zhang, and Stephen M Schueller. 2017. Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annual review of clinical psychology 13 (2017), 23-47.
  32. Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards. 2003. Lying words: Predicting deception from linguistic styles. Personality and social psychology bulletin 29, 5 (2003), 665-675.
  33. James W Pennebaker, Matthias R Mehl, and Kate G Niederhoffer. 2003. Psychological aspects of natural language use: Our words, our selves. Annual review of psychology 54, 1 (2003), 547-577.
  34. Viliam Rapcan, Shona D'Arcy, Sherlyn Yeap, Natasha Afzal, Jogin Thakore, and Richard B Reilly. 2010. Acoustic and temporal analysis of speech: A potential biomarker for schizophrenia. Medical engineering & physics 32, 9 (2010), 1074-1079.
  35. Benjamin Rolland, Ali Amad, Emmanuel Poulet, Régis Bordet, Alexandre Vignaud, Rémy Bation, Christine Delmaire, Pierre Thomas, Olivier Cottencin, and Renaud Jardri. 2015. Resting-state functional connectivity of the nucleus accumbens in auditory and visual hallucinations in schizophrenia. Schizophrenia bulletin 41, 1 (2015), 291-299.
  36. Sohrab Saeb, Emily G Lattie, Konrad P Kording, and David C Mohr. 2017. Mobile phone detection of semantic location and its relationship to depression and anxiety. JMIR mHealth and uHealth 5, 8 (2017), e7297.
  37. Shekhar Saxena, Graham Thornicroft, Martin Knapp, and Harvey Whiteford. 2007. Resources for mental health: scarcity, inequity, and inefficiency. The lancet 370, 9590 (2007), 878-889.
  38. Iris EC Sommer, Kirstin Daalman, Thomas Rietkerk, Kelly M Diederen, Steven Bakker, Jaap Wijkstra, and Marco PM Boks. 2010. Healthy individuals with auditory verbal hallucinations; who are they? Psychiatric assessments of a selected sample of 103 subjects. Schizophrenia bulletin 36, 3 (2010), 633-641.
  39. M Stephane, S Barton, and NN Boutros. 2001. Auditory verbal hallucinations and dysfunction of the neural substrates of speech. Schizophrenia research 50, 1-2 (2001), 61-78.
  40. Rael D Strous, Nelson Cowan, Walter Ritter, and Daniel C Javitt. 1995. Auditory sensory (" echoic") memory dysfunction in schizophrenia. The American journal of psychiatry (1995).
  41. Ryan J Van Lieshout and Joel O Goldberg. 2007. Quantifying self-reports of auditory verbal hallucinations in persons with psychosis. Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement 39, 1 (2007), 73.
  42. Flavie Waters, Daniel Collerton, Dominic H Ffytche, Renaud Jardri, Delphine Pins, Robert Dudley, Jan Dirk Blom, Urs Peter Mosimann, Frank Eperjesi, Stephen Ford, et al. 2014. Visual hallucinations in the psychosis spectrum and comparative information from neurodegenerative disorders and eye disease. Schizophrenia bulletin 40, Suppl_4 (2014), S233-S245.
  43. Weizhe Xu, Weichen Wang, Jake Portanova, Ayesha Chander, Andrew Campbell, Serguei Pakhomov, Dror Ben-Zeev, and Trevor Cohen. 2022. Fully Automated Detection of Formal Thought Disorder with Time-series Augmented Representations for Detection of Incoherent Speech (TARDIS). Journal of Biomedical Informatics (2022), 103998.

Cited by

Sorted by citations

Related

Authors
Wang, W., Xu, W., Chander, A., Nepal, S., Buck, B., Pakhomov, S., Cohen, T., Ben-Zeev, D., & Campbell, A.
Affiliations
  • Dartmouth College (US)
  • University of Washington (US)
  • University of Minnesota (US)
  • University of Washington, Department of Psychiatry and Behavioral Sciences, Seattle, WA, USA
ORCID
Venue
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Publisher
Association for Computing Machinery (ACM)
Published
2023-09-27
Volume
7
Issue
3
Pages
1-29
ISSN
2474-9567 (ISSN-L)
DOI
10.1145/3610890
OpenAlex
W4387086487
Cited by
19
References
82
Citations by year
2026: 7 · 2025: 7 · 2024: 3 · 2023: 2
FWCI
4.91
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion)
Funders
National Institute of Mental Health (R01MH112641)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
smartphone sensing, auditory verbal hallucinations, automatic speech transcription, psychiatric symptom assessment, deep learning, multimodal sensor data, mental health monitoring, psychosis, depression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review