CoChat

Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies

Open access
Chandler, Chelsea, Foltz, Peter W, Elvevåg, Brita · 2022 · Schizophrenia Bulletin
Cited by 46

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Abstract

Objectives: Machine learning (ML) and natural language processing have great potential to improve efficiency and accuracy in diagnosis, treatment recommendations, predictive interventions, and scarce resource allocation within psychiatry. Researchers often conceptualize such an approach as operating in isolation without much need for human involvement, yet it remains crucial to harness human-in-the-loop practices when developing and implementing such techniques as their absence may be catastrophic. We advocate for building ML-based technologies that collaborate with experts within psychiatry in all stages of implementation and use to increase model performance while simultaneously increasing the practicality, robustness, and reliability of the process. Methods: We showcase pitfalls of the traditional ML framework and explain how it can be improved with human-in-the-loop techniques. Specifically, we applied active learning strategies to the automatic scoring of a story recall task and compared the results to a traditional approach. Results: Human-in-the-loop methodologies supplied a greater understanding of where the model was least confident or had knowledge gaps during training. As compared to the traditional framework, less than half of the training data were needed to reach a given accuracy. Conclusions: Human-in-the-loop ML is an approach to data collection and model creation that harnesses active learning to select the most critical data needed to increase a model’s accuracy and generalizability more efficiently than classic random sampling would otherwise allow. Such techniques may additionally operate as safeguards from spurious predictions and can aid in decreasing disparities that artificial intelligence systems otherwise propagate.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/schbul/sbac038

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 29

  1. Corcoran, 2018, Prediction of psychosis across protocols and risk cohorts using automated language analysis, World Psychiatry, 17, 67, 10.1002/wps.20491 · DOI · Paper page
  2. Voppel, 2020, S136. Classifying schizophrenia using phonological, semantic and syntactic features of language: a combinatory machine learning approach, Schizophr Bull., 46, S87, 10.1093/schbul/sbaa031.202 · doi:10.1093/schbul/sbaa031.202
  3. Rezaii, 2019, machine learning approach to predicting psychosis using semantic density and latent content analysis, NPJ Schizophr., 5, 9, 10.1038/s41537-019-0077-9 · DOI · Paper page
  4. Tang, 2021, Natural language processing methods are sensitive to sub-clinical linguistic differences in schizophrenia spectrum disorders, NPJ Schizophr., 7, 25, 10.1038/s41537-021-00154-3 · DOI
  5. Cohen, 2019, Ambulatory vocal acoustics, temporal dynamics, and serious mental illness, J Abnorm Psychol., 128, 97, 10.1037/abn0000397 · DOI
  6. Chandler, 2020, Machine learning for ambulatory applications of neuropsychological testing, Intelligence-Based Med., 1-2, 100006, 10.1016/j.ibmed.2020.100006 · DOI
  7. Cohen, 2016, Vocal expression in schizophrenia: less than meets the ear, J Abnorm Psychol., 125, 299, 10.1037/abn0000136 · DOI
  8. Cohen, 2016, A psychometric investigation of “macroscopic” speech measures for clinical and psychological science, Behav Res Methods., 48, 475, 10.3758/s13428-015-0584-1 · DOI
  9. Cheng, 2018, Modeling Self-Reported and Observed Affect from Speech. Proc, Interspeech, 365, 3
  10. Chandler, 2020, Predicting self-reported affect from speech acoustics and language., 9
  11. Pauselli, 2018, Computational linguistic analysis applied to a semantic fluency task to measure derailment and tangentiality in schizophrenia, Psychiatry Res., 263, 74, 10.1016/j.psychres.2018.02.037 · DOI
  12. Ku, 2021, Computational linguistic analysis applied to a semantic fluency task: A replication among first-episode psychosis patients with and without derailment and tangentiality, Psychiatry Res., 304, 114105, 10.1016/j.psychres.2021.114105 · DOI
  13. Sarzynska-Wawer, 2021, Detecting formal thought disorder by deep contextualized word representations, Psychiatry Res., 304, 114135, 10.1016/j.psychres.2021.114135 · DOI · Paper page
  14. Kim, 2002, Clinical psychologists’ theory-based representations of mental disorders predict their diagnostic reasoning and memory, J Exp Psychol Gen, 131, 451, 10.1037/0096-3445.131.4.451 · doi:10.1037/0096-3445.131.4.451 · Paper page
  15. Grzenda, 2021, Evaluating the machine learning literature: a primer and user’s guide for psychiatrists, Am J Psychiatry, 178, 715, 10.1176/appi.ajp.2020.20030250 · DOI
  16. Hitczenko, 2021, Racial and ethnic biases in computational approaches to psychopathology, Schizophr Bull.
  17. Oliva, 2021, Dosing Discrimination: Regulating PDMP Risk Scores
  18. Chandler, 2019, Overcoming the bottleneck in traditional assessments of verbal memory: Modeling human ratings and classifying clinical group membership., 137, 10.18653/v1/W19-3016 · DOI
  19. Holmlund, 2020, Applying speech technologies to assess verbal memory in patients with serious mental illness, NPJ Digital Med., 3, 33, 10.1038/s41746-020-0241-7 · DOI
  20. Foltz, 2016, Detecting clinically significant events through automated language analysis: Quo imus?, NPJ Schizophr., 2, 15054, 10.1038/npjschz.2015.54 · DOI
  21. Monarch, 2021, Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-Centered AI
  22. Hoffman, 1986, A comparative study of manic vs. schizophrenic speech disorganization, Arch Gen Psychiatry., 43, 831, 10.1001/archpsyc.1986.01800090017003 · DOI · Paper page
  23. Dagan, 1995, Committee-based sampling for training probabilistic classifiers., 150
  24. Siddhant, 2904, Deep Bayesian active learning for natural language processing: results of a large-scale empirical study. In, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018
  25. McCallum, 1998, Employing EM and Pool-Based Active Learning for Text Classification., 350
  26. Henrich, 2010, The weirdest people in the world?, Behavioral and Brain Sciences., 33, 61, 10.1017/S0140525X0999152X · DOI · Paper page
  27. Wechsler, 1997, Wechsler Memory Scale - Third Edition, WMS-III: Administration and scoring manual, The Psychological Corporation
  28. Ribeiro, 2016, “Why Should I Trust You?”: Explaining the Predictions of Any Classifier., 1135, 10.1145/2939672.2939778 · DOI · Paper page
  29. Cohen, 2022, Natural language processing and psychosis: on the need for comprehensive psychometric evaluation, Schizophr Bull., 48, 939, 10.1093/schbul/sbac051 · DOI

Cited by

See all 46

Metadata

Authors
Chandler, C., Foltz, P. W., & Elvevåg, B.
Venue
Schizophrenia Bulletin
Published
2022-05-26
Volume
48
Issue
5
Pages
949-957
ISSN
  • 0586-7614 (ISSN-L)
  • 1745-1701
OpenAlex
W4226086425
Cited by
46
References
13
FWCI
1.59
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies. https://app.cochat.ai/papers/10.1093/schbul/sbac038
The paper
Chandler, C., Foltz, P. W., & Elvevåg, B. (2022). Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies. Schizophrenia Bulletin. https://doi.org/10.1093/schbul/sbac038

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.