CoChat
Sign Up
Cited by 87Suicide and Self-Harm StudiesDigital Mental Health InterventionsMental Health via Writing

Testing Suicide Risk Prediction Algorithms Using Phone Measurements With Patients in Acute Mental Health Settings: Feasibility Study

Open access
Haines-Delmont, Alina, Chahal, Gurdit, Bruen, Ashley Jane, Wall, Abbie, Khan, Christina Tara, Sadashiv, Ramesh, Fearnley, David·2020·JMIR mhealth and uhealth
DOI 10.2196/15901PMC PMC7380988

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Digital phenotyping and machine learning are currently being used to augment or even replace traditional analytic procedures in many domains, including health care. Given the heavy reliance on smartphones and mobile devices around the world, this readily available source of data is an important and highly underutilized source that has the potential to improve mental health risk prediction and prevention and advance mental health globally. OBJECTIVE: This study aimed to apply machine learning in an acute mental health setting for suicide risk prediction. This study uses a nascent approach, adding to existing knowledge by using data collected through a smartphone in place of clinical data, which have typically been collected from health care records. METHODS: We created a smartphone app called Strength Within Me, which was linked to Fitbit, Apple Health kit, and Facebook, to collect salient clinical information such as sleep behavior and mood, step frequency and count, and engagement patterns with the phone from a cohort of inpatients with acute mental health (n=66). In addition, clinical research interviews were used to assess mood, sleep, and suicide risk. Multiple machine learning algorithms were tested to determine the best fit. RESULTS: K-nearest neighbors (KNN; k=2) with uniform weighting and the Euclidean distance metric emerged as the most promising algorithm, with 68% mean accuracy (averaged over 10,000 simulations of splitting the training and testing data via 10-fold cross-validation) and an average area under the curve of 0.65. We applied a combined 5×2 F test to test the model performance of KNN against the baseline classifier that guesses training majority, random forest, support vector machine and logistic regression, and achieved F statistics of 10.7 (P=.009) and 17.6 (P=.003) for training majority and random forest, respectively, rejecting the null of performance being the same. Therefore, we have taken the first steps in prototyping a system that could continuously and accurately assess the risk of suicide via mobile devices. CONCLUSIONS: Predicting for suicidality is an underaddressed area of research to which this paper makes a useful contribution. This is part of the first generation of studies to suggest that it is feasible to utilize smartphone-generated user input and passive sensor data to generate a risk algorithm among inpatients at suicide risk. The model reveals fair concordance between phone-derived and research-generated clinical data, and with iterative development, it has the potential for accurate discriminant risk prediction. However, although full automation and independence of clinical judgment or input would be a worthy development for those individuals who are less likely to access specialist mental health services, and for providing a timely response in a crisis situation, the ethical and legal implications of such advances in the field of psychiatry need to be acknowledged.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/15901

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The most common use of artificial intelligence in the mental health apps was prediction (of mood, stress, and risk) (8/17 studies).

    “The most common uses of artificial intelligence in the mental health apps were prediction (of mood, stress, and risk) (8/17 studies) [31–37,41]”

    From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [41]

  • One study aimed to identify suicide risk in inpatients.

    “one aimed to identify suicide risk in inpatients [41]”

    From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [41]

  • Only one of the studies used a dimension reduction technique to reduce the number of features in order to aid further classification and improve classification reliability.

    “Only one of the studies used a dimension reduction technique to reduce the number of features (e.g. sleep data, journal entries, mood) in order to aid further classification and improve classification reliability [41].”

    From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [41]

  • Random Forest classifier was used for feature selection, risk-stratification of diabetic patients based on symptoms of depression, inferring context for depression level prediction, and predicting mood in the included studies.

    “It is used for feature selection [41], risk-stratification of diabetic patients based on symptoms of depression [31], inferring context for depression level prediction [44], and predicting mood [34,35].”

    From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [41]

  • All of the studies reviewed used data collected via mobile devices to inform AI models-for natural language analysis and output, risk stratification and prediction or for personalising the user's app experience.

    “All of the studies reviewed used data collected via mobile devices to inform AI models—for natural language analysis and output [29,30,38,40,42,43], risk stratification and prediction [31,41] or for personalising the user’s app experience [28,44].”

    From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [41]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. [43]Kohavi R A Study of Cross-validation and Bootstrap for Accuracy Estimation and Model Selection Proceedings of the 14th International Joint Conference on Artificial intelligence 1995 IJCAI'95 August 20-25, 1995 Montreal, Quebec, Canada
  2. [44]Hastie T Tibshirani R Friedman J The Elements of Statistical Learning: Data Mining, Inference, and Prediction Second Edition 2017 New York, USA Springer
  3. [47]Louppe G Wehenkel L Sutera A Geurts P Understanding Variable Importances in Forests of Randomized Trees Proceedings of the 26th International Conference on Neural Information Processing Systems 2013 NIPS'13 December 5-13, 2013 Lake Tahoe, Nevada
  4. [70]Sordo M Zeng Q Oliveira JL Maojo V Martín-Sánchez F Pereira AS On sample size and classification accuracy: a performance comparison Biological and Medical Data Analysis: 6th International Symposium, ISBMDA 2005, Aveiro, Portugal, November 10-11, 2005, Proceedings (Lecture Notes in Computer Science (3745)) 2005 Berlin, Heidelberg Springer

Cited by

Sorted by citations

Related

Authors
Haines-Delmont, A., Chahal, G., Bruen, A. J., Wall, A., Khan, C. T., Sadashiv, R., & Fearnley, D.
Affiliations
  • University of Liverpool (GB)
  • Stanford Medicine (US)
ORCID
Venue
JMIR mhealth and uhealth
Publisher
JMIR Publications Inc.
Published
2020-06-26
Volume
8
Issue
6
Pages
e15901
ISSN
2291-5222 (ISSN-L)
DOI
10.2196/15901
PMC
PMC7380988
OpenAlex
W3007150960
Cited by
87
References
72
Citations by year
2026: 8 · 2025: 13 · 2024: 11 · 2023: 9 · 2022: 38 · 2021: 6 · 2020: 2
FWCI
5.49
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • e-space (Manchester Metropolitan University) (acceptedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Topics
  • Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
Keywords
suicide risk prediction, smartphone data, digital phenotyping, k-nearest neighbors, machine learning classifier comparison, sleep behavior, mood, physical activity, suicide prevention
MeSH terms
Machine Learning, Suicide Prevention, Algorithms, Feasibility Studies, Humans, Mental Health

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review