Deep sequential models for suicidal ideation from multiple source data
Open accessAbstract
This paper presents a novel method for predicting suicidal ideation from electronic health records (EHR) and ecological momentary assessment (EMA) data using deep sequential models. Both EHR longitudinal data and EMA question forms are defined by asynchronous, variable length, randomly sampled data sequences. In our method, we model each of them with a recurrent neural network, and both sequences are aligned by concatenating the hidden state of each of them using temporal marks. Furthermore, we incorporate attention schemes to improve performance in long sequences and time-independent pre-trained schemes to cope with very short sequences. Using a database of 1023 patients, our experimental results show that the addition of EMA records boosts the system recall to predict the suicidal ideation diagnosis from 48.13% obtained exclusively from EHR-based state-of-the-art methods to 67.78%. Additionally, our method provides interpretability through the t-distributed stochastic neighbor embedding (t-SNE) representation of the latent space. Furthermore, the most relevant input features are identified and interpreted medically.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Adding EMA improved prediction rates compared with a model that used medical record data alone in a study using recurrent neural network modeling.
“Another study used recurrent neural network modeling to predict SI with longitudinal electronic medical record data and EMA and found that adding EMA improved prediction rates compared with a model that used medical record data alone (36).”
From New methods for assessing rapid changes in suicide risk (2021) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
(2014) · Cited by 33,419 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
(1998) · PubMed · Cited by 10,216 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
Recurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
A Critical Review of Recurrent Neural Networks for Sequence LearningPreprint
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 2,711 · see citing works
Cross-national prevalence and risk factors for suicidal ideation, plans and attempts
Nock, M. K., Borges, G., Bromet, E. J., Alonso, J., Angermeyer, M., Beautrais, A., Bruffaerts, R., Chiu, W. T., de Girolamo, G., Gluzman, S., et al. (2008) · British Journal of Psychiatry · doi:10.1192/bjp.bp.107.040113 · Cited by 2,654 · see citing works
Suicide and suicidal behaviour
Turecki, G., & Brent, D. A. (2016) · The Lancet · doi:10.1016/s0140-6736(15)00234-2 · Cited by 2,338 · see citing works
Brugha, T. S., Bebbington, P., Tennant, C., & Hurry, J. (1985) · Psychological Medicine · doi:10.1017/s003329170002105x · Cited by 1,108 · see citing works
RETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism
Choi, E., Bahadori, M. T., Kulas, J. A., Schuetz, A., Stewart, W. F., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1608.05745 · Cited by 1,029 · see citing works
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · see citing works
Sánchez–López, M. D. P., & Dresch, V. (2008) · PubMed · Cited by 511 · see citing works
Simon, G. E., Johnson, E., Lawrence, J. M., Rossom, R. C., Ahmedani, B., Lynch, F. L., Beck, A., Waitzfelder, B., Ziebell, R., Penfold, R. B., et al. (2018) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.17101167 · Cited by 400 · cited 4× · see citing works
“is followed for years (in [10], 56% of the patients in the study are treated for more than 5 years, and in [11] the average period is 5.”
BackgroundMethodInfluentialKim, Y., Denton, C., Hoang, L., & Rush, A. M. (2017) · arXiv · doi:10.48550/arxiv.1702.00887 · Cited by 381 · see citing works
Predicting suicidal behavior from longitudinal electronic health records.
Barak-Corren, Y., Castro, V. M., Javitt, S., Hoffnagle, A. G., Dai, Y., Perlis, R. H., Nock, M. K., Smoller, J. W., & Reis, B. Y. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2016.16010077 · Cited by 339 · see citing works
Estimating risk for suicide attempt: Are we asking the right questions?
Baca‐García, E., Pérez-Rodríguez, M. M., Oquendo, M. A., Keyes, K. M., Hasin, D. S., Grant, B. F., & Blanco, C. (2011) · Journal of Affective Disorders · doi:10.1016/j.jad.2011.06.026 · Cited by 225 · see citing works
Digital phenotyping of suicidal thoughts
Kleiman, E. M., Turner, B. J., Fedor, S., Beale, E. E., Picard, R. W., Huffman, J. C., & Nock, M. K. (2018) · Depression and Anxiety · doi:10.1002/da.22730 · Cited by 210 · see citing works
Feasibility and validity of ecological momentary assessment in the investigation of suicide risk
Husky, M., Olié, E., Guillaume, S., Genty, C., Swendsen, J., & Courtet, P. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.08.019 · Cited by 124 · see citing works
Self-citationToward fine-grained phenotyping of suicidal behavior: the role of suicidal subtypes
Bernanke, J. A., Stanley, B. H., & Oquendo, M. A. (2017) · Molecular Psychiatry · doi:10.1038/mp.2017.123 · Cited by 122 · see citing works
Czyz, E. K., King, C. A., & Nahum-Shani, I. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.06.031 · Cited by 112 · see citing works
Ecological momentary assessment is a neglected methodology in suicidology
Davidson, C. L., Anestis, M. D., & Gutierrez, P. M. (2017) · Archives of Suicide Research · doi:10.1080/13811118.2015.1004482 · Cited by 108 · see citing works
Latent Alignment and Variational AttentionPreprint
Deng, Y., Kim, Y., Chiu, J., Guo, D., & Rush, A. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03756 · Cited by 93 · see citing works
User profiles of an electronic mental health tool for ecological momentary assessment: MEmind
Barrigón, M. L., Berrouiguet, S., Carballo, J. J., Bonal-Giménez, C., Fernández‐Navarro, P., Pfang, B., Delgado‐Gómez, D., Courtet, P., Aroca, F., López‐Castromán, J., et al. (2017) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1554 · Cited by 68 · see citing works
Self-citationFrustratingly Short Attention Spans in Neural Language ModelingPreprint
Daniluk, M., Rocktäschel, T., Welbl, J., & Riedel, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04521 · Cited by 58 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [15][15] O. mondiale de la sant´ e, W. H. Organization, W. Staff, and WHO, The ICD-10 classification of mental and behavioural disorders: clinical descriptions and diagnostic guidelines . World Health Organization, 1992, vol. 1. -
- [18][18] R. L. Spitzer, M. Gibbon, J. Williams, and J. Endicott, 'Global assessment of functioning (gaf) scale,' Outcomes assessment in clinical practice , pp. 76-78, 1996. -
- [32][32] L. v. d. Maaten and G. Hinton, 'Visualizing data using t-SNE,' Journal of machine learning research , vol. 9, no. Nov, pp. 2579-2605, 2008.
Cited by
Sorted by citations
Machine learning and the prediction of suicide in psychiatric populations: a systematic review
Pigoni, A., Delvecchio, G., Turtulici, N., Madonna, D., Pietrini, P., Cecchetti, L., & Brambilla, P. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-02852-9 · Cited by 142 · see citing works
Translating promise into practice: a review of machine learning in suicide research and prevention
Kirtley, O. J., van Mens, K., Hoogendoorn, M., Kapur, N., & de Beurs, D. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00254-6 · Cited by 100 · see citing works
Kusuma, K., Larsen, M. E., Quiroz, J. C., Gillies, M. B., Burnett, A., Qian, J., & Torok, M. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.050 · Cited by 90 · see citing works
Machine learning for suicidal ideation identification: A systematic literature review
Heckler, W. F., de Carvalho, J. V., & Barbosa, J. L. V. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107095 · Cited by 67 · see citing works
New methods for assessing rapid changes in suicide risk
Ballard, E. D., Gilbert, J. R., Wusinich, C., & Zarate, C. A. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.598434 · Cited by 63 · see citing works
Using ambulatory assessment to measure dynamic risk processes in affective disorders
Stange, J. P., Kleiman, E. M., Mermelstein, R. J., & Trull, T. J. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.08.060 · Cited by 38 · see citing works
Precision medicine and suicide: an opportunity for digital health
Barrigon, M. L., Courtet, P., Oquendo, M., & Baca-García, E. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1119-8 · Cited by 36 · see citing works
Self-citationAl-Remawi, M. M., Agha, A. S. A. A., Al‐Akayleh, F., Aburub, F. A. F., & Abdel‐Rahem, R. A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e40925 · Cited by 30 · see citing works
Lei, C., Qu, D., Liu, K., & Chen, R. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.33164 · Cited by 27 · see citing works
Suicide and Self-inflicted Injury in Diabetes: A Balancing Act
Barnard-Kelly, K. D., Naranjo, D. M., Majidi, S., Aktürk, H. K., Breton, M. D., Courtet, P., Olié, É., Lal, R., Johnson, N., & Renard, É. (2019) · Journal of Diabetes Science and Technology · doi:10.1177/1932296819891136 · Cited by 25 · see citing works
Self-citationThoth: An intelligent model for assisting individuals with suicidal ideation
Heckler, W. F., Feijó, L. P., de Carvalho, J. V., & Barbosa, J. L. V. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120918 · Cited by 19 · see citing works
Castañeda, P. E. A., Aroca, F., Nicolás, I. M., Espíndola, L. A. C., Barahona, I., Maya‐Hernández, C., Hernández, M. M. L., Mirón, P. C. M., Barrera, D., Aguilar, E. T., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-035041 · Cited by 19 · see citing works
Self-citationSomé, N. H., Noormohammadpour, P., & Lange, S. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1291362 · Cited by 16 · see citing works
The Application of AI to Ecological Momentary Assessment Data in Suicide Research: Systematic Review
Melia, R., Schafer, K. M., Rogers, M. L., Wilson, E., & Joiner, T. E. (2025) · Journal of Medical Internet Research · doi:10.2196/63192 · Cited by 11 · see citing works
From Static to Dynamic Prediction of Suicide Risk
Nguyen, J., Dwyer, D., Nelson, B., & Schmaal, L. (2025) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2025.2426 · Cited by 12 · see citing works
Choo, T.-H. J., Wall, M. M., Brodsky, B. S., Herzog, S., Mann, J. J., Stanley, B., & Galfalvy, H. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.05.093 · Cited by 10 · see citing works
Decoding the cry for help: AI's emerging role in suicide risk assessment
Esmaeilzadeh, P. (2025) · AI and Ethics · doi:10.1007/s43681-025-00758-w · Cited by 11 · see citing works
Identifying high-risk elderly for suicide using machine learning
Lee, H., & Lee, K. (2025) · Empirical Economics · doi:10.1007/s00181-025-02780-7 · Cited by 3 · see citing works
Khosravi, H., Ahmed, I., & Choudhury, A. (2024) · Healthcare · doi:10.3390/healthcare12131262 · Cited by 3 · see citing works
Ballentine, E., Feler, B., Narine, K., & Brown, L. A. (2025) · Journal of the Association of Nurses in AIDS Care · doi:10.1097/jnc.0000000000000586 · Cited by 1 · see citing works
2019 Index IEEE Journal of Biomedical and Health Informatics Vol. 23
(2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2966416
Artificial intelligence in suicide risk assessment: a systematic literature review
Mokheleli, T., Makaba, T., Ndayizigamiye, P., Ntshingila, N., & Twinomurinzi, H. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01031-7
- Authors
- Peis, I., Olmos, P. M., Vera-Varela, C., Barrigon, M. L., Courtet, P., Baca-Garcia, E., & Artes-Rodriguez, A.
- Affiliations
- Instituto de Investigación Sanitaria Gregorio Marañón (ES)
- Hospital Universitario Fundación Jiménez Díaz (ES)
- Université de Montpellier (FR)
- Instituto de Salud Carlos III (ES)
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2019-11
- Volume
- 23
- Issue
- 6
- Pages
- 2286-2293
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W2946874644
- Cited by
- 22
- References
- 26
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 5 · 2023: 2 · 2022: 2 · 2021: 2 · 2020: 1 · 2019: 4
- FWCI
- 1.03
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Journal of Biomedical and Health Informatics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- e-Archivo (Carlos III University of Madrid) (submittedVersion, other-oa)
- HAL (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Spanish MINECO (TEC2015-69868-C2-1-R, TEC2016-78434-C3-3-R, TEC2017-92552-EXP)
- Spanish MICINN (RTI2018-099655-B-I00)
- Comunidad de Madrid (IND2017/TIC-7618, IND2018/TIC-9649, Y2018/TCS-4705, B2017/BMD-3740)
- BBVA Foundation
- ISCIII (PI16/01852)
- AFSP (LSRG-1-005-16)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Keywords
- electronic health records, ecological momentary assessment, recurrent neural networks, attention mechanism, multi-source data fusion, temporal alignment, interpretability, feature selection
- MeSH terms
- Ecological Momentary Assessment, Deep Learning, Suicide Prevention, Adult, Female, Humans, Male, Middle Aged, Models, Psychological, Suicide, Neural Networks, Computer, Electronic Health Records, Suicidal Ideation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
