The DOI matches its Crossref record; 109 of the 109 references it cites were found in a registry or an index.
Machine Learning and Natural Language Processing in Mental Health: Systematic Review
Open accessPublished version of a preprint
Abstract
BACKGROUND: Machine learning systems are part of the field of artificial intelligence that automatically learn models from data to make better decisions. Natural language processing (NLP), by using corpora and learning approaches, provides good performance in statistical tasks, such as text classification or sentiment mining. OBJECTIVE: The primary aim of this systematic review was to summarize and characterize, in methodological and technical terms, studies that used machine learning and NLP techniques for mental health. The secondary aim was to consider the potential use of these methods in mental health clinical practice. METHODS: This systematic review follows the PRISMA (Preferred Reporting Items for Systematic Review and Meta-analysis) guidelines and is registered with PROSPERO (Prospective Register of Systematic Reviews; number CRD42019107376). The search was conducted using 4 medical databases (PubMed, Scopus, ScienceDirect, and PsycINFO) with the following keywords: machine learning, data mining, psychiatry, mental health, and mental disorder. The exclusion criteria were as follows: languages other than English, anonymization process, case studies, conference papers, and reviews. No limitations on publication dates were imposed. RESULTS: A total of 327 articles were identified, of which 269 (82.3%) were excluded and 58 (17.7%) were included in the review. The results were organized through a qualitative perspective. Although studies had heterogeneous topics and methods, some themes emerged. Population studies could be grouped into 3 categories: patients included in medical databases, patients who came to the emergency room, and social media users. The main objectives were to extract symptoms, classify severity of illness, compare therapy effectiveness, provide psychopathological clues, and challenge the current nosography. Medical records and social media were the 2 major data sources. With regard to the methods used, preprocessing used the standard methods of NLP and unique identifier extraction dedicated to medical texts. Efficient classifiers were preferred rather than transparent functioning classifiers. Python was the most frequently used platform. CONCLUSIONS: Machine learning and NLP models have been highly topical issues in medicine in recent years and may be considered a new paradigm in medical research. However, these processes tend to confirm clinical hypotheses rather than developing entirely new information, and only one major category of the population (ie, social media users) is an imprecise cohort. Moreover, some language-specific features can improve the performance of NLP methods, and their extension to other languages should be more closely investigated. However, machine learning and NLP techniques provide useful information from unexplored data (ie, patients' daily habits that are usually inaccessible to care providers). Before considering It as an additional tool of mental health care, ethical issues remain and should be discussed in a timely manner. Machine learning and NLP methods may offer multiple perspectives in mental health research but should also be considered as tools to support clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Insel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D. S., Quinn, K., Sanislow, C., & Wang, P. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2010.09091379 · Cited by 7,067 · cited 2×
“The former includes data on genetics and neuroscience in its classification of mental health disorders, whereas the latter is solely based on clinical data [107].”
An introduction to latent semantic analysis
Landauer, T. K., Foltz, P. W., & Laham, D. (1998) · Discourse Processes · doi:10.1080/01638539809545028 · Cited by 4,880 · cited 1×
“…new matrix represent aggregates of words and aggregates of documents where each dimension is not interpretable per se, but when words or documents are represented as vectors in this new latent system of coordinates, the scalar product of vectors can be used as a semantic relatedness measure [34].”
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837 · cited 1×
“Indeed, ML is widely used in critical disease models in cardiology, neurology, and diabetes research [6] to automatically identify heart disease risk factors [7], to classify primary progressive aphasia subtypes [8], and for the characterization and diagnosis of cognitive impairments [9], diabetes,…”
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
Motivational Interviewing: Helping People Change
Tober, G. (2013) · Alcohol and Alcoholism · doi:10.1093/alcalc/agt010 · Cited by 4,267
Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466 · cited 1×
“At the same time, even though thousands of papers have been published using medical data, very few have made meaningful contributions to clinical practices [111].”
Machine Learning Applications in Cancer Prognosis and Prediction
Kourou, K., Exarchos, T. P., Exarchos, K. P., Karamouzis, M. V., & Fotiadis, D. I. (2015) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2014.11.005 · Cited by 3,338
A few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326
Deep learning for computational biology
Angermueller, C., Pärnamaa, T., Parts, L., & Stegle, O. (2016) · Molecular Systems Biology · doi:10.15252/msb.20156651 · Cited by 1,473 · cited 1×
“For example, they can help doctors improve their diagnoses, forecast disease outbreaks, and customize treatments [19,20], provide better patient care [21], and predict the splicing activity of individual exons and chromatin marks from DNA sequences [22].”
Applications of Machine Learning in Cancer Prediction and Prognosis
Cruz, J. A., & Wishart, D. S. (2006) · Cancer Informatics · doi:10.1177/117693510600200030 · Cited by 1,371
Benchmarking state-of-the-art classification algorithms for credit scoring: an update of research
Lessmann, S., Baesens, B., Seow, H.-V., & Thomas, L. C. (2015) · European Journal of Operational Research · doi:10.1016/j.ejor.2015.05.030 · Cited by 1,284
Machine learning in mental health: a scoping review of methods and applications
Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019) · Psychological Medicine · doi:10.1017/s0033291719000151 · Cited by 1,047 · cited 1×
“Indeed, mental health care is also benefiting from the advancements in ML [27-29].”
It's only a computer: virtual humans increase willingness to disclose
Lucas, G. M., Gratch, J., King, A., & Morency, L.-P. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.04.043 · Cited by 977 · cited 2×
“In addition, Lucas et al [51] demonstrated through a clinical trial that when people believed they were interacting with a computer rather than an actual clinician, they reported less fear of self-disclosure, reported reduced impression management behaviors, experienced more ease in expressing the…”
Applications of machine learning in cancer prediction and prognosis.
Cruz, J. A., & Wishart, D. S. (2007) · PubMed Central · Cited by 879
Automated analysis of free speech predicts psychosis onset in high-risk youths
Bedi, G., Carrillo, F., Cecchi, G. A., Slezak, D. F., Sigman, M., Mota, N. B., Ribeiro, S., Javitt, D. C., Copelli, M., & Corcoran, C. M. (2015) · npj Schizophrenia · doi:10.1038/npjschz.2015.30 · Cited by 673 · cited 4×
“Each community has a central node: Perlis et al [55] are cited in 7 other papers, Jackson et al [73] are cited in 4 other papers, Carrell et al [74] and Afshar et al [75] are cited in 2 other papers, and Bedi et al [39] are cited in 2 other papers.”
InfluentialTwitter as a Tool for Health Research: A Systematic Review
Sinnenberg, L. E., Buttenheim, A. M., Padrez, K., Mancheno, C., Ungar, L., & Merchant, R. M. (2016) · American Journal of Public Health · doi:10.2105/ajph.2016.303512 · Cited by 666 · cited 1×
“For instance, Sinnenberg et al [36] published a systematic review about Twitter as a tool for health research that included 137 different studies and analyzed over 5 billion tweets using ML; Castro et al [37] have processed 4.2 million EHRs spanning a period of over 20 years.”
InfluentialMachine learning for email spam filtering: review, approaches and open research problems
Dada, E. G., Bassi, J. S., Chiroma, H., Abdulhamid, S. M., Adetunmbi, A. O., & Ajibuwa, O. (2019) · Heliyon · doi:10.1016/j.heliyon.2019.e01802 · Cited by 604
Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0174 · Cited by 540
Natural language processing of social media as screening for suicide risk
Coppersmith, G., Leary, R., Crutchley, P., & Fine, A. (2018) · Biomedical Informatics Insights · doi:10.1177/1178222618792860 · Cited by 506 · cited 4×
“An exception to this is the group of OurDataHelps.org users [77] who volunteered to participate in scientific research and filled out a questionnaire to provide information about themselves.”
InfluentialMachine Learning and the Profession of Medicine
Darcy, A. M., Louie, A. K., & Roberts, L. W. (2016) · JAMA · doi:10.1001/jama.2015.18421 · Cited by 448 · cited 2×
“…NLP can help identify people at clinical risk for depression or suicidal ideation, who most likely do not have access to mental health providers and/or a primary care doctor [61]; however, this reduces confidentiality protection and can lead to increased vulnerability in certain populations [21].”
Prediction of psychosis across protocols and risk cohorts using automated language analysis
Corcoran, C. M., Carrillo, F., Slezak, D. F., Bedi, G., Klim, C. C., Javitt, D. C., Bearden, C. E., & Cecchi, G. (2018) · World Psychiatry · doi:10.1002/wps.20491 · Cited by 381 · cited 1×
“…(PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85]; eating disorders (2 cases) [89,96]; and bipolar disorder (2 cases) [37,102].”
Ly, K. H., Ly, A.-M., & Andersson, G. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.10.002 · Cited by 368
Researching Mental Health Disorders in the Era of Social Media: Systematic Review
Wongkoblap, A., Vadillo, M. A., & Ćurčin, V. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7215 · Cited by 286 · cited 1×
“Although ML and NLP provide new tools and strategies for psychiatric research and practice [52], it should be kept in mind that their use frequently raises ethical and legal concerns over consent to personal data use and data anonymization.”
Automated Breast Cancer Diagnosis Based on Machine Learning Algorithms
Dhahri, H., Maghayreh, E. A., Mahmood, A., Elkilani, W. S., & Faisal, M. (2019) · Journal of Healthcare Engineering · doi:10.1155/2019/4253641 · Cited by 240 · cited 1×
“…disease models in cardiology, neurology, and diabetes research [6] to automatically identify heart disease risk factors [7], to classify primary progressive aphasia subtypes [8], and for the characterization and diagnosis of cognitive impairments [9], diabetes, and cardiovascular disorders [10-17].”
Alanazi, H. O., Abdullah, A. H., & Qureshi, K. N. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0715-6 · Cited by 231 · cited 1×
“Using ML in general and NLP methods in particular, one can create semiautomated systems (operating under human supervision) aiming to improve the specificity of diagnosis, knowledge of psychophysiology, speed of diagnosis, and more accurate estimations of disease severity [48].”
Automated classification of primary progressive aphasia subtypes from narrative speech transcripts
Fraser, K. C., Meltzer, J. A., Graham, N. L., Leonard, C., Hirst, G., Black, S. E., & Rochon, E. (2014) · Cortex · doi:10.1016/j.cortex.2012.12.006 · Cited by 229 · cited 1×
“…disease models in cardiology, neurology, and diabetes research [6] to automatically identify heart disease risk factors [7], to classify primary progressive aphasia subtypes [8], and for the characterization and diagnosis of cognitive impairments [9], diabetes, and cardiovascular disorders [10-17].”
Diabetes Prediction Using Different Machine Learning Approaches
Sonar, P., & Jayamalini, K. (2019) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc.2019.8819841 · Cited by 206
Perlis, R. H., Iosifescu, D. V., Castro, V. M., Murphy, S. N., Gainer, V. S., Minnier, J., Cai, T., Goryachev, S., Zeng, Q., Gallagher, P. J., et al. (2012) · Psychological Medicine · doi:10.1017/s0033291711000997 · Cited by 193 · cited 1×
“Each community has a central node: Perlis et al [55] are cited in 7 other papers, Jackson et al [73] are cited in 4 other papers, Carrell et al [74] and Afshar et al [75] are cited in 2 other papers, and Bedi et al [39] are cited in 2 other papers.”
Digital diabetes: Perspectives for diabetes prevention, management and research
Fagherazzi, G., & Ravaud, P. (2018) · Diabetes & Metabolism · doi:10.1016/j.diabet.2018.08.012 · Cited by 191
Cook, B. L., Progovac, A. M., Chen, P., Mullin, B. O., Hou, S. S., & Baca‐García, E. (2016) · Computational and Mathematical Methods in Medicine · doi:10.1155/2016/8708434 · Cited by 176 · cited 2×
“…found that ML and NLP techniques performed well, although they were not necessarily better than a practitioner’s ability to predict the clinical risk of suicide in their patients [61,62]; thus, the authors proposed statistical NLP approaches to be used in collaboration with clinical practice.”
Semi-Supervised Approach to Monitoring Clinical Depressive Symptoms in Social Media
Yazdavar, A. H., Al-Olimat, H. S., Ebrahimi, M., Bajaj, G., Banerjee, T., Thirunarayan, K., Pathak, J., & Sheth, A. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3123028 · Cited by 173 · cited 2×
“For example, Roysden et al [56] screened administrative data and EHRs from a population of 12,759 patients; Maguen et al [63] compared over 8,168,330 clinical notes collected over 15 years; and Yazdavar et al [79] analyzed posts authored by 4000 Twitter users.”
Fernandes, A. C., Dutta, R., Velupillai, S. U., Sanyal, J., Stewart, R. J., & Chandran, D. (2018) · Scientific Reports · doi:10.1038/s41598-018-25773-2 · Cited by 168 · cited 1×
“Fernandes et al [60] studied EHRs from a cohort of individuals with a history of suicide attempts and a cohort of individuals with a history of suicidal ideation only.”
Gholami-Kordkheili, F., Wild, V., & Strech, D. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2708 · Cited by 166 · cited 1×
“Twitter and other social networks, with almost 3 billion users globally, have become significant sources of information for medical use and research [112].”
Tanana, M., Hallgren, K. A., Imel, Z. E., Atkins, D. C., & Srikumar, V. (2016) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2016.01.006 · Cited by 149 · cited 4×
“All studies were thoroughly screened, and their main ideas are summarized in individual tables (Multimedia Appendix 2 [37-41, 43, 47,”
InfluentialMeasuring Use of Evidence Based Psychotherapy for Posttraumatic Stress Disorder
Shiner, B. R., DʼAvolio, L., Nguyen, T. M., Zayed, M. H., Young‐Xu, Y., Desai, R. A., Schnurr, P. P., Fiore, L. D., & Watts, B. V. (2012) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-012-0421-0 · Cited by 142 · cited 3×
“…psychiatric diagnoses were addiction to alcohol or illicit drugs (6 cases) [43,65,66,75,84,86]; posttraumatic stress disorder (PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2…”
InfluentialValidation of Electronic Health Record Phenotyping of Bipolar Disorder Cases and Controls
Castro, V. M., Minnier, J., Murphy, S. N., Kohane, I. S., Churchill, S. E., Gainer, V. S., Cai, T., Hoffnagle, A. G., Dai, Y. G., Block, S. R., et al. (2014) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2014.14030423 · Cited by 141 · cited 3×
“All studies were thoroughly screened, and their main ideas are summarized in individual tables (Multimedia Appendix 2 [37-41, 43, 47,”
InfluentialPatel, R., Lloyd, T., Jackson, R., Ball, M., Shetty, H., Broadbent, M. T. M., Geddes, J. R., Stewart, R. J., McGuire, P. K., & Taylor, M. J. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-007504 · Cited by 140 · cited 1×
“…(PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85]; eating disorders (2 cases) [89,96]; and bipolar disorder (2 cases) [37,102].”
Analyzing big data in social media: Text and network analyses of an eating disorder forum
Moessner, M., Feldhege, J., Wolf, M., & Bauer, S. (2018) · International Journal of Eating Disorders · doi:10.1002/eat.22878 · Cited by 122 · cited 1×
“…(PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85]; eating disorders (2 cases) [89,96]; and bipolar disorder (2 cases) [37,102].”
Kaur, P., & Sharma, M. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1341-2 · Cited by 119
Artificial Intelligence in Behavioral and Mental Health Care
(2016) · Elsevier eBooks · doi:10.1016/c2013-0-12824-3 · Cited by 117
Using natural language processing to identify problem usage of prescription opioids
Carrell, D., Cronkite, D., Palmer, R. E., Saunders, K. W., Gross, D. E., Masters, E. T., Hylan, T. R., & Korff, M. V. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.09.002 · Cited by 114 · cited 2×
“Each community has a central node: Perlis et al [55] are cited in 7 other papers, Jackson et al [73] are cited in 4 other papers, Carrell et al [74] and Afshar et al [75] are cited in 2 other papers, and Bedi et al [39] are cited in 2 other papers.”
Understanding Depressive Symptoms and Psychosocial Stressors on Twitter: A Corpus-Based Study
Mowery, D. L., Smith, H. A., Cheney, T., Stoddard, G., Coppersmith, G., Bryan, C. J., & Conway, M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6895 · Cited by 114 · cited 3×
“By analyzing tweets, Mowery et al [106] found that there may be alternative ways in which people express depression.”
Carson, N. J., Mullin, B. O., Román, M. J. S., Lu, F. Q., Yang, K. G., Menezes, M., & Cook, B. L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0211116 · Cited by 110 · cited 1×
“Corpora can also be small, as demonstrated in the study conducted by Carson et al [38], who treated 73 respondents’ unstructured clinical notes, or in the study by Bedi et al [39], in which only 34 participants’ 1-hour-long narrative interviews were considered.”
Studying expressions of loneliness in individuals using twitter: an observational study
Guntuku, S. C., Schneider, R., Pelullo, A., Young, J. F., Wong, V., Ungar, L., Polsky, D. E., Volpp, K. G. M., & Merchant, R. M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-030355 · Cited by 110
Drug side effect extraction from clinical narratives of psychiatry and psychology patients
Sohn, S., Kocher, J., Chute, C. G., & Savova, G. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000351 · Cited by 108 · cited 1×
“Treatment issues such as adherence or misuse are also depicted (6 cases) [56,72,74,81,101,103].”
Patel, R., Jayatilleke, N., Broadbent, M. T. M., Chang, C., Foskett, N., Gorrell, G., Hayes, R. D., Jackson, R., Johnston, C., Shetty, H., et al. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-007619 · Cited by 107 · cited 1×
“…(PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85]; eating disorders (2 cases) [89,96]; and bipolar disorder (2 cases) [37,102].”
Tran, T., & Kavuluru, R. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.010 · Cited by 101 · cited 3×
“A total of 8 studies were transnosographic [59,67-71,”
Afshar, M., Phillips, A., Karnik, N., Mueller, J., To, D., Gonzalez, R., Price, R., Cooper, R., Joyce, C., & Dligach, D. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy166 · Cited by 94 · cited 2×
“Each community has a central node: Perlis et al [55] are cited in 7 other papers, Jackson et al [73] are cited in 4 other papers, Carrell et al [74] and Afshar et al [75] are cited in 2 other papers, and Bedi et al [39] are cited in 2 other papers.”
Cohen, A. S., Mitchell, K. R., & Elvevåg, B. (2014) · Schizophrenia Research · doi:10.1016/j.schres.2014.09.013 · Cited by 91 · cited 2×
“In addition, NLP is essential in psychiatry because language-based deficits are common symptoms of depression, behavioral disorder, ASD, personality disorder, and schizophrenia [30].”
Pestian, J. P., Grupp‐Phelan, J., Cohen, K. B., Meyers, G., Richey, L., Matykiewicz, P., & Sorter, M. T. (2015) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12180 · Cited by 81 · cited 2×
“…found that ML and NLP techniques performed well, although they were not necessarily better than a practitioner’s ability to predict the clinical risk of suicide in their patients [61,62]; thus, the authors proposed statistical NLP approaches to be used in collaboration with clinical practice.”
Epidemiology from Tweets: Estimating Misuse of Prescription Opioids in the USA from Social Media
Chary, M. A., Genes, N. G., Giraud-Carrier, C. G., Hanson, C. L., Nelson, L. S., & Manini, A. F. (2017) · Journal of Medical Toxicology · doi:10.1007/s13181-017-0625-5 · Cited by 75 · cited 2×
“Treatment issues such as adherence or misuse are also depicted (6 cases) [56,72,74,81,101,103].”
He, Q., Veldkamp, B. P., Glas, C. A. W., & de Vries, T. (2017) · Assessment · doi:10.1177/1073191115602551 · Cited by 74 · cited 4×
“Data extraction was also used for diagnosis: He et al [47] diagnosed PTSD with an accuracy of 82% after analyzing free texts written by trauma survivors.”
InfluentialLe, D. V., Montgomery, J., Kirkby, K. C., & Scanlan, J. D. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.08.007 · Cited by 73
Intimate insight: MDMA changes how people talk about significant others
Baggott, M. J., Kirkpatrick, M. G., Bedi, G., & de Wit, H. (2015) · Journal of Psychopharmacology · doi:10.1177/0269881115581962 · Cited by 69 · cited 2×
“The following are some examples of studies that pursue this objective: Baggott et al [65] found that MDMA (3,4-méthylènedioxy-N-méthylamphétamine; Ecstasy) altered individuals’ speech patterns more frequently than the placebo and led to an increase in both positive and negative social and sexual…”
Maguen, S., Madden, E., Patterson, O. V., DuVall, S. L., Goldstein, L. A., Burkman, K. M., & Shiner, B. R. (2018) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-018-0850-5 · Cited by 69
Oh, J., Yun, K., Hwang, J.-H., & Chae, J. (2017) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2017.00192 · Cited by 67
Iqbal, E., Mallah, R., Jackson, R., Ball, M., Ibrahim, Z. M., Broadbent, M. T. M., Dzahini, O., Stewart, R. J., Johnston, C., & Dobson, R. (2015) · PLoS ONE · doi:10.1371/journal.pone.0134208 · Cited by 66
Metzger, M.-H., Tvardik, N., Gicquel, Q., Bouvry, C., Poulet, E., & Potinet, V. (2016) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1522 · Cited by 64 · cited 2×
“A total of 2 studies targeted automated epidemiological monitoring: Metzger et al [57] provided a method of detecting suicide attempts from EHRs and Leroy et al [58] achieved automatic extraction of criteria for ASD from EHRs with an accuracy of 76%.”
InfluentialPredicting Social Anxiety Treatment Outcome Based on Therapeutic Email Conversations
Hoogendoorn, M., Berger, T., Schulz, A., Stolz, T., & Peter Szolovits, B. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2601123 · Cited by 60 · cited 1×
“…(PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85]; eating disorders (2 cases) [89,96]; and bipolar disorder (2 cases) [37,102].”
Parthipan, A., Banerjee, I., Humphreys, K. N., Asch, S. M., Curtin, C. M., Carroll, I. R., & Hernandez‐Boussard, T. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210575 · Cited by 60
Clustering algorithms for bank customer segmentation
Zakrzewska, D., & Murlewski, J. (2005) · International Conference on Intelligent Systems Design and Applications (ISDA) · doi:10.1109/isda.2005.33 · Cited by 59
McKernan, L. C., Clayton, E. W., & Walsh, C. G. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00650 · Cited by 52
Zhong, Q., Mittal, L. P., Nathan, M., Brown, K. M., Gonzalez, D. K., Cai, T. A., Finan, S., Gelaye, B., Avillach, P., Smoller, J. W., et al. (2018) · European Journal of Epidemiology · doi:10.1007/s10654-018-0470-0 · Cited by 51 · cited 1×
“Zhong et al [78] have the highest degree: this paper has more than three common references with as many as eight other papers, in fact, with 8 references.”
A robust voting approach for diabetes prediction using traditional machine learning techniques
Mahabub, A. (2019) · SN Applied Sciences · doi:10.1007/s42452-019-1759-7 · Cited by 51 · cited 1×
“…disease models in cardiology, neurology, and diabetes research [6] to automatically identify heart disease risk factors [7], to classify primary progressive aphasia subtypes [8], and for the characterization and diagnosis of cognitive impairments [9], diabetes, and cardiovascular disorders [10-17].”
Suicide Among Older Adults Living in or Transitioning to Residential Long-term Care, 2003 to 2015
Mezuk, B. R., Ko, T. M., Kalesnikava, V. A., & Jurgens, D. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.5627 · Cited by 46
Yan, H., Fitzsimmons‐Craft, E. E., Goodman, M., Krauss, M. J., Das, S., & Cavazos‐Rehg, P. A. (2019) · International Journal of Eating Disorders · doi:10.1002/eat.23148 · Cited by 45 · cited 1×
“…(PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85]; eating disorders (2 cases) [89,96]; and bipolar disorder (2 cases) [37,102].”
Jackson, R., Ball, M., Patel, R., Hayes, R., Dobson, R., & Stewart, R. J. (2014) · PubMed Central · doi:10.13140/2.1.3722.9121 · Cited by 44
Gondy Leroy, A., Gu, Y., Pettygrove, S. D., Galindo, M. K., Arora, A., & Kurzius‐Spencer, M. (2018) · Journal of Medical Internet Research · doi:10.2196/10497 · Cited by 43 · cited 2×
“…or illicit drugs (6 cases) [43,65,66,75,84,86]; posttraumatic stress disorder (PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85];…”
An automatic system to identify heart disease risk factors in clinical texts over time
Chen, Q., Li, H., Tang, B., Wang, X., Liu, X., Liu, Z., Liu, S., Wang, W., Deng, Q., Zhu, S., et al. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.09.002 · Cited by 41 · cited 1×
“…is widely used in critical disease models in cardiology, neurology, and diabetes research [6] to automatically identify heart disease risk factors [7], to classify primary progressive aphasia subtypes [8], and for the characterization and diagnosis of cognitive impairments [9], diabetes, and…”
Filannino, M., Stubbs, A., & Uzuner, Ö. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.04.017 · Cited by 41 · cited 2×
“A total of 8 studies were transnosographic [59,67-71,”
Takano, K., Ueno, M., Moriya, J., Mori, M., Nishiguchi, Y., & Raes, F. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0753-x · Cited by 40 · cited 1×
“Only 1 study on mechanical restraints [90] and 1 on cognitive troubles [97] were found.”
The rate of reply and nature of responses to suicide-related posts on Twitter
O’Dea, B., Achilles, M. R., Larsen, M. E., Batterham, P. J., Calear, A. L., & Christensen, H. M. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.07.004 · Cited by 40 · cited 1×
“Through analyses of Twitter posts, O’Dea et al [49] identified the importance of creating real-time campaigns to increase help-seeking behaviors and reduce the stigma attached to mental health.”
Machine Learning and Evidence-Based Medicine
Scott, I. A. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0115 · Cited by 39 · cited 2×
“…traditional epidemiologic approach of evidence-based medicine owing to its high processing speed and ability to handle large volumes of data with heterogeneous variables (electronic health records, administrative data sets, wearable sensors, genomic and proteomic databanks, and social media) [18].”
Clustering Suicide Attempters: Impulsive-Ambivalent, Well-Planned, or Frequent
Lopez-Castroman, J., Nogue, E., Guillaume, S., Picot, M. C., & Courtet, P. (2016) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.15m09882 · Cited by 38 · cited 1×
“From a mental health perspective, the prevention of suicidal risk has recently been substantially studied [23-26].”
Machine Learning Will Change Medicine
Forsting, M. (2017) · Journal of Nuclear Medicine · doi:10.2967/jnumed.117.190397 · Cited by 38 · cited 1×
“Forsting et al [45] acknowledge that ML and NLP techniques can be useful for optimism bias (eg, the difference between a person’s expectation and the actual outcome or the concept that a clinician may think that his or her patient’s problem falls solely into a specific discipline in which the…”
Danielsen, A. A., Fenger, M. H. J., Østergaard, S. D., Nielbo, K. L., & Mors, O. (2019) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13061 · Cited by 38 · cited 1×
“Only 1 study on mechanical restraints [90] and 1 on cognitive troubles [97] were found.”
Is it ethical to use prognostic estimates from machine learning to treat psychosis
(2018) · AMA Journal of Ethics · doi:10.1001/amajethics.2018.804 · Cited by 33
Analyzing depression tendency of web posts using an event-driven depression tendency warning model
Tung, C.-M., & Lu, W. (2015) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2015.10.003 · Cited by 32
Candyflipping and Other Combinations: Identifying Drug–Drug Combinations from an Online Forum
Chary, M. A., Yi, D., & Manini, A. F. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00135 · Cited by 32 · cited 4×
“In addition, when Chary et al [66] focused on Lycaeum users and Coppersmith et al [76] mentioned participants from a company , the lack of precise information on the participants of a cohort was obvious.”
InfluentialExtracting Alcohol and Substance Abuse Status from Clinical Notes: The Added Value of Nursing Data
Topaz, M., Murga, L., Bar‐Bachar, O., Cato, K., & Collins, S. (2019) · Studies in health technology and informatics · doi:10.3233/shti190386 · Cited by 32 · cited 1×
“Other psychiatric diagnoses were addiction to alcohol or illicit drugs (6 cases) [43,65,66,75,84,86]; posttraumatic stress disorder (PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2…”
The Research Domain Criteria (RDoC) and the historical roots of psychopathology: A viewpoint
Stoyanov, D., Telles‐Correia, D., & Cuthbert, B. N. (2019) · European Psychiatry · doi:10.1016/j.eurpsy.2018.11.007 · Cited by 29 · cited 1×
“Pro-RDoC practitioners argue that DSM syndromes have significant limitations when used as phenotypes for identifying biomarkers and specific genetic variants associated with mental illness [108].”
Scalable mental health analysis in the clinical whitespace via natural language processing
Coppersmith, G., Hilland, C., Frieder, O., & Leary, R. (2017) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi.2017.7897288 · Cited by 28 · cited 3×
“In addition, when Chary et al [66] focused on Lycaeum users and Coppersmith et al [76] mentioned participants from a company , the lack of precise information on the participants of a cohort was obvious.”
Doan, S., Ritchart, A., Perry, N., Chaparro, J. D., & Conway, M. (2017) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.5939 · Cited by 28 · cited 7×
“Typical examples are in studies by Doan et al [41] and Jackson et al [73], in which tweets were selected because they contained the hashtags #stress and #relax and their words were used in a bag-of-words without any further linguistic treatment [41] or tweets were selected based on the presence of…”
InfluentialValidation of an alcohol misuse classifier in hospitalized patients
To, D., Sharma, B., Karnik, N. S., Joyce, C., Dligach, D., & Afshar, M. (2019) · Alcohol · doi:10.1016/j.alcohol.2019.09.008 · Cited by 27
Advances in Decision Sciences, Image Processing, Security and Computer Vision
(2019) · Learning and analytics in intelligent systems · doi:10.1007/978-3-030-24322-7 · Cited by 26
Counting trees in Random Forests: Predicting symptom severity in psychiatric intake reports
Scheurwegs, E., Sushil, M., Tulkens, S., Daelemans, W. M. P., & Luyckx, K. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.007 · Cited by 24
Text Classification to Inform Suicide Risk Assessment in Electronic Health Records
Bittar, A., Velupillai, S. U., Roberts, A., & Dutta, R. (2019) · Studies in health technology and informatics · doi:10.3233/shti190179 · Cited by 24
Yu, L.-C., Chan, C., Lin, C.-C., & Lin, I.-C. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.01.006 · Cited by 23
Berrouiguet, S., Billot, R., Larsen, M. E., López‐Castromán, J., Jaussent, I., Walter, M., Lenca, P., Baca‐García, E., & Courtet, P. (2018) · JMIR Mental Health · doi:10.2196/mental.9766 · Cited by 21
Identifying epilepsy psychiatric comorbidities with machine learning
Glauser, T., Santel, D., Delbello, M., Faist, R., Toon, T., Clark, P. O., McCourt, R., Wissel, B. D., & Pestian, J. (2019) · Acta Neurologica Scandinavica · doi:10.1111/ane.13216 · Cited by 21
Automatic recognition of symptom severity from psychiatric evaluation records
Goodwin, T. R., Maldonado, R., & Harabagiu, S. M. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.05.020 · Cited by 20 · cited 3×
“Goodwin et al [59] classified symptoms of patients with psychosis into 4 different levels of severity (absent, mild, moderate, and severe) using statistical analyses.”
Investigación traslacional en psiquiatría: el marco Research Domain Criteria (RDoC)
Vilar, A., Pérez, V., Blasco, M. J., Pérez-Gallo, E., Ballester, L., Vila, S., Alonso, J., Serrano‐Blanco, A., & Forero, C. G. (2018) · Revista de Psiquiatría y Salud Mental · doi:10.1016/j.rpsm.2018.04.002 · Cited by 20
Barroilhet, S. A., Pellegrini, A. M., McCoy, T. H., & Perlis, R. H. (2019) · Psychological Medicine · doi:10.1017/s0033291719002320 · Cited by 20 · cited 1×
“…(PTSD; 3 cases) [47,63,64]; neurodevelopmental disorders (3 cases) [42,58,93]; psychotic spectrum disorders, including schizophrenia (3 cases) [39,95,100]; anxiety (2 cases) [41,98]; personality disorder (1 case) [85]; eating disorders (2 cases) [89,96]; and bipolar disorder (2 cases) [37,102].”
Sentiment Analysis in Children with Neurodevelopmental Disorders in an Ingroup/Outgroup Setting
Paz, E. V., Martino, M., Hyland, M., Corleto, M., Puga, C., Peralta, M., Deltetto, N., Kuhlmann, T., Cavalié, D., Leist, M., et al. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-019-04242-3 · Cited by 18
Translational research in psychiatry: The Research Domain Criteria Project (RDoC)
Vilar, A., Pérez, V., Blasco, M. J., Pérez-Gallo, E., Ballester, L., Vila, S., Alonso, J., Serrano‐Blanco, A., & Forero, C. G. (2019) · Revista de Psiquiatría y Salud Mental (English Edition) · doi:10.1016/j.rpsmen.2018.04.002 · Cited by 17
Luo, S. X., Shinall, J. A., Peterson, B. S., & Gerber, A. J. (2015) · Autism Research · doi:10.1002/aur.1581 · Cited by 16 · cited 4×
“However, in practice, the selected studies tend to confirm clinical hypotheses based on fundamental clinical intuitions, namely language abnormalities in adults with ASD [42].”
InfluentialAn Advanced Conceptual Diagnostic Healthcare Framework for Diabetes and Cardiovascular Disorders
Sharma, M., Singh, G., & Singh, R. (2018) · ICST Transactions on Scalable Information Systems · doi:10.4108/eai.19-6-2018.154828 · Cited by 16
Posada, J., Barda, A. J., Shi, L., Xue, D., Ruiz, V., Kuan, P.-H., Ryan, N. D., & Tsui, F. R. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.05.019 · Cited by 15
Karthik, A. A., Moulahi, B., Azé, J., & Bringay, S. (2019) · Studies in health technology and informatics · doi:10.3233/shti190181 · Cited by 15
Automatic classification of RDoC positive valence severity with a neural network
Clark, C., Wellner, B., Davis, R., Aberdeen, J. S., & Hirschman, L. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.005 · Cited by 13
Hart, K. L., Perlis, R. H., & McCoy, T. H. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.09.043 · Cited by 12 · cited 1×
“Treatment issues such as adherence or misuse are also depicted (6 cases) [56,72,74,81,101,103].”
Dai, H.-J., Su, E. C., Uddin, M., Jonnagaddala, J., Wu, C., & Syed-Abdul, S. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.08.009 · Cited by 10 · cited 2×
“Similarly, studies rarely consider cultural or ethnic differences within a sample [80].”
Oliva, J., Serrano, J. I., del Castillo, M. D., & Iglesias, Á. (2014) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2014.04.002 · Cited by 9 · cited 1×
“…disease models in cardiology, neurology, and diabetes research [6] to automatically identify heart disease risk factors [7], to classify primary progressive aphasia subtypes [8], and for the characterization and diagnosis of cognitive impairments [9], diabetes, and cardiovascular disorders [10-17].”
Analyzing Abusive Text Messages to Detect Digital Dating Abuse
Roy, T., McClendon, J. L., & Hodges, L. F. (2018) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2018.00039 · Cited by 9 · cited 3×
“As stated in the Results section, one of the 3 main population categories is social network or chat users [40,41,66,77,79], whose members are predominantly young.”
Roysden, N., & Wright, A. T. (2015) · PubMed Central · Cited by 8
Défense de la clinique en psychiatrie
(2014) · Érès eBooks · doi:10.3917/eres.sasso.2014.01 · Cited by 5
Towards Constructing a New Taxonomy for Psychiatry Using Self-reported Symptoms
Ross, J. A., Neylan, T. C., Weiner, M. W., Linda, C., Samuelson, K. W., & Sim, I. (2015) · Studies in health technology and informatics · doi:10.3233/978-1-61499-564-7-736 · Cited by 2 · cited 1×
“A concrete application of this new system used statistical NLP methods to create a phenotypic homogenous cohort that allowed a better comparison [109].”
Research works printed but not linked: 7
- GauthierEBrissonLLencaPRagusaSBreast Cancer Risk Score: a Data Mining Approach to Improve ReadabilityThe International Conference on Data Mining2011CDM'11June 23-27, 2011Las Vegas, United States1521
- GauthierEBrissonLLencaPClavel-ChapelonFRagusaSChallenges to Building a Platform for a Breast Cancer Risk ScoreSixth International Conference on Research Challenges in Information Science201205RCIS'12June 1-7, 2012Valencia, Spain
- Bughin, J, 2017, McKinsey Global Institute, 80
- Hays, DG, 1967, Introduction to Computational Linguistics, Mathematical Linguistics and Automatic Language Processing
- Chollet, F, 2017, Deep Learning with Python
- Sarasohn-Kahn, J, 2008, Technical report, California HealthCare Foundation
- Ross, J, 2015, Stud Health Technol Inform, 216, 736
Cited by
Sorted by citations
Alqahtani, T., Badreldin, H. A., Alrashed, M., Alshaya, A. I., Alghamdi, S. S., bin Saleh, K., Alowais, S. A., Alshaya, O. A., Rahman, I., Al Yami, M. S., et al. (2023) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2023.05.016 · Cited by 565
Machine learning-based approach: global trends, research directions, and regulatory standpoints
Pugliese, R., Regondi, S., & Marini, R. (2021) · Data Science and Management · doi:10.1016/j.dsm.2021.12.002 · Cited by 425
Large Language Models for Mental Health Applications: Systematic Review
Guo, Z., Lai, A., Thygesen, J. H., Farrington, J., Keen, T., & Li, K. (2024) · JMIR Mental Health · doi:10.2196/57400 · Cited by 269
“The existing literature includes a review of the application of machine learning and NLP in mental health [47], analyses of LLMs in medicine [32], and a scoping review of LLMs in mental health.”
BackgroundMethodTimmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221134490 · Cited by 263
Modern views of machine learning for precision psychiatry
Chen, Z. S., Kulkarni, P., Galatzer‐Levy, I. R., Bigio, B., Nasca, C., & Zhang, Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100602 · Cited by 241
“Language is not only one of the primary expressions of human behavior that carries a variety of implicit and explicit markers relevant to mental health [193, 194], but also more abundantly available compared to speech data.”
Background
Metadata
- Authors
- Le Glaz, A., Haralambous, Y., Kim-Dufor, D.-H., Lenca, P., Billot, R., Ryan, T. C., Marsh, J., DeVylder, J., Walter, M., Berrouiguet, S., & Lemey, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Centre Hospitalier Régional Universitaire de Brest (FR)
- Johns Hopkins University (US)
- Fordham University (US)
- Université de Bretagne Occidentale (FR)
- Corresponding author
- Lemey, Christophe
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-05-04
- Volume
- 23
- Issue
- 5
- Pages
- e15708
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC8132982
- OpenAlex
- W3089862415
- Cited by
- 576
- References
- 109
- Citations by year
- 2026: 102 · 2025: 155 · 2024: 144 · 2023: 105 · 2022: 57 · 2021: 13
- FWCI
- 7.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Oskar-Bordeaux (Universite de Bordeaux) (submittedVersion, cc-by)
- Funders
- Agence Nationale de la Recherche (ANR-18-RHUS-0014, ANR18)
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- machine learning, natural language processing, mental health, mental illness, symptom extraction, treatment effectiveness, social media, electronic health records, clinical practice, artificial intelligence in psychiatry, language-specific features, nosography, clinical decision support
- MeSH terms
- Machine Learning, Data Management, Artificial Intelligence, Humans, Mental Health, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
