Integrating Artificial and Human Intelligence in Complex, Sensitive Problem Domains: Experiences from Mental Health
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
This article presents a position highlighting the importance of combining artificial intelligence approaches with human intelligence, in other words, the involvement of humans. To do so, we specifically focus on problems of societal significance, stemming from complex, sensitive domains. We first discuss our prior work across a series of projects surrounding social media and mental health, and identify major themes for which augmentation of AI systems and techniques with human feedback has been and can be fruitful and meaningful. We then conclude by noting the implications, in terms of opportunities as well as challenges, that can be drawn from our position, both relating to the specific domain of mental health and for AI researchers and practitioners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · cited 1×
“In prior joint work (De Choudhury et al. 2016), the authors developed a causal inference framework (Pearl 2009) to assess the likelihood that an individual will transition to discussions of suicidal ideation, given a history of mental health discourse on social media.”
BackgroundCombining satellite imagery and machine learning to predict poverty
Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016) · Science · doi:10.1126/science.aaf7894 · Cited by 1,664
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Mental Health Discourse on reddit: Self-Disclosure, Social Support, and Anonymity
De Choudhury, M., & De, S. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14526 · Cited by 836 · cited 1×
“Utilizing comments received on posts shared in Reddit mental health communities as a proxy for social support, in recent research (De Choudhury and De 2014; De Choudhury and Kici-man 2017), we developed a human-machine hybrid statistical methodology that modeled and quantified the effects of the…”
MethodInstagram photos reveal predictive markers of depression
Reece, A. G., & Danforth, C. M. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0110-z · Cited by 519
Social media as a measurement tool of depression in populations
De Choudhury, M., Counts, S., & Horvitz, E. (2013) · Proceedings of the 5th Annual ACM Web Science Conference · doi:10.1145/2464464.2464480 · Cited by 497
The Language of Social Support in Social Media and Its Effect on Suicidal Ideation Risk
De Choudhury, M., & Kiciman, E. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14891 · Cited by 202
Birnbaum, M. L., Ernala, S. K., Rizvi, A. F., De Choudhury, M., & Kane, J. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7956 · Cited by 168 · cited 3×
“2014), shown in figure 2, and that employed expert-generated clinical appraisals from clinical psychologists and psychiatrists to assess and curate the quality of online data related to schizophrenia and psychosis (Birnbaum et al. 2017).”
BackgroundInfluentialModeling and Understanding Visual Attributes of Mental Health Disclosures in Social Media
Manikonda, L., & De Choudhury, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025932 · Cited by 116
Linguistic Markers Indicating Therapeutic Outcomes of Social Media Disclosures of Schizophrenia
Ernala, S. K., Rizvi, A. F., Birnbaum, M. L., Kane, J. M., & De Choudhury, M. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134678 · Cited by 99 · cited 1×
“To enable such comparison, we clustered these linguistic cues via spectral clustering to identify, via Our work (Ernala et al. 2017) showing notable changes in linguistic organization in a clinically appraised psychotic population following diagnosis disclosures on Twitter (day 0 is day of…”
Method
Cited by
Artificial intelligence for mental health and mental illnesses: an overview
Graham, S., Depp, C., Lee, E. E., Nebeker, C., Tu, X., Kim, H.-C., & Jeste, D. V. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1094-0 · Cited by 970
Bickman, L. (2020) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01065-8 · Cited by 184
A Human-Centered Review of Algorithms used within the U.S. Child Welfare System
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376229 · Cited by 116
“âĂŹs [46] work in mental health is a good example, where the researchers validated constructs, focused on data biases and unobserved factors, as well as conducted sensitivity analysis.”
BackgroundWhere is the Human?: Bridging the Gap Between AI and HCI
Inkpen, K., Chancellor, S., De Choudhury, M., Veale, M., & Baumer, E. P. S. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3299002 · Cited by 54
Self-citationKumar, H., Yoo, S., Bernuy, A., Shi, J., Luo, H., Williams, J. J., Kuzminykh, A., Anderson, A., & Kornfield, R. (2024) · doi:10.2139/ssrn.4892998 · Cited by 9
Metadata
- Authors
- De Choudhury, M., & Kiciman, E.
- Venue
- AI Magazine
- Publisher
- Wiley
- Published
- 2018-09
- Volume
- 39
- Issue
- 3
- Pages
- 69-80
- ISSN
- 0738-4602 (ISSN-L)
- 2371-9621
- OpenAlex
- W2890180596
- Cited by
- 27
- References
- 11
- FWCI
- 5.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Integrating Artificial and Human Intelligence in Complex, Sensitive Problem Domains: Experiences from Mental Health. https://app.cochat.ai/papers/10.1609/aimag.v39i3.2815
- The paper
- De Choudhury, M., & Kiciman, E. (2018). Integrating Artificial and Human Intelligence in Complex, Sensitive Problem Domains: Experiences from Mental Health. AI Magazine. https://doi.org/10.1609/aimag.v39i3.2815
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.