ChatGPT Giving Relationship Advice – How Reliable Is It?
Open accessAbstract
In the evolving realm of natural language processing (NLP), generative AI models like ChatGPT are increasingly utilized across various applications. Among the possible purposes, many people are considering asking ChatGPT for relationship advice. However, the lack of in-depth examination of ChatGPT's response quality could be concerning when it is used for personal topics like mental health issues and intimate relationship problems. In these topics, a piece of misleading advice could cause harmful repercussions. In response to people's growing interest in using ChatGPT as a relationship advisor, our research evaluates ChatGPT's proficiency in discerning relationship advice. Specifically, we investigate its alignment with human judgements. We conducted our analysis with 13,138 Reddit posts about intimate relationship problems to examine the overall alignment. Furthermore, we investigate ChatGPT's consistency in judging intimate relationship advice by re-prompting identical queries. Our results indicate a significant disparity between ChatGPT and human judgments, with the model displaying inconsistency in its own decisions. Our findings emphasize the need for comprehensive insights into ChatGPT's mechanisms for intimacy problems and future improvements in its proficiency in helping people's relationship struggles.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
All datasets are publicly available except Hou et al. (2024), from whom the authors obtained explicit consent.
“All the datasets are publicly available beside Hou et al. ( 2024), from whom we obtained explicit consent from the authors.”
From Sycophantic AI decreases prosocial intentions and promotes dependence (2026) · cites this paper as Hou et al. (2024)
The Dataset section says Hou et al.’s refined dataset accompanies the paper, and the Conclusion says it is included in the supplemental material. That contradicts the claim that Hou et al.’s dataset is an exception to public availability. The paper does not establish whether the citing authors obtained explicit consent. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredA computational study compared ChatGPT's relationship advice with relationship advice provided by laypersons in a Reddit forum and found low concordance.
“Instead of benchmarking ChatGPT's relationship advice against expert evaluations, a computational study compared it with relationship advice provided by laypersons in a Reddit forum and found low concordance [57].”
From The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024 (2024) · cites this paper as [57]
The Introduction and Conclusion report weak alignment between ChatGPT’s rankings of relationship suggestions and Reddit users’ judgments (Kendall’s τ < 0.10). But the Limitations section says the study excluded generating advice with ChatGPT: it compared rankings of existing suggestions, not ChatGPT’s own advice with advice written by Reddit users. No page numbers are available in our copy. (LLM, conf=0.90)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1× · see citing works
“ChatGPT’s ability to give prudent relationship advice has not yet been exhaustively explored, leading to prevalent concerns about its response quality (Sallam 2023).”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1× · see citing works
“A pioneering study in 2017 involving the Woebot Chatbot found a significant reduction in depression symptoms among its users (Fitzpatrick, Darcy, and Vierhile 2017).”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Physical health consequences of physical and psychological intimate partner violence
Coker, A. L. (2000) · Archives of Family Medicine · doi:10.1001/archfami.9.5.451 · Cited by 1,166 · see citing works
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099 · see citing works
Studying Reddit: A Systematic Overview of Disciplines, Approaches, Methods, and Ethics
Proferes, N., Jones, N., Gilbert, S., Fiesler, C., & Zimmer, M. (2021) · Social Media + Society · doi:10.1177/20563051211019004 · Cited by 537 · cited 1× · see citing works
“We collected a dataset of 13,138 Reddit posts on relationship issues spanning from 2015 to 2023 with Reddit APIs, the official study tools for researchers (Proferes et al. 2021).”
MethodDomestic Violence and Abuse in Intimate Relationship from Public Health Perspective
Rakovec-Felser, Z. (2014) · Health Psychology Research · doi:10.4081/hpr.2014.1821 · Cited by 195 · see citing works
Fernández-Fuertes, A. A., & Fuertes, A. (2010) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2010.01.002 · Cited by 190 · see citing works
What ChatGPT Tells Us about Gender: A Cautionary Tale about Performativity and Gender Biases in AI
Gross, N. (2023) · Social Sciences · doi:10.3390/socsci12080435 · Cited by 162 · see citing works
Mental health values of professional therapists: A national interdisciplinary survey.
Jensen, J. P., & Bergin, A. E. (1988) · Professional Psychology Research and Practice · doi:10.1037/0735-7028.19.3.290 · Cited by 152 · see citing works
(2010) · Princeton University Press eBooks · doi:10.1515/9781400832200.40 · Cited by 135 · see citing works
Artificial intelligence in the era of ChatGPT - Opportunities and challenges in mental health care
Singh, O. P. (2023) · Indian Journal of Psychiatry · doi:10.4103/indianjpsychiatry.indianjpsychiatry_112_23 · Cited by 131 · see citing works
Quality of information and appropriateness of ChatGPT outputs for urology patients
Cocci, A., Pezzoli, M., Re, M. L., Russo, G. I., Asmundo, M. G., Fode, M. M., Cacciamani, G. E., Cimino, S., Minervini, A., & Durukan, E. (2023) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-023-00705-y · Cited by 131 · see citing works
A new era in Internet interventions: The advent of Chat-GPT and AI-assisted therapist guidance
Carlbring, P., Hadjistavropoulos, H. D., Kleiboer, A. M., & Andersson, G. (2023) · Internet Interventions · doi:10.1016/j.invent.2023.100621 · Cited by 120 · see citing works
An Attachment Perspective on Abusive Dynamics in Intimate Relationships.
Bartholomew, K., & Allison, C. J. (2006) · Cited by 114 · see citing works
Assessing social anxiety using gps trajectories and point-of-interest data
Huang, Y., Xiong, H., Leach, K., Zhang, Y., Chow, P., Fua, K., Teachman, B. A., & Barnes, L. E. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971761 · Cited by 109 · see citing works
Self-citationSex-Related Communication With Mothers and Close Friends During the Transition to University
Lefkowitz, E. S., & Espinosa‐Hernández, G. (2007) · The Journal of Sex Research · doi:10.1080/00224490709336789 · Cited by 89 · see citing works
Close Relationships and Happiness
Saphire-Bernstein, S., & Taylor, S. E. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199557257.013.0060 · Cited by 86 · see citing works
Adam, E. K., Chyu, L., Hoyt, L. T., Doane, L. D., Boisjoly, J., Duncan, G. J., Chase-Lansdale, P. L., & McDade, T. W. (2011) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2010.12.012 · Cited by 75 · see citing works
Smith, T. W., & Baucom, B. (2017) · American Psychologist · doi:10.1037/amp0000123 · Cited by 62 · see citing works
Physical Dating Violence Among High School Students--United States, 2003
Black, M. C., Noonan, R. K., Legg, M.-L., Eaton, D. E., & Breiding, M. J. (2006) · PsycEXTRA Dataset · doi:10.1037/e514442006-003 · Cited by 61 · see citing works
Elkington, K. S., Hackler, D., Walsh, T. A., Latack, J. A., McKinnon, K., Borges, C., Wright, E. R., & Wainberg, M. L. (2013) · Journal of Adolescent Research · doi:10.1177/0743558412467686 · Cited by 45 · see citing works
Quality of information and appropriateness of ChatGPT outputs for urology patients
Cocci, A., Pezzoli, M., Re, M. L., Russo, G. I., Asmundo, M. G., Fode, M. M., Cacciamani, G. E., Cimino, S., Minervini, A., & Durukan, E. (2023) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-023-00754-3 · Cited by 45 · see citing works
Considering ChatGPT to be the first aid for young adults on mental health issues
Aminah, S., Hidayah, N. E. F., & Ramli, M. (2023) · Journal of Public Health · doi:10.1093/pubmed/fdad065 · Cited by 29 · see citing works
Sex-Related Communication With Mothers and Close Friends During the Transition to University
Lefkowitz, E. S., & Espinosa‐Hernández, G. (2007) · The Journal of Sex Research · doi:10.1207/s15598519jsr4401_3 · Cited by 26 · see citing works
McKiernan, A., Ryan, P. H., McMahon, E., & Butler, E. (2017) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2016.0450 · Cited by 16 · see citing works
Mental health values of professional therapists: A national interdisciplinary survey.
Jensen, J. P., & Bergin, A. E. (1988) · Professional Psychology Research and Practice · doi:10.1037//0735-7028.19.3.290 · Cited by 13 · see citing works
Accessibility for mental healthcare
Chrysikou, E. G. (2013) · Facilities · doi:10.1108/02632771311324972 · Cited by 12 · see citing works
Assessing social anxiety using gps trajectories and point-of-interest dataPreprint
Huang, Y., Xiong, H., Leach, K., et al. (2021) · doi:10.31234/osf.io/cv6nw · Cited by 12 · see citing works
Self-citationTo seek or not to seek advice: Talking about romantic issues during emerging adulthood
Tagliabue, S., Olivari, M. G., Giuliani, C., & Confalonieri, E. (2018) · Europe’s Journal of Psychology · doi:10.5964/ejop.v14i1.1454 · Cited by 8 · see citing works
Vallade, J. I., Dillow, M. R., & Myers, S. A. (2016) · Communication Quarterly · doi:10.1080/01463373.2015.1103291 · Cited by 4 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 2
- American Academy of Child and Adolescent Psychiatry Committee on Health Care Access and Economics Task Force on Mental Health. 2009. Improving mental health services in primary care: reducing administrative and finan- cial barriers to access and collaboration. Pediatrics, 123(4): 1248-1251.
- Collisson, B.; Cordoviz, P.; Ponce de Leon, L.; Guillen, S.; Shier, J.; and Xiao, Z. 2018. " Should I Break Up or Make Up?" A Text Analysis of Online Relationship Advice. North American Journal of Psychology, 20(2).
Cited by
Sorted by citations
Sycophantic AI decreases prosocial intentions and promotes dependence
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026) · Science · doi:10.1126/science.aec8352 · Cited by 185 · see citing works
“We aggregate data from existing studies of human vs. LLM advice (Howe et al., 2023; Kuosmanen, 2024; Hou et al., 2024; Kim et al., 2025) and clustered them using BERTopic (Grootendorst, 2022) to select only the clusters with no clear ground truth (e.g., romantic relationships, interpersonal issues,…”
The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024
Döring, N., Le, T. D., Vowels, L. M., Vowels, M. J., & Marcantonio, T. L. (2024) · Current Sexual Health Reports · doi:10.1007/s11930-024-00397-y · Cited by 65 · see citing works
- Authors
- Hou, H., Leach, K., & Huang, Y.
- Affiliations
- Vanderbilt University (US)
- Venue
- Proceedings of the International AAAI Conference on Web and Social Media
- Published
- 2024-05-28
- Volume
- 18
- Pages
- 610-623
- ISSN
- 2162-3449 (ISSN-L)
- 2334-0770
- OpenAlex
- W4399205253
- Cited by
- 14
- References
- 33
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 2
- FWCI
- 6.39
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the International AAAI Conference on Web and Social Media (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Social Psychology › Attachment and Relationship Dynamics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- ChatGPT, relationship advice, human-AI alignment, Reddit posts, AI chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
