ChatGPT’s advice is perceived as better than that of professional advice columnists
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
ChatGPT is a high-performance large language model that has the potential to significantly improve human-computer interactions. It can provide advice on a range of topics, but it is unclear how good this advice is relative to that provided by competent humans, especially in situations where empathy is required. Here, we report the first investigation of whether ChatGPT’s responses are perceived as better than those of humans in a task where humans were attempting to be empathetic. Fifty social dilemma questions were randomly selected from 10 well-known advice columns. In a pre-registered survey, participants (N = 404) were each shown one question, along with the corresponding response by an advice columnist and by ChatGPT. ChatGPT’s advice was perceived as more balanced, complete, empathetic, helpful, and better than the advice provided by professional advice columnists (all values of p < 0.001). Although participants could not determine which response was written by ChatGPT (54%, p = 0.29), most participants preferred that their own social dilemma questions be answered by a human than by a computer (77%, p < 0.001). ChatGPT’s responses were longer than those produced by the advice columnists (mean 280.9 words vs. 142.2 words, p < 0.001). In a second pre-registered survey, each ChatGPT answer was constrained to be approximately the same length as that of the advice columnist (mean 143.2 vs. 142.2 words, p = 0.95). This survey (N = 401) replicated the above findings, showing that the benefit of ChatGPT was not solely due to it writing longer answers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 1×
“It has also sparked significant interest within the academic community, leading to a wealth of scholarly literature (Kaddour et al., 2023; Ray, 2023).”
BackgroundThe influence of empathy in human-robot relations
Leite, I., Pereira, A., Mascarenhas, S., Martinho, C., Prada, R., & Paiva, A. (2013) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2012.09.005 · Cited by 344 · cited 1×
“Empathy plays a vital role in many domains (Hoffman, 2000; Sanders et al., 2021); if ChatGPT were to fail to exhibit sufficient empathy, this would adversely affect the quality of its interactions with humans (Leite et al., 2013).”
BackgroundStudy and Analysis of Chat GPT and its Impact on Different Fields of Study
Kalla, D., Smith, N., Kuraku, S., & Samaah, F. (2023) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.5281/zenodo.7767674 · Cited by 284
Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study
Nov, O., Singh, N., & Mann, D. (2023) · JMIR Medical Education · doi:10.2196/46939 · Cited by 198
Challenges and Applications of Large Language Models
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., & McHardy, R. (2023) · arXiv · doi:10.48550/arxiv.2307.10169 · Cited by 196 · cited 1×
“It has also sparked significant interest within the academic community, leading to a wealth of scholarly literature (Kaddour et al., 2023; Ray, 2023).”
BackgroundIs ChatGPT Equipped with Emotional Dialogue Capabilities?
Zhao, W., Zhao, Y., Lu, X., Wang, S., Tong, Y., & Qin, B. (2023) · arXiv · doi:10.48550/arxiv.2304.09582 · Cited by 92
Leveraging Large Language Models for Generating Responses to Patient Messages
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A. (2023) · doi:10.1101/2023.07.14.23292669 · Cited by 43
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41
Cited by
Sycophantic AI decreases prosocial intentions and promotes dependence
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026) · Science · doi:10.1126/science.aec8352 · Cited by 185
“We aggregate data from existing studies of human vs. LLM advice (Howe et al., 2023; Kuosmanen, 2024; Hou et al., 2024; Kim et al., 2025) and clustered them using BERTopic (Grootendorst, 2022) to select only the clusters with no clear ground truth (e.g., romantic relationships, interpersonal issues,…”
ChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe14030043 · Cited by 69
Global trends and hotspots of ChatGPT in medical research: a bibliometric and visualized study
Liu, L., Qu, S., Zhao, H., Kong, L., Xie, Z., Jiang, Z., & Zou, P. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1406842 · Cited by 14
Metadata
- Authors
- Howe, P. D. L., Fay, N., Saletta, M., & Hovy, E.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2023-11-21
- Volume
- 14
- Pages
- 1281255-1281255
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W4388941016
- Cited by
- 42
- References
- 9
- FWCI
- 12.14
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ChatGPT’s advice is perceived as better than that of professional advice columnists. https://app.cochat.ai/papers/10.3389/fpsyg.2023.1281255
- The paper
- Howe, P. D. L., Fay, N., Saletta, M., & Hovy, E. (2023). ChatGPT’s advice is perceived as better than that of professional advice columnists. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2023.1281255
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.