CoChat
Sign Up
PreprintCited by 4Speech and dialogue systemsTopic ModelingEthics and Social Impacts of AI

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

Open access
Liu, Ryan, Sumers, Theodore R., Dasgupta, Ishita, Griffiths, Thomas L.·2024·arXiv
DOI 10.48550/arxiv.2402.07282arXiv 2402.07282
Open this paper in CoChat — free accountOpen PDF

Abstract

In day-to-day communication, people often approximate the truth - for example, rounding the time or omitting details - in order to be maximally helpful to the listener. How do large language models (LLMs) handle such nuanced trade-offs? To address this question, we use psychological models and experiments designed to characterize human behavior to analyze LLMs. We test a range of LLMs and explore how optimization for human preferences or inference-time reasoning affects these trade-offs. We find that reinforcement learning from human feedback improves both honesty and helpfulness, while chain-of-thought prompting skews LLMs towards helpfulness over honesty. Finally, GPT-4 Turbo demonstrates human-like response patterns including sensitivity to the conversational framing and listener's decision context. Our findings reveal the conversational values internalized by LLMs and suggest that even these abstract values can, to a degree, be steered by zero-shot prompting.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.07282

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several different notions of honesty are in use in RLHF fine-tuning approaches to LLMs, which are often conflated with truthfulness.

    “Several different notions of honesty are in use in RLHF fine-tuning approaches to LLMs, which are often conflated with ‘truthfulness’ (e.g. in the introduction of Liu et al., 2024).”

    From Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback (2025) · cites this paper as Liu et al., 2024

    The Introduction equates “honesty” with “truthfulness,” and §3 says measures of honesty in LLM research range from broad principles to narrow requirements for external-source support. The paper also discusses RLHF (§2.1), but these passages do not establish that the different notions are specifically in use in RLHF fine-tuning approaches. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. · arXiv
  2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. · arXiv
  3. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. · arXiv
  4. Benz, A. Utility and relevance of answers. In Game theory and pragmatics, pp. 195-219. Springer, 2006.
  5. Binz, M. and Schulz, E. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, 2023.
  6. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. · arXiv
  7. Carston, R. and Wearing, C. Metaphor, hyperbole and simile: A pragmatic approach. Language and Cognition, 3(2):283-312, 2011.
  8. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
  9. Cooper, B., Cohen, T. R., Huppert, E., Levine, E. E., and Fleeson, W. Honest behavior: Truth-seeking, belief- speaking, and fostering understanding of the truth in oth- ers. Academy of Management Annals, 17(2):655-683, 2023.
  10. Dale, R. and Reiter, E. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233-263, 1995.
  11. Dasgupta, I., Lampinen, A. K., Chan, S. C., Creswell, A., Kumaran, D., McClelland, J. L., and Hill, F. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051, 2022. · arXiv
  12. Durmus, E., Nyugen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., et al. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388, 2023. · arXiv
  13. Frank, M. C. and Goodman, N. D. Predicting pragmatic reasoning in language games. Science, 334(6084):998- 998, 2012.
  14. Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., and Darrell, T. Speaker-follower models for vision- and-language navigation. Advances in Neural Informa- tion Processing Systems, 31, 2018.
  15. Fried, D., Tomlin, N., Hu, J., Patel, R., and Nematzadeh, A. Pragmatics in language grounding: Phenomena, tasks, and modeling approaches. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12619- 12640, 2023.
  16. Ganguli, D., Askell, A., Schiefer, N., Liao, T., Lukošiūtė, K., Chen, A., Goldie, A., Mirhoseini, A., Olsson, C., Hernan- dez, D., et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459, 2023. · arXiv
  17. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020. · arXiv
  18. Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. · arXiv
  19. Goodman, N. D. and Frank, M. C. Pragmatic language inter- pretation as probabilistic inference. Trends in Cognitive Sciences, 20(11):818-829, 2016.
  20. Grice, H. P. Meaning. Philosophical Review, 66(3):377- 388, 1957.
  21. Grice, H. P. Logic and conversation. In Speech acts, pp. 41-58. Brill, 1975.
  22. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pp. 3929- 3938, 2020.
  23. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
  24. Johnson-Laird, P. N., Legrenzi, P., and Legrenzi, M. S. Rea- soning and a sense of reality. British Journal of Psychol- ogy, 63(3):395-400, 1972.
  25. Kao, J. T., Wu, J. Y., Bergen, L., and Goodman, N. D. Nonliteral understanding of number words. Proceedings of the National Academy of Sciences, 111(33):12002- 12007, 2014.
  26. Kasirzadeh, A. and Gabriel, I. In conversation with artifi- cial intelligence: aligning language models with human values. Philosophy & Technology, 36(2):1-24, 2023.
  27. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35:22199- 22213, 2022.
  28. Komeili, M., Shuster, K., and Weston, J. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8460-8478, 2022.
  29. Lewis, D. K. Convention: A Philosophical Study. John Wiley & Sons, 1969.
  30. Liang, P. et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
  31. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. · arXiv
  32. Lipkin, B., Wong, L., Grand, G., and Tenenbaum, J. B. Eval- uating statistical language models as pragmatic reasoners. arXiv preprint arXiv:2305.01020, 2023. · arXiv
  33. Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell- Gillingham, L., Irving, G., et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. · arXiv
  34. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with hu- man feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
  35. Nguyen, K. Language models are pragmatic speakers. arXiv preprint arXiv:2305.17760, 2023. · arXiv
  36. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
  37. Parikh, P. A game-theoretic account of implicature. In Proceedings of the 4th conference on Theoretical aspects of reasoning about knowledge, pp. 85-94, 1992.
  38. Popa-Wyatt, M. Embedding irony and the seman- tics/pragmatics distinction. Inquiry, 62(6):674-699, 2019.
  39. Prystawski, B. and Goodman, N. D. Why think step-by- step? reasoning emerges from the locality of experience. arXiv preprint arXiv:2304.03843, 2023. · arXiv
  40. Ruis, L., Khan, A., Biderman, S., Hooker, S., Rocktäschel, T., and Grefenstette, E. Large language models are not zero-shot communicators. arXiv preprint arXiv:2210.14986, 2022. · arXiv
  41. Shanahan, M. Talking about large language models. Com- munications of the ACM, 67(2):68-79, 2024.
  42. Sravanthi, S. L., Doshi, M., Kalyan, T. P., Murthy, R., Bhat- tacharyya, P., and Dabre, R. Pub: A pragmatics un- derstanding benchmark for assessing LLMs' pragmatics capabilities. arXiv preprint arXiv:2401.07078, 2024. · arXiv
  43. Srivastava, A. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
  44. Sumers, T. R., Ho, M. K., Griffiths, T. L., and Hawkins, R. D. Reconciling truthfulness and relevance as epistemic and decision-theoretic utility. Psychological Review, 2023.
  45. Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT Press, 2018.
  46. Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., and Ganguli, D. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689, 2023. · arXiv
  47. Tendahl, M. and Gibbs Jr, R. W. Complementary perspec- tives on metaphor: Cognitive linguistics and relevance theory. Journal of Pragmatics, 40(11):1823-1864, 2008.
  48. Tessler, M. H. and Goodman, N. D. The language of gener- alization. Psychological Review, 126(3):395, 2019.
  49. Van Der Henst, J.-B., Carles, L., and Sperber, D. Truthful- ness and relevance in telling the time. Mind & Language, 17(5):457-466, 2002.
  50. van Rooij, R. Questioning to resolve decision problems. Linguistics and Philosophy, 26(6):727-763, 2003.
  51. van Rooij, R. and Schulz, K. Generics and typicality: a bounded rationality approach. Linguistics and Philoso- phy, 43(1):83-117, 2020.
  52. Vélez, N., Chen, A. M., Burke, T., Cushman, F. A., and Gershman, S. J. Teachers recruit mentalizing regions to represent learners' beliefs. Proceedings of the National Academy of Sciences, 120(22):e2215015120, 2023.
  53. Wang, H. A pragmatic analysis of white lies from the stand- point of speakers. Proceedings of the 3rd International Conference on Culture, Education and Economic Devel- opment of Modern Society (ICCESE 2019), 2019.
  54. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24837, 2022.
  55. Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. Taxonomy of risks posed by lan- guage models. In Proceedings of the 2022 ACM Confer- ence on Fairness, Accountability, and Transparency, pp. 214-229, 2022.
  56. Wilson, D. and Sperber, D. Truthfulness and relevance. Mind, 111(443):583-632, 2002.
  57. Wilson, D. and Sperber, D. Relevance theory. The handbook of pragmatics, pp. 606-632, 2006.

Cited by

Sorted by citations

Metadata

Authors
Liu, R., Sumers, T. R., Dasgupta, I., & Griffiths, T. L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.07282
arXiv
2402.07282
OpenAlex
W4391832888
Cited by
1
References
1
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Defense Science and Engineering Graduate
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
large language models, truth approximation, reinforcement learning from human feedback, chain-of-thought prompting, zero-shot prompting, GPT-4 Turbo, human preferences

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review