Evaluating Psychological Safety of Large Language Models
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
In this work, we designed unbiased prompts to systematically evaluate the psychological safety of large language models (LLMs).First, we tested five different LLMs by using two personality tests: Short Dark Triad (SD-3) and Big Five Inventory (BFI).All models scored higher than the human average on SD-3, suggesting a relatively darker personality pattern.Despite being instruction fine-tuned with safety metrics to reduce toxicity, InstructGPT, GPT-3.5, and GPT-4 still showed dark personality patterns; these models scored higher than self-supervised GPT-3 on the Machiavellianism and narcissism traits on SD-3.Then, we evaluated the LLMs in the GPT series by using well-being tests to study the impact of fine-tuning with more training data.We observed a continuous increase in the well-being scores of GPT models.Following these observations, we showed that finetuning Llama-2-chat-7B with responses from BFI using direct preference optimization could effectively reduce the psychological toxicity of the model.Based on the findings, we recommended the application of systematic and comprehensive psychological metrics to further evaluate and improve the safety of LLMs.Our code is available at https://github.com/DAMO- NLP-SG/PsychSafety.Warning: This paper contains examples with potentially harmful content.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Utilizing the 4,318 DPO question–answer pairs, we fine-tuned the Llama-2-chat-7B model using DPO with LoRA (Hu et al., 2021), resulting in the creation of a new model named P-Llama-2-chat-7B.”
MethodSelf-citationTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationIntroducing the Short Dark Triad (SD3): A Brief Measure of Dark Personality Traits
Jones, D. N., & Paulhus, D. L. (2014) · Assessment · doi:10.1177/1073191113514105 · Cited by 2,162
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Self-citationRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · cited 1×
“It has been shown that LLMs can be sensitive to the order, format and wordings of the input prompt (Lu et al., 2022; Zhao et al., 2021).”
BackgroundThe Dark Triad, happiness and subjective well-being
Egan, V., Chan, S., & Shorter, G. W. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.01.004 · Cited by 165 · cited 1×
“We also devised an effective way to fine-tune LLMs with direct preference optimization (DPO) to return a more positive personality pattern. et al., 2017; Jonason et al., 2015; Hmieleski and Lerner, 2016; Egan et al., 2014; Kay and Saucier, 2020; Butler, 2015; Adler, 2017).”
MethodUnveiling the Implicit Toxicity in Large Language Models
Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., & Huang, M. (2023) · arXiv · doi:10.48550/arxiv.2311.17391 · Cited by 40
Cited by
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514
Is GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., & Bing, L. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.626 · Cited by 149
Self-citation
Metadata
- Authors
- Li, X., Li, Y., Qiu, L., Joty, S., & Bing, L.
- Venue
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Published
- 2024
- Pages
- 1826-1843
- OpenAlex
- W4404782670
- Cited by
- 41
- References
- 10
- FWCI
- 13.13
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating Psychological Safety of Large Language Models. https://app.cochat.ai/papers/10.18653/v1/2024.emnlp-main.108
- The paper
- Li, X., Li, Y., Qiu, L., Joty, S., & Bing, L. (2024). Evaluating Psychological Safety of Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2024.emnlp-main.108
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.