CoChat
Sign Up
PreprintCited by 1

ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models

Jentzsch, Sophie, Kersting, Kristian·2023·arXiv
DOI 10.48550/arxiv.2306.04563arXiv 2306.04563
Open this paper in CoChat — free account

Abstract

Humor is a central aspect of human communication that has not been solved for artificial agents so far. Large language models (LLMs) are increasingly able to capture implicit and contextual information. Especially, OpenAI's ChatGPT recently gained immense public attention. The GPT3-based model almost seems to communicate on a human level and can even tell jokes. Humor is an essential component of human communication. But is ChatGPT really funny? We put ChatGPT's sense of humor to the test. In a series of exploratory experiments around jokes, i.e., generation, explanation, and detection, we seek to understand ChatGPT's capability to grasp and reproduce human humor. Since the model itself is not accessible, we applied prompt-based experiments. Our empirical evidence indicates that jokes are not hard-coded but mostly also not newly generated by the model. Over 90% of 1008 generated jokes were the same 25 Jokes. The system accurately explains valid jokes but also comes up with fictional explanations for invalid jokes. Joke-typical characteristics can mislead ChatGPT in the classification of jokes. ChatGPT has not solved computational humor yet but it can be a big leap toward "funny" machines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.04563

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ChatGPT demonstrates impressive capabilities in NLP tasks but falls short in generating humorous responses.

    “They found that while ChatGPT demonstrates impressive capabilities in NLP tasks, it falls short in generating humorous responses.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [84]

  • LLMs face difficulties in capturing the subtleties and context-dependent nature of humor.

    “This study emphasizes the importance of humor in human communication and the difficulties that LLMs face in capturing the subtleties and context-dependent nature of humor.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [84]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 23

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
  2. Xu Guo, Han Yu, Boyang Li, Hao Wang, Pengwei Xing, Siwei Feng, Zaiqing Nie, and Chunyan Miao. 2022. Federated learning for personalized humor recognition. ACM Transactions on Intelligent Sys- tems and Technology (TIST), 13(4):1-18.
  3. Md Kamrul Hasan, Sangwu Lee, Wasifur Rah- man, Amir Zadeh, Rada Mihalcea, Louis-Philippe Morency, and Ehsan Hoque. 2021. Humor knowl- edge enriched transformer for understanding mul- timodal humor. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 12972-12980.
  4. Sophie Jentzsch, Patrick Schramowski, Constantin Rothkopf, and Kristian Kersting. 2019. Semantics derived automatically from language corpora con- tain human-like moral choices. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 37-44.
  5. Sophie Jentzsch and Cigdem Turan. 2022. Gender bias in bert-measuring and analysing biases through sentiment rating in a realistic downstream classifica- tion task. In Proceedings of the 4th Workshop on Gen- der Bias in Natural Language Processing (GeBNLP), pages 184-199.
  6. Yuta Kayatani, Zekun Yang, Mayu Otani, Noa Gar- cia, Chenhui Chu, Yuta Nakashima, and Haruo Take- mura. 2021. The laughing machine: Predicting hu- mor in video. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pages 2073-2082.
  7. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language un- derstanding. In Proceedings of NAACL-HLT, pages 4171-4186.
  8. JA Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. Semeval 2021 task 7: Hahackathon, detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 105- 119.
  9. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
  10. Victor Raskin. 1979. Semantic mechanisms of hu- mor. In Annual Meeting of the Berkeley Linguistics Society, volume 5, pages 325-335.
  11. G Ritchie, R Manurung, H Pain, and D O'Mara. 2006. Computational humor. IEEE intelligent sys- tems, 21(2):59-69.
  12. Brandon M Savage, Heidi L Lujan, Raghavendar R Thipparthi, and Stephen E DiCarlo. 2017. Humor, laughter, learning, and health! a brief review. Ad- vances in physiology education.
  13. Patrick Schramowski, Cigdem Turan, Nico Ander- sen, Constantin A Rothkopf, and Kristian Kersting. 2022. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258-268.
  14. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv e-prints, pages arXiv-1707.
  15. Oliviero Stock and Carlo Strapparava. 2005. Ha- hacronym: A computational humor system. In Pro- ceedings of the ACL Interactive Poster and Demon- stration Sessions, pages 113-116.
  16. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  18. Tony Veale. 2021. Your Wit is My Command: Build- ing AIs with a Sense of Humor. Mit Press.
  19. Jonas Wagner and Sina Zarrieß. 2022. Do gender neutral affixes naturally reduce gender bias in static word embeddings? In Proceedings of the 18th Con- ference on Natural Language Processing (KONVENS 2022), pages 88-97.
  20. Xingbo Wang, Yao Ming, Tongshuang Wu, Haipeng Zeng, Yong Wang, and Huamin Qu. 2021. Dehu- mor: Visual analytics for decomposing humor. IEEE Transactions on Visualization and Computer Graph- ics, 28(12):4609-4623.
  21. Orion Weller and Kevin Seppi. 2019. Humor de- tection: A transformer gets the last laugh. pages 3621-3625.
  22. Thomas Winters. 2021. Computers learning humor is no joke. Harvard Data Science Review, 3(2).
  23. Dolf Zillmann and Jennings Bryant. 1980. Misat- tribution theory of tendentious humor. Journal of Experimental Social Psychology, 16(2):146-160.

Cited by

Sorted by citations

Authors
Jentzsch, S., & Kersting, K.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2306.04563
arXiv
2306.04563
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review