CoChat

A pilot study of measuring emotional response and perception of LLM-generated questionnaire and human-generated questionnaires

Open access
Zou, Zhao, Mubin, Omar, Alnajjar, Fady, Ali, Luqman · 2024 · Scientific Reports
Cited by 45

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

The advent of ChatGPT has sparked a heated debate surrounding natural language processing technology and AI-powered chatbots, leading to extensive research and applications across various disciplines. This pilot study aims to investigate the impact of ChatGPT on users' experiences by administering two distinct questionnaires, one generated by humans and the other by ChatGPT, along with an Emotion Detecting Model. A total of 14 participants (7 female and 7 male) aged between 18 and 35 years were recruited, resulting in the collection of 8672 ChatGPT-associated data points and 8797 human-associated data points. Data analysis was conducted using Analysis of Variance (ANOVA). The results indicate that the utilization of ChatGPT enhances participants' happiness levels and reduces their sadness levels. While no significant gender influences were observed, variations were found about specific emotions. It is important to note that the limited sample size, narrow age range, and potential cultural impacts restrict the generalizability of the findings to a broader population. Future research directions should explore the impact of incorporating additional language models or chatbots on user emotions, particularly among specific age groups such as older individuals and teenagers. As one of the pioneering works evaluating the human perception of ChatGPT text and communication, it is noteworthy that ChatGPT received positive evaluations and demonstrated effectiveness in generating extensive questionnaires.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-024-53255-1

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Research works printed but not linked: 16

  1. Ghosal, S. ChatGPT on Characteristic Mode Analysis. TechRxiv. https://doi.org/10.36227/techrxiv.21900342.v1 (2023). · DOI
  2. Yenduri, G. et al. GPT (Generative Pre-trained Transformer)–A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (Springer, 2022).
  3. Karaali, G. Artificial intelligence, basic skills, and quantitative literacy. Numeracy 16(1), 9 (2023). · DOI · Paper page
  4. Mijwil, M. & Aljanabi, M. Towards artificial intelligence-based cybersecurity: The practices and ChatGPT generated ways to combat cybercrime. Iraqi J. Comput. Sci. Math. 4(1), 65–70 (2023).
  5. Aydın, Ö. & Karaarslan, E. Is ChatGPT Leading Generative AI? What is Beyond Expectations? What is Beyond Expectations (Springer, 2023).
  6. Meng, J. & Dai, Y. Emotional support from AI chatbots: Should a supportive partner self-disclose or not?. J. Comput.-Mediat. Commun. 26(4), 207–222 (2021). · DOI · Paper page
  7. Uludag, K. Testing Creativity of ChatGPT in Psychology: Interview with ChatGPT. SSRN 4390872 (2023). · DOI
  8. Xu, Q., Yan, J., & Cao, C. Emotional communication between chatbots and users: An empirical study on online customer service system. In Artificial Intelligence in HCI: 3rd International Conference, AI-HCI 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26–July 1, 2022, Proceedings 513–530 (Springer International Publishing, 2022). · DOI
  9. Yenduri, G. et al. Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions. arXiv preprint arXiv:2305.10435 (2023). · arXiv
  10. Mindoro, J. N., Pilueta, N. U., Austria, Y. D., Lacatan, L. L., & Dellosa, R. M. (2020, August). Capturing students’ attention through visible behavior: A prediction utilizing YOLOv3 approach. In 2020 11th IEEE control and system graduate research colloquium (ICSGRC) (pp. 328–333). IEEE. · DOI
  11. Nepal, U. & Eslamiat, H. Comparing YOLOv3, YOLOv4 and YOLOv5 for autonomous landing spot detection in faulty UAVs. Sensors 22(2), 464 (2022). · DOI · Paper page
  12. Jiang, Y. COTS recognition and detection based on Improved YOLO v5 model. In 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP) 830–833 (IEEE, 2022). · DOI
  13. Trabelsi, Z., Alnajjar, F., Parambil, M. M. A., Gochoo, M. & Ali, L. Real-time attention monitoring system for classroom: A deep learning approach for student’s behavior recognition. Big Data Cogn. Comput. 7(1), 48 (2023). · DOI · Paper page
  14. Horvat, M., Kukolja, D. & Ivanec, D. Comparing affective responses to standardized pictures and videos: A study report. arXiv preprint arXiv:1505.07398 (2015). · arXiv
  15. Wallbott, H. G. & Scherer, K. R. Assessing emotion by questionnaire. In The Measurement of Emotions 1989 Jan 1 55–82 (Academic Press, 2022). · DOI
  16. Forni-Santos, L. & Osório, F. L. Influence of gender in the recognition of basic facial expressions: A critical literature review. World J. Psychiatry 5(3), 342 (2015). · DOI · Paper page

Cited by

See all 45

Metadata

Authors
Zou, Z., Mubin, O., Alnajjar, F., & Ali, L.
Venue
Scientific Reports
Published
2024-02-02
Volume
14
Issue
1
Pages
2781-2781
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4391474613
Cited by
45
References
8
FWCI
18.19
Type
journal article
Language
en
Open access
gold
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A pilot study of measuring emotional response and perception of LLM-generated questionnaire and human-generated questionnaires. https://app.cochat.ai/papers/10.1038/s41598-024-53255-1
The paper
Zou, Z., Mubin, O., Alnajjar, F., & Ali, L. (2024). A pilot study of measuring emotional response and perception of LLM-generated questionnaire and human-generated questionnaires. Scientific Reports. https://doi.org/10.1038/s41598-024-53255-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.