A pilot study of measuring emotional response and perception of LLM-generated questionnaire and human-generated questionnaires
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
The advent of ChatGPT has sparked a heated debate surrounding natural language processing technology and AI-powered chatbots, leading to extensive research and applications across various disciplines. This pilot study aims to investigate the impact of ChatGPT on users' experiences by administering two distinct questionnaires, one generated by humans and the other by ChatGPT, along with an Emotion Detecting Model. A total of 14 participants (7 female and 7 male) aged between 18 and 35 years were recruited, resulting in the collection of 8672 ChatGPT-associated data points and 8797 human-associated data points. Data analysis was conducted using Analysis of Variance (ANOVA). The results indicate that the utilization of ChatGPT enhances participants' happiness levels and reduces their sadness levels. While no significant gender influences were observed, variations were found about specific emotions. It is important to note that the limited sample size, narrow age range, and potential cultural impacts restrict the generalizability of the findings to a broader population. Future research directions should explore the impact of incorporating additional language models or chatbots on user emotions, particularly among specific age groups such as older individuals and teenagers. As one of the pioneering works evaluating the human perception of ChatGPT text and communication, it is noteworthy that ChatGPT received positive evaluations and demonstrated effectiveness in generating extensive questionnaires.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338 · cited 1×
“The model was trained using an open dataset called the AffectNet dataset 21 .”
MethodChatGPT User Experience: Implications for Education
Zhai, X. (2023) · doi:10.2139/ssrn.4312418 · Cited by 935
What ChatGPT and generative AI mean for science
Stokel-Walker, C., & Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-00340-6 · Cited by 828 · cited 1×
“Additionally, ChatGPT can generate text on a wide range of topics, making it a multipurpose tool for tasks such as writing, summarizing, and responding to user inquiries 17 .”
The benefits and challenges of chatgpt: An overview
Deng, J., & Lin, Y. (2023) · Frontiers in Computing and Intelligent Systems · doi:10.54097/fcis.v2i2.4465 · Cited by 611
OpenAI ChatGPT Generated Literature Review: Digital Twin in Healthcare
Aydın, Ö., & Karaarslan, E. (2022) · doi:10.2139/ssrn.4308687 · Cited by 391
Gender Differences in Emotional Response: Inconsistency between Experience and Expressivity
Deng, Y., Chang, L., Yang, M., Huo, M., & Zhou, R. (2016) · PLOS ONE · doi:10.1371/journal.pone.0158666 · Cited by 382 · cited 1×
“Additionally, a study conducted in 2016 by Deng et al. 16 explored gender differences in emotional expression and revealed that women often report more intense feelings.”
BackgroundAlshater, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4312358 · Cited by 304
Fischer, A. H., Kret, M. E., & Broekens, J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0190712 · Cited by 266 · cited 1×
“Our investigation was motivated by existing literature, such as the work of Fischer, Kret, and Broekens, which suggests that females tend to react more sensitively than males when detecting emotions 15 .”
Method
Research works printed but not linked: 16
- Ghosal, S. ChatGPT on Characteristic Mode Analysis. TechRxiv. https://doi.org/10.36227/techrxiv.21900342.v1 (2023). · DOI
- Yenduri, G. et al. GPT (Generative Pre-trained Transformer)–A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (Springer, 2022).
- Karaali, G. Artificial intelligence, basic skills, and quantitative literacy. Numeracy 16(1), 9 (2023). · DOI · Paper page
- Mijwil, M. & Aljanabi, M. Towards artificial intelligence-based cybersecurity: The practices and ChatGPT generated ways to combat cybercrime. Iraqi J. Comput. Sci. Math. 4(1), 65–70 (2023).
- Aydın, Ö. & Karaarslan, E. Is ChatGPT Leading Generative AI? What is Beyond Expectations? What is Beyond Expectations (Springer, 2023).
- Meng, J. & Dai, Y. Emotional support from AI chatbots: Should a supportive partner self-disclose or not?. J. Comput.-Mediat. Commun. 26(4), 207–222 (2021). · DOI · Paper page
- Uludag, K. Testing Creativity of ChatGPT in Psychology: Interview with ChatGPT. SSRN 4390872 (2023). · DOI
- Xu, Q., Yan, J., & Cao, C. Emotional communication between chatbots and users: An empirical study on online customer service system. In Artificial Intelligence in HCI: 3rd International Conference, AI-HCI 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26–July 1, 2022, Proceedings 513–530 (Springer International Publishing, 2022). · DOI
- Yenduri, G. et al. Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions. arXiv preprint arXiv:2305.10435 (2023). · arXiv
- Mindoro, J. N., Pilueta, N. U., Austria, Y. D., Lacatan, L. L., & Dellosa, R. M. (2020, August). Capturing students’ attention through visible behavior: A prediction utilizing YOLOv3 approach. In 2020 11th IEEE control and system graduate research colloquium (ICSGRC) (pp. 328–333). IEEE. · DOI
- Nepal, U. & Eslamiat, H. Comparing YOLOv3, YOLOv4 and YOLOv5 for autonomous landing spot detection in faulty UAVs. Sensors 22(2), 464 (2022). · DOI · Paper page
- Jiang, Y. COTS recognition and detection based on Improved YOLO v5 model. In 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP) 830–833 (IEEE, 2022). · DOI
- Trabelsi, Z., Alnajjar, F., Parambil, M. M. A., Gochoo, M. & Ali, L. Real-time attention monitoring system for classroom: A deep learning approach for student’s behavior recognition. Big Data Cogn. Comput. 7(1), 48 (2023). · DOI · Paper page
- Horvat, M., Kukolja, D. & Ivanec, D. Comparing affective responses to standardized pictures and videos: A study report. arXiv preprint arXiv:1505.07398 (2015). · arXiv
- Wallbott, H. G. & Scherer, K. R. Assessing emotion by questionnaire. In The Measurement of Emotions 1989 Jan 1 55–82 (Academic Press, 2022). · DOI
- Forni-Santos, L. & Osório, F. L. Influence of gender in the recognition of basic facial expressions: A critical literature review. World J. Psychiatry 5(3), 342 (2015). · DOI · Paper page
Cited by
Sai, S., Yashvardhan, U., Chamola, V., & Sikdar, B. (2024) · IEEE Access · doi:10.1109/access.2024.3385107 · Cited by 125
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
Huang, Y., Wu, R., He, J., & Xiang, Y. (2024) · Journal of Global Health · doi:10.7189/jogh.14.04070 · Cited by 72
Conceptualizing generative AI as style engines: Application archetypes and implications
Riemer, K., & Peter, S. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102824 · Cited by 63
Metadata
- Authors
- Zou, Z., Mubin, O., Alnajjar, F., & Ali, L.
- Venue
- Scientific Reports
- Published
- 2024-02-02
- Volume
- 14
- Issue
- 1
- Pages
- 2781-2781
- ISSN
- 2045-2322 (ISSN-L)
- OpenAlex
- W4391474613
- Cited by
- 45
- References
- 8
- FWCI
- 18.19
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A pilot study of measuring emotional response and perception of LLM-generated questionnaire and human-generated questionnaires. https://app.cochat.ai/papers/10.1038/s41598-024-53255-1
- The paper
- Zou, Z., Mubin, O., Alnajjar, F., & Ali, L. (2024). A pilot study of measuring emotional response and perception of LLM-generated questionnaire and human-generated questionnaires. Scientific Reports. https://doi.org/10.1038/s41598-024-53255-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.