Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Open accessThe DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Collecting data is one of the bottlenecks of Human-Computer Interaction (HCI) research. Motivated by this, we explore the potential of large language models (LLMs) in generating synthetic user research data. We use OpenAI’s GPT-3 model to generate open-ended questionnaire responses about experiencing video games as art, a topic not tractable with traditional computational user models. We test whether synthetic responses can be distinguished from real responses, analyze errors of synthetic data, and investigate content similarities between synthetic and real data. We conclude that GPT-3 can, in this context, yield believable accounts of HCI experiences. Given the low cost and high speed of LLM data generation, synthetic data should be useful in ideating and piloting new experiments, although any findings must obviously always be validated with real data. The results also raise concerns: if employed by malicious users of crowdsourcing services, LLMs may make crowdsourcing of self-report data fundamentally unreliable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This work could serve as an example to support the promising future of AI-mediated healthcare communication [53,79].”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [53]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“LLM scaling laws predict that their performance will improve with new and even larger models [32, 66, 78], and the quality metrics of our Experiment 3 also indicate that larger scale yields more human-like data.”
BackgroundUMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101
Dimensionality reduction for visualizing single-cell data using UMAP
Becht, E., McInnes, L., Healy, J., Dutertre, C.-A., Kwok, I. W. H., Ng, L. G., Ginhoux, F., & Newell, E. W. (2019) · Nature Biotechnology · doi:10.1038/nbt.4314 · Cited by 5,358
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Colman (2022) · doi:10.1073/pnas · Cited by 4,370
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Prospect Theory: An Analysis of Decision Under Risk
Kahneman, D., & Tversky, A. (2013) · World Scientific Handbook in Financial Economics Series · doi:10.1142/9789814417358_0006 · Cited by 3,091
Detection theory: A user's guide
Lories, G. (1993) · Behavioural Processes · doi:10.1016/0376-6357(93)90083-4 · Cited by 2,835
hdbscan: Hierarchical density based clustering
McInnes, L., Healy, J., & Astels, S. (2017) · The Journal of Open Source Software · doi:10.21105/joss.00205 · Cited by 2,833
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Gorilla in our midst: An online behavioral experiment builder
Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020) · Behavior Research Methods · doi:10.3758/s13428-019-01237-x · Cited by 1,862
BERTopic: Neural topic modeling with a class-based TF-IDF procedure
Grootendorst, M. (2022) · arXiv · doi:10.48550/arxiv.2203.05794 · Cited by 1,423 · cited 1×
“completions to PROMPT 3) is based on the observation The embedding, dimensionality reduction, and clustering steps that GPT-3 can be prompted to perform a form of qualitative in- are the same as in the BerTopic topic mining approach [26].”
ResultEmergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]
Cambria, E., & White, B. (2014) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2014.2307227 · Cited by 1,102
Contrastive Representation Learning: A Framework and Review
Le-Khac, P. H., Healy, G., & Smeaton, A. F. (2020) · IEEE Access · doi:10.1109/access.2020.3031549 · Cited by 862
On the Number of Linear Regions of Deep Neural Networks
Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1402.1869 · Cited by 840
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · cited 2×
“…known to make factual errors, but can generate text in many literary styles [8], perform sentiment analysis [64], generate human-like self-reports of emotion [76], and predict how political views afect voting behavior and which words people associate with members of diferent political parties [2].”
BackgroundTwo decades of statistical language modeling: where do we go from here?
Rosenfeld, R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880083 · Cited by 735
Vanderwende, L., Daumé, H., & Kirchhoff, K. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 685
Bargas-Avila, J. A., & Hornbæk, K. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979336 · Cited by 660 · cited 1×
“, [3, 61, 72]), but collecting such data can be slow and expensive.”
BackgroundExtending Fitts' law to two-dimensional tasks
MacKenzie, I. S., & Buxton, W. (1992) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '92 · doi:10.1145/142750.142794 · Cited by 617
How many words do we read per minute? A review and meta-analysis of reading rate
Brysbaert, M. (2019) · Journal of Memory and Language · doi:10.1016/j.jml.2019.104047 · Cited by 553
AI-synthesized faces are indistinguishable from real faces and more trustworthy
Nightingale, S. J., & Farid, H. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120481119 · Cited by 397
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 2×
“LLM representations have also been observed to encode emotion and sentiment [36, 64].”
BackgroundEnsuring survey research data integrity in the era of internet bots
Griffin, M., Martino, R. J., LoSchiavo, C., Comer-Carruthers, C., Krause, K. D., Stults, C. B., & Halkitis, P. N. (2022) · Quality & Quantity · doi:10.1007/s11135-021-01252-1 · Cited by 314
Sauro, J., & Lewis, J. R. (2016) · Quantifying the User Experience · doi:10.1016/b978-0-12-802308-2.00002-3 · Cited by 312
Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.11990 · Cited by 286
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 4×
“LLM scaling laws predict that their performance will improve with new and even larger models [32, 66, 78], and the quality metrics of our Experiment 3 also indicate that larger scale yields more human-like data.”
BackgroundMethodInfluentialSocial Simulacra: Creating Populated Prototypes for Social Computing Systems
Park, J. S., Popowski, L., Cai, C., Morris, M. R., Liang, P., & Bernstein, M. S. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545616 · Cited by 212
An optimal frequency domain filter for edge detection in digital pictures
(1979) · NCHU Institutional Repository CRIS · doi:10.1109/tpami · Cited by 198
Fooled twice – people cannot detect deepfakes but think they can
Köbis, N., Doležalová, B., & Soraperra, I. (2021) · doi:10.2139/ssrn.3832978 · Cited by 194
Fooled twice: People cannot detect deepfakes but think they can
Köbis, N. C., Doležalová, B., & Soraperra, I. (2021) · iScience · doi:10.1016/j.isci.2021.103364 · Cited by 193
Transformers as Soft Reasoners over Language
Clark, P., Tafjord, O., & Richardson, K. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/537 · Cited by 176
A Bermuda Triangle?: A Review of Method Application and Triangulation in User Experience Evaluation
Pettersson, I., Lachner, F., Frison, A.-K., Riener, A., & Butz, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174035 · Cited by 168
Visual Exploration of Semantic Relationships in Neural Word Embeddings
Liu, S., Bremer, P., Thiagarajan, J. J., Srikumar, V., Wang, B., Livnat, Y., & Pascucci, V. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2745141 · Cited by 139
Touchscreen Typing As Optimal Supervisory Control
Jokinen, J., Acharya, A., Uzair, M., Jiang, X., & Oulasvirta, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445483 · Cited by 124
Frozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113 · cited 1×
“There is evidence that next token prediction can even result in computational operations and procedures that generalize to other data domains such as image understanding [45].”
BackgroundJukebox: A Generative Model for Music
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00341 · Cited by 96
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022) · arXiv · doi:10.48550/arxiv.2201.02177 · Cited by 83
On Accurate Evaluation of GANs for Language Generation
Semeniuta, S., Severyn, A., & Gelly, S. (2018) · arXiv · doi:10.48550/arxiv.1806.04936 · Cited by 76
Is this a useful instrument? An introduction to Rasch measurement models
(2020) · UpSpace Institutional Repository (University of Pretoria) · doi:10.17605/osf.io · Cited by 67
Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning
Cheema, N., Frey-Law, L. A., Naderi, K., Lehtinen, J., Slusallek, P., & Hämäläinen, P. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376701 · Cited by 66
Self-citationDelobelle, P., Tokpo, E., Calders, T., & Berendt, B. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.122 · Cited by 57
Oulasvirta, A., Kristensson, P. O., Bi, X., Howes, A., et al. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198799603.001.0001 · Cited by 53
User language model for collaborative personalized search
Xue, G.-R., Han, J., Yu, Y., & Yang, Q. (2009) · ACM Transactions on Information Systems · doi:10.1145/1462198.1462203 · Cited by 46
Predicting Game Difficulty and Churn Without Players
Roohi, S., Relas, A., Takatalo, J., Heiskanen, H., & Hämäläinen, P. (2020) · Proceedings of the Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3410404.3414235 · Cited by 34
Self-citationLanguage Models Can Generate Human-Like Self-Reports of Emotion
Tavast, M., Kunnari, A., & Hämäläinen, P. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490100.3516464 · Cited by 34
Self-citationA lingering question addressed: Reading rate and most efficient listening rate are highly similar.
Kuperman, V., Kyröläinen, A.-J., Porretta, V., Brysbaert, M., & Yang, S. (2021) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0000932 · Cited by 29
"My Soul Got a Little Bit Cleaner": Art Experience in Videogames
Bopp, J. A., Vornhagen, J. B., & Mekler, E. D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3474664 · Cited by 27
It's time to rediscover HCI models
Oulasvirta, A. (2019) · interactions · doi:10.1145/3330340 · Cited by 26
ProGen: Language Modeling for Protein Generation
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 26
Neural Language Models as What If? -Engines for HCI Research
Hämäläinen, P., Tavast, M., & Kunnari, A. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490100.3516458 · Cited by 12
Self-citationCrowdsourcing in observational and experimental research
Zallot, C., Paolacci, G., Chandler, J., & Sisso, I. (2021) · Routledge eBooks · doi:10.4324/9781003025245-12 · Cited by 10
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPPreprint
Ye, Q., Lin, B. Y., & Ren, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08835 · Cited by 2
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J., Rytting, C., & Wingate, D. (2022) · arXiv · doi:10.48550/arxiv.2209.06899
Research works printed but not linked: 17
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
- 10.5555/3001460.3001507 · doi:10.5555/3001460.3001507
- Paul M Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement.Journal of experimental psychology 47, 6 (1954), 381–391.
- Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Vol. 30. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
- Mijin Kwon, Tor Wager, and Jonathan Phillips. 2022. Representations of emotion concepts: Comparison across pairwise, appraisal feature-based, and word embedding-based similarity spaces. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 44.
- Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. 2019. Improved Precision and Recall Metric for Assessing Generative Models. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2019/file/0234c510bc6d908b28c70ff313743079-Paper.pdf
- Guillaume Lample and François Charton. 2019. Deep Learning For Symbolic Mathematics. In International Conference on Learning Representations.
- Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning. PMLR, 6565–6576.
- Ziming Liu Ouail Kitouni Niklas Nolte Eric J Michaud Max Tegmark and Mike Williams. 2022. Towards Understanding Grokking: An Effective Theory of Representation Learning. In Advances in Neural Information Processing Systems.
- Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. 2021. Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.). Vol. 34. Curran Associates, Inc., 25192–25204. https://proceedings.neurips.cc/paper/2021/file/d3e2e8f631bd9336ed25b8162aef8782-Paper.pdf
- Jorge Pérez, Pablo Barceló, and Javier Marinkovic. 2021. Attention is Turing-Complete.J. Mach. Learn. Res. 22, 75 (2021), 1–35.
- Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. 2016. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems 29 (2016), 3360–3368.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8821–8831. https://proceedings.mlr.press/v139/ramesh21a.html
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008.
- Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 12697–12706. https://proceedings.mlr.press/v139/zhao21c.html
Cited by
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
“Other empirical studies have replicated existing social science studies [46], political surveys [92], and generated synthetic data [39].”
BackgroundGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 596
Wang, A., Morgenstern, J., & Dickerson, J. P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00986-z · Cited by 528
Quantifying the Persona Effect in LLM Simulations
Hu, T., & Collier, N. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.554 · Cited by 326
Metadata
- Authors
- Hämäläinen, P., Tavast, M., & Kunnari, A.
- Affiliations
- Aalto University (FI)
- University of Helsinki (FI)
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-19
- OpenAlex
- W4366549000
- Semantic Scholar
- 0ffd57884d7957f6b5634b9fa24843dc3759668f
- Cited by
- 248
- References
- 46
- Citations by year
- 2026: 48 · 2025: 98 · 2024: 86 · 2023: 16
- FWCI
- 74.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- Työväentutkimus Vuosikirja (submittedVersion)
- Funders
- Academy of Finland (318937)
- European Commission Horizon 2020 FET Proactive program (101017779)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, synthetic data generation, GPT-3, survey responses, video games as art, self-reported data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.