The DOI matches its Crossref record.
Questioning the Survey Responses of Large Language Models
Open accessAbstract
Surveys have recently gained popularity as a tool to study large language models. By comparing survey responses of models to those of human reference populations, researchers aim to infer the demographics, political opinions, or values best represented by current language models. In this work, we critically examine this methodology on the basis of the well-established American Community Survey by the U.S. Census Bureau. Evaluating 43 different language models using de-facto standard prompting methodologies, we establish two dominant patterns. First, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter "A". Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses, irrespective of model size or pre-training data. As a result, in contrast to conjectures from prior work, survey-derived alignment measures often permit a simple explanation: models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform for any survey under consideration.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models have their own biases and are limited to analyzing only the textual content of the tweets, excluding images, videos, and linked websites.
“large language models have their own biases [20,44] and are limited to analyzing only the textual content of the tweets, excluding images, videos, and linked websites.”
From Twitter's Algorithm: Amplifying Anger, Animosity, and Affective Polarization (2023) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Still being checked.
Research works printed but not linked: 39
- Abid, A., Farooqi, M., and Zou, J. (2021). Persistent anti-muslim bias in large language models. In AAAI/ACM Conference on AI, Ethics, and Society, pages 298-306.
- Aher, G. V., Arriaga, R. I., and Kalai, A. T. (2023). Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337-371.
- AlKhamissi, B., ElNokrashy, M., Alkhamissi, M., and Diab, M. (2024). Investigating cultural alignment of large language models. In Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404-12422.
- Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., and Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337-351.
- Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Van Der Wal, O. (2023). Pythia: a suite for analyzing large language models across training and scaling. In International Conference on Machine Learning.
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877-1901.
- Cruz, A. F., Hardt, M., and Mendler-Dünner, C. (2024). Evaluating language models as risk scores. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Ding, F., Hardt, M., Miller, J., and Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems.
- Dominguez-Olmedo, R., Dorner, F. E., and Hardt, M. (2024). Training on the test task confounds evaluation and emergence. ArXiv preprint arXiv:2407.07890. · arXiv
- Dorner, F., Sühr, T., Samadi, S., and Kelava, A. (2023). Do personality tests generalize to large language models? In NeurIPS Workshop on Socially Responsible Language Modelling Research.
- Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. (2024). The llama 3 herd of models. ArXiv preprint arXiv:2407.21783. · arXiv
- Durmus, E., Nyugen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., et al. (2023). Towards measuring the representation of subjective global opinions in language models. ArXiv preprint arXiv:2306.16388. · arXiv
- Feng, S., Park, C. Y., Liu, Y., and Tsvetkov, Y. (2023). From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. Findings of the Association for Computational Linguistics.
- Groves, R., Fowler, F., Couper, M., Lepkowski, J., Singer, E., and Tourangeau, R. (2009). Survey Methodology. Wiley.
- Hartmann, J., Schwenzow, J., and Witte, M. (2023). The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation. ArXiv preprint arXiv:2301.01768. · arXiv
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In International Conference on Learning Representations.
- Jiang, H., Beeferman, D., Roy, B., and Roy, D. (2022). CommunityLM: Probing Partisan Worldviews from Language Models. In International Conference on Computational Linguistics.
- Kim, J. and Lee, B. (2023). AI-Augmented Surveys: Leveraging Large Language Models for Opinion Prediction in Nationally Representative Surveys. ArXiv preprint arxiv:2305.09620. · arXiv
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
- Lee, S., Peng, T.-Q., Goldberg, M. H., Rosenthal, S. A., Kotcher, J. E., Maibach, E. W., and Leiserowitz, A. (2023). Can large language models capture public opinion about global warming? an empirical assessment of algorithmic fidelity and bias. ArXiv preprint arXiv:2311.00217. · arXiv
- Li, T., Khashabi, D., Khot, T., Sabharwal, A., and Srikumar, V. (2020). Unqovering stereotyping biases via underspecified questions. In Findings of the Association for Computational Linguistics, pages 3475-3489.
- Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wang, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N., Khattab, O., Henderson, P., Huang, Q., Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. (2022). Holistic evaluation of language models. ArXiv preprint arxiv:2211.09110. · arXiv
- Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. (2022). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Annual Meeting of the Association for Computational Linguistics, volume 1, pages 8086-8098.
- Mao, A., Raman, N., Shu, M., Li, E., Yang, F., and Boyd-Graber, J. (2021). Eliciting bias in question answering models through ambiguity. In Workshop on Machine Reading for Question Answering, pages 92-99.
- Mellon, J., Bailey, J., Scott, R., Breckwoldt, J., Miori, M., and Schmedeman, P. (2022). Do ais know what the most important issue is? using language models to code open-text social survey responses at scale. SSRN Electronic Journal.
- Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, pages 2381-2391.
- Motoki, F., Pinho Neto, V., and Rodrigues, V. (2023). More human than human: Measuring chatgpt political bias. Available at SSRN 4372349.
- Nelson, L. K., Burk, D., Knudsen, M., and McCall, L. (2021). The future of coding: A comparison of hand-coding and three types of computer-assisted text analysis methods. Sociological Methods & Research, 50(1):202-237.
- OpenAI (2023). Gpt-4 technical report. ArXiv preprint arXiv:2303.08774. · arXiv
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems.
- Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., Jones, A., Chen, A., Mann, B., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Khundadze, G., Kernion, J., Landis, J., Kerr, J., Mueller, J., Hyun, J., Landau, J., Ndousse, K., Goldberg, L., Lovitt, L., Lucas, M., Sellitto, M., Zhang, M., Kingsland, N., Elhage, N., Joseph, N., Mercado, N., DasSarma, N., Rausch, O., Larson, R., McCandlish, S., Johnston, S., Kravec, S., El Showk, S., Lanham, T., Telleen-Lawton, T., Brown, T., Henighan, T., Hume, T., Bai, Y., Hatfield-Dodds, Z., Clark, J., Bowman, S. R., Askell, A., Grosse, R., Hernandez, D., Ganguli, D., Hubinger, E., Schiefer, N., and Kaplan, J. (2023). Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics, pages 13387-13434.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing.
- Robinson, J. and Wingate, D. (2023a). Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations.
- Robinson, J. and Wingate, D. (2023b). Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations.
- Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., and Pauly, M. (2023). The Self-Perception and Political Biases of ChatGPT. ArXiv preprint arXiv:2304.07333. · arXiv
- Sanders, N. E., Ulinich, A., and Schneier, B. (2023). Demonstrations of the potential of ai-based political issue polling. ArXiv preprint arXiv:2307.04781. · arXiv
- Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. (2023). Whose opinions do language models reflect? International Conference on Machine Learning.
- Scherrer, N., Shi, C., Feder, A., and Blei, D. (2024). Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36.
Cited by
Sorted by citations
Using LLMs for Market Research
Brand, J., Israeli, A., & Ngwe, D. (2023) · doi:10.2139/ssrn.4395751 · Cited by 190
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68
Milli, S., Carroll, M., Wang, Y., Pandey, S., & Dragan, A. D. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16941 · Cited by 200
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data
Dorner, F. E., Nastl, V. Y., & Hardt, M. (2024) · arXiv · doi:10.48550/arxiv.2410.13341 · Cited by 31
Self-citationMachine Bias. How Do Generative Language Models Answer Opinion Polls?
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2025) · Sociological Methods & Research · doi:10.1177/00491241251330582 · Cited by 48
Toward LLM-agent-based modeling of transportation systems: A conceptual framework
Liu, T., Yang, J., & Yin, Y. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100001 · Cited by 28
Ceron, T., Falk, N., Barić, A., Nikolaev, D., & Padó, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00710 · Cited by 23
Conversations at Scale: Robust AI-led Interviews with a Simple Open-Source PlatformPreprint
Geiecke, F., & Jaravel, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4974382 · Cited by 11
Vox Populi, Vox AI? Using Large Language Models to Estimate German Vote Choice
von der Heyde, L., Haensch, A., & Wenz, A. (2025) · Social Science Computer Review · doi:10.1177/08944393251337014 · Cited by 23
Exploring LLMs for Automated Generation and Adaptation of Questionnaires
Adhikari, D. M., Hartland, A., Weber, I. G., & Cannanure, V. K. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736606 · Cited by 9
Examining the Feasibility of Large Language Models as Survey Respondents
Kitadai, A., Ogawa, K., & Nishino, N. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825497 · Cited by 7
How do Generative Language Models Answer Opinion Polls?Preprint
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2024) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/r2pnb · Cited by 12
Suicide- and crisis-risk detection using large language models in mental-health chatbotsPreprint
Weber, S., Klebe, D., Wolf, L., Aeberli, C., Homan, S., Psathakis, N., Casanova, A., Macias, J. A. G., Sykstus, J., Li, J., et al. (2026) · medRxiv · doi:10.64898/2026.01.12.26343914 · Cited by 6
Beck, J., Kemeter, L. M., Dürrbeck, K., Abdalla, M. H. I., & Kreuter, F. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3528192 · Cited by 8
Assessing Bias in LLM-Generated Synthetic Datasets: The Case of German Voter BehaviorPreprint
von der Heyde, L., Haensch, A., & Wenz, A. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/97r8s · Cited by 5
A Technological Construction of Society
Gmyrek, P., Lutz, C., & Newlands, G. (2024) · International Labour Office eBooks · doi:10.54394/uqoq5153 · Cited by 4
A Reference Model for Empirically Comparing LLMs with Humans
Schneider, K. A., Fotrousi, F., & Wohlrab, R. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1109/icse-seis66351.2025.00018 · Cited by 3
Vox Populi, Vox AI? Using Language Models to Estimate German Public OpinionPreprint
von der Heyde, L., Haensch, A., & Wenz, A. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/8je9g · Cited by 3
Gmyrek, P., Lutz, C., & Newlands, G. (2024) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/8bc4n · Cited by 3
Question form Matters: Examining Trust in Government Through Open and Closed Survey Items
Bernhard-Harrer, J., & Pfaff, K. (2025) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smaf014 · Cited by 2
Prompting for Policy: Forecasting Macroeconomic Scenarios with Synthetic LLM Personas
Iadisernia, G., & Camassa, C. (2025) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3768292.3770385 · Cited by 2
Valuing time in silicon: Can large language models replicate human value of travel time
Yan, Y., Liu, T., & Yin, Y. (2026) · Travel Behaviour and Society · doi:10.1016/j.tbs.2026.101245 · Cited by 3
Albert Gatt, A. (2025) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199384655.013.896 · Cited by 2
Hwang, K., Kim, M., Kim, M. A., Park, C., Park, Y., Lee, C., Lim, J., & Oh, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129548 · Cited by 1
Can large language models mimic airline passenger preferences?
Voltes-Dorta, A., & Suau-Sánchez, P. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100034 · Cited by 1
Rojas, C. L., Genç, H. U., Bozzon, A., & Colombo, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790436 · Cited by 1
Boullineau, E., & Arciniegas, J. D. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7369759 · Cited by 1
Scale Development at a Scale: Exploring the Use of Synthetic Responses in Survey ResearchPreprint
Bonaretti, D., Barreto, T. S., Parra, A., & Catalano, B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4641661
The Impact of Reasoning AI on Consumer Purchases: Evidence from an Online Platform Field ExperimentPreprint
Yan, S., Zhong, H., Zhong, Z., Zhou, W., & Mehta, N. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5631050
Park, J. H., & Yang, J.-S. (2026) · Scientometrics · doi:10.1007/s11192-026-05571-8 · Cited by 2
Rozado, D., & Tetlock, P. E. (2026) · Theory and Society · doi:10.1007/s11186-026-09719-6
Informing AI Policy Assessment using Large-Scale Simulation of Interventions
Barnett, J., Kieslich, K., Helberger, N., & Diakopoulos, N. A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812334
Gil-Biraud, V. E., de Castro, P., & Merillas, O. F. (2026) · Ense�anza de las Ciencias Sociales · doi:10.1344/eccss2026.1.1
Drašković, N., Rešetar, M., & Vejzagić, V. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591963
Artificial Consensus: AI and Survey MeasurementPreprint
Cauthorn, T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7101198
Edmonds, B. (2026) · Springer proceedings in complexity · doi:10.1007/978-3-032-31712-4_9
Allocating Human Oversight in AI-Enabled AnalyticsPreprint
Ye, Z., Lyu, J., & Tao, R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6561778
Yan, P. S., Li, H., Chen, N., Li, L., & Chen, P. (2026) · Theory and Society · doi:10.1007/s11186-026-09735-6
Understanding user responses to incentives in shared micromobility with LLM-based digital twins
Guan, R., De, A., & Zhou, Y. (2026) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2026.105205
Beyond Human-like Responses: Evaluating the Implicit Personality Alignment of Large Language ModelsPreprint
HU, Q., Xie, F., Li, Z., Hu, X., Zhang, M., Shen, M., & Zhou, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7312181
LLMs Do Not Emulate PopulationsPreprint
Green, B. S., & Leo, G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7352138
Global Trends in Research Ethics in the Context of Artificial Intelligence
Resulbegoviq, H. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1232-1.ch012
The ESG Paradox: A Behavioural Study of Consumer Response to Firms’ ESG Engagement
Zhao, T. (2026) · International Journal of Economic Behavior and Organization · doi:10.11648/j.ijebo.20261403.13
Beyond Human-like Responses: Evaluating the Implicit Personality Alignment of Large Language ModelsPreprint
HU, Q., Xie, F., Li, Z., Hu, X., Zhang, M., Shen, M., & Zhou, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7468072
Ideology by Alphabet: Option Order and the Measurement of Machine Political PreferencesPreprint
Olah, T., Erdey, L., Tőkés, T., & Nádasi, L. (2026) · Research Square · doi:10.21203/rs.3.rs-11063104/v1
Nguyen, N. T., & Pham, V.-T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40581-4_9
Coarse-to-Fine Label Distribution Learning for High-Cardinality Survey Simulation
Li, J., Liu, H., Meng, Z., Qiu, C., & Gu, J. (2026) · International Journal of Web Services Research · doi:10.4018/ijwsr.424308
Just a Simulation? A Review of Large Language Models as Subjects in Experimental EconomicsPreprint
Kitadai, A., & Nishino, N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7613458
Synthetische Befragungsdaten mit Digital Twins: Wie LLMs belastbare Daten liefern
Stahl, F., & Reissner, L. (2026) · Springer eBooks · doi:10.1007/978-3-662-73474-2_7
Large language models for computational social science
Xu, F., Shao, C., Chen, L., Zeng, Q., Chen, Z., Sukiennik, N., Wang, J., Gao, C., Wang, H., Lian, J., et al. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02539-9 · Cited by 1
Metadata
- Authors
- Dominguez-Olmedo, R., Hardt, M., & Mendler-Dünner, C.
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 45850-45878
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415797819
- Cited by
- 48
- References
- 0
- Citations by year
- 2026: 26 · 2025: 12 · 2024: 6
- FWCI
- 57.61
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Hector Stiftung
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, survey responses, question answering, factual accuracy, hallucinations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
