ChatGPT as Research Scientist: Probing GPT’s capabilities as a Research Librarian, Research Ethicist, Data Generator, and Data Predictor
Open accessThe DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
How good a research scientist is ChatGPT? We systematically probed the capabilities of GPT-3.5 and GPT-4 across four central components of the scientific process: as a Research Librarian, Research Ethicist, Data Generator, and Novel Data Predictor, using psychological science as a testing field. In Study 1 (Research Librarian), unlike human researchers, GPT-3.5 and GPT-4 hallucinated, authoritatively generating fictional references 36.0% and 5.4% of the time, respectively, although GPT-4 exhibited an evolving capacity to acknowledge its fictions. In Study 2 (Research Ethicist), GPT-4 (though not GPT-3.5) proved capable of detecting violations like p-hacking in fictional research protocols, correcting 88.6% of blatantly presented issues, and 72.6% of subtly presented issues. In Study 3 (Data Generator), both models consistently replicated patterns of cultural bias previously discovered in large language corpora, indicating that ChatGPT can simulate known results, an antecedent to usefulness for both data generation and skills like hypothesis generation. Contrastingly, in Study 4 (Novel Data Predictor), neither model was successful at predicting new results absent in their training data, and neither appeared to leverage substantially new information when predicting more vs. less novel outcomes. Together, these results suggest that GPT is a flawed but rapidly improving librarian, a decent research ethicist already, capable of data generation in simple domains with known characteristics but poor at predicting novel patterns of empirical data to aid future experimentation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The findings of this study align with previous research across various disciplines showing that citation fabrication and bibliographic inaccuracies are common in LLM-generated outputs [7-9,14].”
From Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“That most models produced overgeneralizations reliably across time suggests that they may have incorporated an ‘algorithmic bias’ from the training data similar to the way models trained on linguistic data can acquire gender, racial, or political biases, by picking up human tendencies implicit in training text corpora [51–53].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings thus offer novel contributions to research on overgeneralization in science communication [17,18,54] and the shortcomings of LLMs in text summarization [48,52,55], revealing a subtle form of ‘hallucination’ that has not yet been documented in the literature.”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [52]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The rise of new deep learning architectures like the Transformer ( 1 ) has yielded models capable of an array of impressive tasks, from seamless human - like conversation to composing sym-phonies.”
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · cited 1×
“As just a few examples, machine learning has been used to discover new protein structures ( 3 ), to render complicated problems more tractable in quantum mechanics ( 4 ) and fluid dynamics ( 5 ), and to accelerate the retrosynthesis of organic molecules ( 6 ).”
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 3×
“For analysis, GPT’s estimates were treated as analogous to cosine similarity measures from word embedding research (38).”
Self-citationJost, J. T., Banaji, M. R., & Nosek, B. A. (2004) · Political Psychology · doi:10.1111/j.1467-9221.2004.00402.x · Cited by 2,883
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 1×
“One source of these issues is unquestionably poor statistical practices by generally well - intentioned scientists: Studies show that poor statistical practices are widely prevalent across scientific research (31, 32).”
Solving the quantum many-body problem with artificial neural networks
Carleo, G., & Troyer, M. (2017) · Science · doi:10.1126/science.aag2302 · Cited by 2,215
Planning chemical syntheses with deep neural networks and symbolic AI
Segler, M. H. S., Preuss, M., & Waller, M. P. (2018) · Nature · doi:10.1038/nature25978 · Cited by 1,847 · cited 1×
“As just a few examples, machine learning has been used to discover new protein structures ( 3 ), to render complicated problems more tractable in quantum mechanics ( 4 ) and fluid dynamics ( 5 ), and to accelerate the retrosynthesis of organic molecules ( 6 ).”
Hofmann, W., Gawronski, B., Gschwendner, T., Le, H., & Schmitt, M. (2005) · Personality and Social Psychology Bulletin · doi:10.1177/0146167205275613 · Cited by 1,642 · cited 2×
“Critically, accumulated evidence suggests that implicit attitudes—automatic associations held between attitude objects, typically measured by reaction - time tasks—are distinct from explicit attitudes captured in self - reports ( 49 ).”
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1×
“While capable of passing standardized tests in varied domains, these models are currently strikingly poor at solving even simple mathematical problems ( 15 ).”
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · cited 1×
“The search for relevant scientific discourse and evidence is a basic building block of the scientific process.”
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · cited 1×
“Large - scale studies have demonstrated limited replication of research in Psychology (24, 25), Economics (26), the Social Sciences more generally (27), and Medicine (28–30).”
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · cited 1×
“One source of these issues is unquestionably poor statistical practices by generally well - intentioned scientists: Studies show that poor statistical practices are widely prevalent across scientific research (31, 32).”
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · cited 1×
“Large - scale studies have demonstrated limited replication of research in Psychology (24, 25), Economics (26), the Social Sciences more generally (27), and Medicine (28–30).”
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · cited 1×
“It is also plausible that this is an emergent property, arising from GPT - 4’s larger - scale training ( 23 ).”
Pervasiveness and correlates of implicit attitudes and stereotypes
Nosek, B. A., Smyth, F. L., Hansen, J. J., Devos, T., Lindner, N. M., Ranganath, K. A., Smith, C. T., Olson, K. R., Chugh, D., Greenwald, A. G., et al. (2007) · European Review of Social Psychology · doi:10.1080/10463280701489053 · Cited by 1,247 · cited 1×
“These stereotypes have been robustly studied in human subjects, using both implicit and explicit measures (41, 42).”
Self-citationAutonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050 · cited 1×
“More detailed statistical reporting may be found in SI Appendix , section S9 .”
Machine learning–accelerated computational fluid dynamics
Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., Brenner, M. P., & Hoyer, S. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2101784118 · Cited by 1,042
Math = male, me = female, therefore math ≠ me.
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.83.1.44 · Cited by 941
Self-citationMen Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 1×
“This may reflect the troubling tendency for AI systems to amplify biases in their training data ( 43 – 45 ).”
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · cited 2×
“Several recent articles suggest, for example, that LLMs can mimic responses from human research subjects (34–36), with some even suggesting they may significantly supplant them (37).”
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643 · cited 1×
“This aligns with prior research showing that more powerful models tend to intrinsically learn human biases more precisely ( 47 ).”
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2020) · arXiv · doi:10.48550/arxiv.2004.09456 · Cited by 643 · cited 1×
“This aligns with prior research showing that more powerful models tend to intrinsically learn human biases more precisely ( 47 ).”
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · cited 1×
“While LLMs selectively show emergent abilities on tasks drawn from cognitive psychology, their performance can deteriorate when stimuli are edited to be less familiar ( 16 ), and indeed transformers are more generally less successful in solving problems external to their training data ( 17 ).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Xu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.01219 · Cited by 544 · cited 1×
“They are also infamously prolific generators of convincing falsehoods, col-loquially termed “hallucinations” ( 13 , 14 ).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534 · cited 1×
“They are also infamously prolific generators of convincing falsehoods, col-loquially termed “hallucinations” ( 13 , 14 ).”
Fabrication and errors in the bibliographic citations generated by ChatGPT
Walters, W. H., & Wilder, E. I. (2023) · Scientific Reports · doi:10.1038/s41598-023-41032-5 · Cited by 499 · cited 1×
“They are also infamously prolific generators of convincing falsehoods, col-loquially termed “hallucinations” ( 13 , 14 ).”
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462 · cited 1×
“AI’s ability to comprehensively cull prior, relevant scientific articles is therefore critical.”
Can AI language models replace human participants?
Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.04.008 · Cited by 405
Gender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381 · cited 1×
“Research has shown that these models can amplify patterns of bias in their training corpus ( 11 , 12 ).”
AI and the transformation of social science research
Grossmann, I., Feinberg, M., Parker, D. C., Christakis, N. A., Tetlock, P. E., & Cunningham, W. A. (2023) · Science · doi:10.1126/science.adi1778 · Cited by 377 · cited 2×
“Several recent articles suggest, for example, that LLMs can mimic responses from human research subjects (34–36), with some even suggesting they may significantly supplant them (37).”
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369
Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
Bianchi, F., Kalluri, P., Durmus, E., Ladhak, F., Cheng, M., Nozza, D., Hashimoto, T., Jurafsky, D., Zou, J., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594095 · Cited by 342 · cited 1×
“Research has shown that these models can amplify patterns of bias in their training corpus ( 11 , 12 ).”
Self-citationAn open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312
Jablonka, K. M., Ai, Q., Al-Feghali, A., Badhwar, S., Bocarsly, J. D., Bran, A. M., Bringuier, S., Brinson, L. C., Choudhary, K., Circi, D., et al. (2023) · Digital Discovery · doi:10.1039/d3dd00113j · Cited by 218 · cited 1×
“Recent research suggests that LLMs, when fine - tuned for the task, can achieve scientific synthesis and inference on par with state - of - the - art special purpose models ( 7 ), and may be informally wielded by enterprising scientists to accelerate a range of research tasks ( 8 ).”
Guo, W., & Caliskan, A. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462536 · Cited by 180 · cited 1×
“In recent years, a significant body of literature has accumulated suggesting that human - like biases and stereotypes emerge from semantic patterns in large language corpora ( 38 – 40 ).”
Self-citationThe Internal State of an LLM Knows When It's Lying
Azaria, A., & Mitchell, T. (2023) · arXiv · doi:10.48550/arxiv.2304.13734 · Cited by 179 · cited 1×
“These results build upon research suggesting that LLMs can internally represent the truth or fiction of their statements ( 22 ).”
The Internal State of an LLM Knows When It’s Lying
Azaria, A., & Mitchell, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.68 · Cited by 175 · cited 1×
“These results build upon research suggesting that LLMs can internally represent the truth or fiction of their statements ( 22 ).”
Charlesworth, T. E. S., Yang, V., Mann, T. C., Kurdi, B., & Banaji, M. R. (2021) · Psychological Science · doi:10.1177/0956797620963619 · Cited by 161 · cited 6×
“More detailed statistical reporting may be found in SI Appendix , section S4 .”
InfluentialSelf-citationUsing Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Aher, G., Arriaga, R. I., & Kalai, A. T. (2022) · arXiv · doi:10.48550/arxiv.2208.10264 · Cited by 161 · cited 1×
“Large - scale studies have demonstrated limited replication of research in Psychology (24, 25), Economics (26), the Social Sciences more generally (27), and Medicine (28–30).”
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level ConstraintsPreprint
Zhao, J., Wang, T., Yatskar, M., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09457 · Cited by 122 · cited 1×
“This may reflect the troubling tendency for AI systems to amplify biases in their training data ( 43 – 45 ).”
A Mathematical Investigation of Hallucination and Creativity in GPT Models
Lee, M. (2023) · Mathematics · doi:10.3390/math11102320 · Cited by 105 · cited 1×
“There is a likely tradeoff between novelty and truth in LLMs: Hallucinations might be inevitable in a model capable of creativity ( 50 ).”
Emergent autonomous scientific research capabilities of large language models
Boiko, D. A., MacKnight, R., & Gomes, G. (2023) · arXiv · doi:10.48550/arxiv.2304.05332 · Cited by 72
Large Language Models for Scientific Synthesis, Inference and ExplanationPreprint
Zheng, Y., Koh, H. Y., Ju, J.-X., Nguyen, A., May, L. T., Webb, G. I., & Pan, S.-R. (2023) · arXiv.org · doi:10.48550/arxiv.2310.07984 · Cited by 64
Charlesworth, T. E. S., Navon, M., Rabinovich, Y., Lofaro, N., & Kurdi, B. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01851-2 · Cited by 46 · cited 1×
“For this, we used a second paper by Charlesworth and colleagues ( 48 ), which introduced a novel data-set: the Project Implicit International Dataset.”
Bias Amplification in Artificial Intelligence Systems
Lloyd, K. (2018) · arXiv · doi:10.48550/arxiv.1809.07842 · Cited by 42
Directional Bias Amplification
Wang, A., & Russakovsky, O. (2021) · arXiv · doi:10.48550/arxiv.2102.12594 · Cited by 30 · cited 1×
“The improvement of ChatGPT on this task was thus decisive: while the earlier model performed poorly, the more recent iteration was quite successful and could provide value to scientists in this domain.”
Jablonka, K. M., Ai, Q., Al‐Feghali, A., Badhwar, S., Bocarsly, J. D., Bran, A. M., Bringuier, S. A., Brinson, L. C., Choudhary, K. K., Çırcı, D., et al. (2023) · PubMed · doi:10.25418/crick.24297949.v1 · Cited by 29
Rawte, V., Priya, P., Tonmoy, S. M. T. I., Zaman, S. M. M., Sheth, A., & Das, A. (2023) · arXiv · doi:10.48550/arxiv.2309.11064 · Cited by 16 · cited 1×
“Prompt - engineering research indicates that more formal prompt language elicits fewer hallucinations from LLMs (21).”
Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models
Yadlowsky, S., Doshi, L., & Tripuraneni, N. (2023) · arXiv · doi:10.48550/arxiv.2311.00871 · Cited by 7 · cited 1×
“While LLMs selectively show emergent abilities on tasks drawn from cognitive psychology, their performance can deteriorate when stimuli are edited to be less familiar ( 16 ), and indeed transformers are more generally less successful in solving problems external to their training data ( 17 ).”
(1924) · Nature · doi:10.1038/114373a0 · Cited by 2
Research works printed but not linked: 3
- X. Zhang, Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv [Preprint] (2023). https://arxiv.org/pdf/2307.08423.pdf (Accessed 19 January 2024). · arXiv
- OpenAI, GPT-4 technical report. arXiv [Preprint] (2023). https://arxiv.org/pdf/2303.08774.pdf (Accessed 19 January 2024). · arXiv
- S. A. Lehr, A. Caliskan, S. Liyanage, M. R. Banaji, Data from “ChatGPT as research scientist: Probing GPT’s capabilities as a research librarian, research ethicist, data generator and data predictor.” OSF. https://osf.io/sdahr/. Deposited 10 June 2024. PMC11363351 39163339
Cited by
Using LLMs for Market Research
Brand, J., Israeli, A., & Ngwe, D. (2023) · doi:10.2139/ssrn.4395751 · Cited by 190
Risks of AI scientists: prioritizing safeguarding over autonomy
Tang, X., Jin, Q., Zhu, K., Yuan, T., Zhang, Y., Zhou, W., Qu, M., Zhao, Y., Tang, J., Zhang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-63913-1 · Cited by 51
“For instance, in chemistry and biology, these AI scientists can design experiments, control laboratory equipment, and make research decisions [13, 14, 2, 15, 16, 17].”
AgentRxiv: Towards Collaborative Autonomous ResearchPreprint
Schmidgall, S., & Moor, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.18102 · Cited by 141
Generative AI in Marketing and Principles for Ethical Design and Deployment
Hermann, E., & Puntoni, S. (2025) · Journal of Public Policy & Marketing · doi:10.1177/07439156241309874 · Cited by 134
“…(Qiu, Singh, and Srinivasan 2023) and human preferences of intertemporal rewards (Goli and Singh 2024) and appears to be limited in predicting new outcomes absent in its training data and in leveraging substantially new information when predicting more versus less novel outcomes (Lehr et al. 2024).”
Generalization bias in large language model summarization of scientific research
Peters, U., & Chin-Yee, B. (2025) · Royal Society Open Science · doi:10.1098/rsos.241776 · Cited by 126
Metadata
- Authors
- Lehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R.
- Affiliations
- Cangrade, Inc., Watertown, MA 02472
- University of Washington (US)
- Harvard University (US)
- Corresponding author
- Lehr, Steven A.
- Banaji, Mahzarin R.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2024-08-20
- Volume
- 121
- Issue
- 35
- Pages
- e2404328121-e2404328121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11363351
- OpenAlex
- W4401729468
- Cited by
- 66
- References
- 44
- Citations by year
- 2026: 21 · 2025: 39 · 2024: 6
- FWCI
- 27.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- US National Institute of Standards and Technology (60NANB23D194)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- ChatGPT, large language models, GPT-3.5, GPT-4, research librarianship, research ethics, hallucinations, data generation, hypothesis generation, artificial intelligence in science
- MeSH terms
- Humans, Research Personnel, Librarians, Ethicists, Ethics, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.