The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Llama 2: Open Foundation and Fine-Tuned Chat Models
Open accessAbstract
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Llama 2's total training data was 1.8T tokens (Touvron et al., 2023), while Llama 3 used 3.7T tokens for mathematical reasoning (Grattafiori et al., 2024).”
From LIMO: Less is More for Reasoning (2025) · cites this paper as Touvron et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, the most widely used open-source LLMs can only support a few dozen back-and-forth messages or reason about a short document before exceeding their maximum input length (Touvron et al., 2023).”
From MemGPT: Towards LLMs as Operating Systems (2023) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“current knowledge injection methods often suffer from computational overhead [13, 10], catastrophic forgetting [63], and overfitting [64].”
From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similarly, Llama2 [254] demonstrates that post-RLHF, the model consistently generated factual responses to prompts on factual information.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [254]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“contemporary widely used LMs such as Llama 2 (Touvron et al., 2023) are trained on trillions of words (Figure 1).”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on the GPT-3.5 results, it is unlikely that STOP would consistently work with any open-source LM at the time of writing (Touvron et al., 2023; Jiang et al., 2023).”
From Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation (2023) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We evaluate our OS-inspired design in two domains where the limited context windows of modern LLMs severely handicaps their performance: document analysis, where MemGPT is able to analyze large documents that far exceed the underlying LLM's context window, and multi-session chat, where MemGPT can create conversational agents that remember, reflect, and evolve dynamically through long-term inter”
From MemGPT: Towards LLMs as Operating Systems (2023) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A limiting factor for in-context learning (ICL) in LLMs is the context window, restricting prior research to the few-shot ICL regime.”
From Many-Shot In-Context Learning (2024) · cites this paper as [57]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“existing foundation models for vision and language such as CLIP [8], SigLIP [9], and Llama 2 [10] are capable of these types of generalization and more, stemming from the priors captured by their Internet-scale pretraining datasets.”
From OpenVLA: An Open-Source Vision-Language-Action Model (2024) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“recent Large Language Models (LLMs) (Touvron et al., 2023; OpenAI, 2023; Anil et al., 2023) have shown impressive capabilities in processing and generating text with remarkable accuracy, even outperforming human experts across diverse specialized domains including math, physics, history, law, medicine, and ethics.”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This model uses 32,000 tokens and 4,096 dimensions for its token embeddings.”
From The Linear Representation Hypothesis and the Geometry of Large Language Models (2023) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Its seven billion parameters are pre-trained on two trillion sentencepiece (Kudo & Richardson, 2018) tokens, 90% of which is in English.”
From The Linear Representation Hypothesis and the Geometry of Large Language Models (2023) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, finetuning and pretraining are computation-expensive [13, 10] and usually suffer from catastrophic forgetting [63] and overfitting [64].”
From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“A representative example of such data-driven methods can be found in the development of Llama-2 [254], where the most factual sources within the pretraining data were up-sampled in order to reduce hallucinations.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [254]
The cited source says what the claim attributes to it.
In the citing paper
“The winners of the challenge's various tracks made contributions that led to impressive improvements in our evaluation over not just the BabyLM baselines, but also the massively pretrained Llama 2 model (Touvron et al., 2023).”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“As instruction tuning does not require so much data (Zhou et al., 2023a;Touvron et al., 2023)”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“we note that LLaMA-2-Chat-7B also undergoes RLHF [60]”
From Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings (2024) · cites this paper as [60]
The cited source says what the claim attributes to it.
In the citing paper
“Large language models (LLMs) have revolutionized AI, but are constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis.”
From MemGPT: Towards LLMs as Operating Systems (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“State-of-the-art LLMs [66, 67, 7, 52] are generally implemented using the Transformer architecture, which is a multi-layer feedforward network equipped with attention mechanism [71].”
From Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“We have seen similar positive performance from other recent models, like Llama 2 (7B and 13B) chat variants (Touvron et al., 2023).”
From NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Human feedback is commonly utilized to finetune AI assistants.”
From AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Llama 2 is pre-trained on 2 trillion tokens of data using the autoregressive language modeling task.”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [73]
The cited source says what the claim attributes to it.
In the citing paper
“Prior studies showed that within the same LLM family, larger models are associated with better performance [59, 67, 73].”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [73]
The cited source says what the claim attributes to it.
In the citing paper
“Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least 100 × : from only a few thousand tokens in GPT-3 [8] and Llama 2 [57] to 1M tokens in Gemini 1.5 Pro [16].”
From Many-Shot In-Context Learning (2024) · cites this paper as [57]
The cited source says what the claim attributes to it.
In the citing paper
“In particular, SELF-RAG outperforms retrieval-augmented ChatGPT on four tasks, Llama2-chat (Touvron et al., 2023) and Alpaca (Dubois et al., 2023) on all tasks.”
From CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Intriguingly, our program with the 13b variant of the model is competitive with their 34b-based results even though we don't use human reasoning chains in our program.”
From DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The Llama2 authors (Touvron et al., 2023) presents 28.7% for llama2-13b , 42.2% for llama2-34b , and 56.8% for llama2-70b .”
From DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023) · cites this paper as Touvron et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“A model whose weights are published33,34,35,36 can, in principle, be retrieved and rerun by others later, giving the runtime a snapshottable character more like that of conventional software environments and narrowing the shelf-life gap that closed-weight models create.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [33]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“some game environments are procedurally generated and game states grow exponentially, making games more robust against evaluation dataset contamination as observed in recent works (Touvron et al., 2023).”
From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“generalization to other architectures should be verified [Touvron et al., 2023, Achiam et al., 2023].”
From Accurate and Efficient Long-Term Memory for LLM Agents (2026) · cites this paper as Touvron et al., 2023
Experimental — accuracy not yet measuredIn the citing paper
“Errors in retrieving recent information may be intrinsic to the current LLMs because the pretraining procedures weight all text in the training data equally, without specific emphasis on more recent or high-quality information.27”
From Performance of Large Language Models on Medical Oncology Examination Questions (2024) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Self-citationKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Self-citationEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 4×
“In Table 24, we present fine-grained results from the English part of the AGI Eval (Zhong et al., 2023) benchmark.”
BackgroundInfluential
Research works printed but not linked: 98
- Daron Acemoglu and Pascual Restrepo. Artificial intelligence, automation, and work. In The economics of artificial intelligence: An agenda , pages 197-236. University of Chicago Press, 2018.
- Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021b. · arXiv
- Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, and Chris Olah. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021a. · arXiv
- YuntaoBai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022a. · arXiv
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022b. · arXiv
- Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 610-623, 2021a.
- Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 610-623, 2021b.
- A Stevie Bergman, Gavin Abercrombie, Shannon L Spruit, Dirk Hovy, Emily Dinan, Y-Lan Boureau, and Verena Rieser. Guiding the release of safer e2e conversational ai through value sensitive design. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue , pages 39-52, 2022.
- Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , pages 7432-7439, 2020.
- Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1004-1015, 2021.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings. neurips.cc/paper\_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .
- Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomás Mikolov. Enriching word vectors with subword information. CoRR , abs/1607.04606, 2016. URL http://arxiv.org/abs/1607.04606 . · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
- Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2174-2184, 2018.
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.
- Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022. · arXiv
- Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 7282-7296, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.565. URL https://aclanthology.org/2021.acl-long.565 . · DOI · Paper page
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019. · arXiv
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018. · arXiv
- Jiawen Deng, Hao Sun, Zhexin Zhang, Jiale Cheng, and Minlie Huang. Recent advances towards safe, responsible, and moral dialogue systems: A survey. arXiv preprint arXiv:2302.09270 , 2023. · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. · arXiv
- Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc'Aurelio Ranzato. Residual energy-based models for text generation. In International Conference on Learning Representations , 2019.
- Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. Anticipating safety issues in e2e conversational ai: Framework and tooling. arXiv preprint arXiv:2107.03451 , 2021. · arXiv
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 862-872, 2021.
- Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 1286-1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.98. URL https://aclanthology.org/2021.emnlp-main. 98 . · DOI · Paper page
- Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 5547-5569. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/du22c.html .
- Jesse Dodge, Taylor Prewitt, Remi Tachet Des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A Smith, Nicole DeCario, and Will Buchanan. Measuring the carbon intensity of ai in cloud instances. arXiv preprint arXiv:2206.05229 , 2022. · arXiv
- Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V -usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 5988-6008. PMLR, 17-23 Jul 2022.
- Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 , 2022. · arXiv
- Prakhar Ganesh, Hongyan Chang, Martin Strobel, and Reza Shokri. On the impact of machine learning randomness on group fairness. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages 1789-1800, 2023.
- Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamil˙ e Lukoši¯ ut˙ e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459 , 2023. · arXiv
- Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research , 77:103-166, 2023.
- Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056 , 2023. · arXiv
- Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S Lee, David Brooks, and Carole-Jean Wu. Act: designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual International Symposium on Computer Architecture , pages 784-799, 2022a.
- Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717 , 2023. · arXiv
- Udit Gupta, Young Guen Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin Sean Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing carbon: The elusive environmental footprint of computing. IEEE Micro , 2022b.
- Kilem Li Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology , 61(1):29-48, 2008.
- Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3309-3326, 2022.
- Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 , 2020. · arXiv
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020. · arXiv
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022. · arXiv
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021. · arXiv
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id= rygGQyrFvH .
- Saghar Hosseini, Hamid Palangi, and Ahmed Hassan Awadallah. An empirical study of metrics to measure representational harms in pre-trained language models. arXiv preprint arXiv:2301.09211 , 2023. · arXiv
- Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689 , 2022. · arXiv
- Fan Huang, Haewoon Kwak, and Jisun An. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint arXiv:2302.07736 , 2023. · arXiv
- Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 , 2017. · arXiv
- Clayton Hutto and Eric Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media , volume 8, pages 216-225, 2014.
- Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversationsdemocratizing large language model alignment. arXiv preprint arXiv:2304.07327 , 2023. · arXiv
- James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.
- Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302.08582 , 2023. · arXiv
- Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. Language generation models can cause harm: So what can we do about it? an actionable survey. arXiv preprint arXiv:2210.07700 , 2022. · arXiv
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 2019.
- Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, 2022.
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019. · arXiv
- Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021. · arXiv
- Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688 , 2023. · arXiv
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 , 2023. · arXiv
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. · arXiv
- Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 , 2023. · arXiv
- Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. CoRR , abs/1810.03993, 2018. URL http://arxiv.org/abs/1810.03993 . · arXiv
- Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 , 2018. · arXiv
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Lonbrown Ouyanbrown, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. In arXiv , 2021.
- Cuong V. Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXiv:1908.01091 , 2019. · arXiv
- OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774 . · DOI · Paper page
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.
- David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350 , 2021. · arXiv
- Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations , 2021.
- Stephen Roller, Y-Lan Boureau, Jason Weston, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff, et al. Open-domain conversational agents: Current progress, open problems, and future directions. arXiv preprint arXiv:2006.12442 , 2020. · arXiv
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.
- Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728 , 2019. · arXiv
- Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022. · arXiv
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. · arXiv
- Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Discriminative adversarial search for abstractive summarization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 8555-8564. PMLR, 13-18 Jul 2020a. URL https://proceedings.mlr.press/v119/ scialom20a.html .
- Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Coldgans: Taming language gans with cautious sampling strategies. Advances in Neural Information Processing Systems , 33:18978-18989, 2020b.
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 12007-12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.823 .
- Eric Michael Smith and Adina Williams. Hi, my name is martha: Using names to measure and mitigate bias in generative dialogue models. arXiv preprint arXiv:2109.03300 , 2021. · arXiv
- Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arxiv:2305.17493 , 2023. · arXiv
- Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 'i'm sorry to hear that': Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 9180-9211, 2022.
- Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS , 2020.
- Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III, Jesse Dodge, Ellie Evans, Sara Hooker, et al. Evaluating the social impact of generative ai systems in systems and society. arXiv preprint arXiv:2306.05949 , 2023. · arXiv
- Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-ofthought can solve them. arXiv preprint arXiv:2210.09261 , 2022. · arXiv
- Yarden Tal, Inbal Magar, and Roy Schwartz. Fewer errors, but more stereotypes? the effect of model size on gender bias. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP) , pages 112-120, Seattle, Washington, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.gebnlp-1.13. URL https://aclanthology.org/2022.gebnlp-1.13 . · DOI
- AlonTalmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 , 2018. · arXiv
- Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 , 2022. · arXiv
- Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.
- Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022. · arXiv
- Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2021.
- Michael Webb. The impact of artificial intelligence on the labor market. Available at SSRN 3482150 , 2019.
- Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022a. URL https://openreview.net/forum?id=gEZrGCozdqR .
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824-24837, 2022b.
- Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems , 4:795-813, 2022.
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019a. · arXiv
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems , 32, 2019b.
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. · arXiv
- Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 , 2023. · arXiv
- Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations , 2022.
- Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867 , 2023. · arXiv
Cited by
Sorted by citations
Items 201–300 of 417
Large language model trained on clinical oncology data predicts cancer progression
Zhu, M., Lin, H., Jiang, J., Jinia, A. J., Jee, J., Pichotta, K., Waters, M., Rose, D., Schultz, N., Chalise, S., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01780-2 · Cited by 30
“All MSK patients included in this study were part of institutional review board (IRB)-approved research protocols (MSK IRB Protocols 12 – 245).”
Digital forgetting in large language models: a survey of unlearning methods
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11078-6 · Cited by 31
A psychometric framework for evaluating and shaping personality traits in large language models
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01115-6 · Cited by 30
Large Language Models Are Not Robust Multiple Choice Selectors
Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2023) · arXiv · doi:10.48550/arxiv.2309.03882 · Cited by 29
Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.890 · Cited by 31
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28
“For instance, Devin , an AI software engineer developed by Cognition Labs, has demonstrated the ability to plan, code, debug, and deploy software with minimal oversight.”
A Continued Pretrained LLM Approach for Automatic Medical Note Generation
Yuan, D., Rastogi, E., Naik, G., Rajagopal, S. P., Goyal, S., Zhao, F., Chintagunta, B., & Ward, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.47 · Cited by 28
JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3511593 · Cited by 36
The Platonic Representation Hypothesis
Huh, M., Cheung, B., Wang, T., & Isola, P. (2024) · arXiv · doi:10.48550/arxiv.2405.07987 · Cited by 35
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.260 · Cited by 33
Self-citationSelf-Evaluation Guided Beam Search for Reasoning
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.-Y., He, J., & Xie, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1802 · Cited by 27
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27
Self-citationFANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Kim, H., Sclar, M., Zhou, X., Bras, R., Kim, G., Choi, Y., & Sap, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.890 · Cited by 27
AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War
Humphreys, D. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00928-y · Cited by 26
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
“These models, such as OpenAI’s GPT [4], Meta’s LLaMA [124], and Google’s Gemini series [123], are trained on vast text corpora and rely on large-scale artificial neural networks with billions, sometimes trillions, of parameters.”
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Chen, H., Fang, Z., Singla, Y., & Dredze, M. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.182 · Cited by 31
Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives
Zhang, W., Shen, Y., Wu, L., Peng, Q., Wang, J., Zhuang, Y., & Lu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.197 · Cited by 27
Computing Power and the Governance of Artificial Intelligence
Sastry, G., Heim, L., Belfield, H., Anderljung, M., Brundage, M., Hazell, J., O'Keefe, C., Hadfield, G. K., Ngo, R., Pilz, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.08797 · Cited by 26
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26
Mixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 386
“5-72B-Chat, WizardLM-8x22B (Xu et al., 2023a), LLaMA-3-70B-Instruct (Touvron et al., 2023b), Mixtral-8x22B-v0.”
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25
Scherer, M., Macan, L., Jung, V. J. B., Wiese, P., Bompani, L., Burrello, A., Conti, F., & Benini, L. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3443718 · Cited by 26
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · arXiv · doi:10.48550/arxiv.2402.13178 · Cited by 312
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Ha, H., Florence, P., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.14535 · Cited by 24
“We evaluate on the proprietary GPT3 [66] (175B text-davinci-003) and the open LLAMA2 [67] (7B and 13B).”
MethodKeloth, V. K., Selek, S., Chen, Q., Gilman, C., Fu, S., Dang, Y., Chen, X., Hu, X., Zhou, Y., He, H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01645-8 · Cited by 24
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., & Goldstein, T. (2024) · arXiv · doi:10.48550/arxiv.2401.12070 · Cited by 24
TRAM: Benchmarking Temporal Reasoning for Large Language Models
Wang, Y., & Zhao, Y. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.382 · Cited by 24
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2311.07919 · Cited by 61
Self-citationRLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., & Synnaeve, G. (2024) · arXiv · doi:10.48550/arxiv.2410.02089 · Cited by 23
Generative AI for crystal structures: A review
De Breuck, P.-P., Wang, H.-C., Rignanese, G.-M., Botti, S., & Marques, M. A. L. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01881-2 · Cited by 24
Li, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D. (2024) · arXiv · doi:10.48550/arxiv.2402.05120 · Cited by 253
Evaluating the World Model Implicit in a Generative Model
Vafa, K., Chen, J., Rambachan, A., Kleinberg, J., & Mullainathan, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-0846 · Cited by 22
Routerdc: Query-based router by dual contrastive learning for assembling large language models
Chen, S., Jiang, W., Lin, B., Kwok, J., & Zhang, Y. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2120 · Cited by 22
Self-citationXiao, C., Cai, J., Zhao, W., Lin, B., Zeng, G., Zhou, J., Zheng, Z., Han, X., Liu, Z., & Sun, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01137-0 · Cited by 24
“Specifically, we evaluated the density of the following models: Llama series of models 4,11,63 , Falcon 64 , MPT 65 , Phi series of models 66–68 , Mistral series of models 69 , StableLM 70 , TinyLlama 71 , MiniCPM series of models 17 , DeepSeek series of models 72 and Gemma series of models 45,73 .”
On the Convergence of Zeroth-Order Federated Tuning for Large Language Models
Ling, Z., Chen, D., Yao, L., Li, Y., & Shen, Y. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671865 · Cited by 21
LM-Polygraph: Uncertainty estimation for language models
Fadeeva, E., Vashurin, R., Tsvigun, A., Vazhentsev, A., Petrakov, S., Fedyanin, K., Vasilev, D., Goncharova, E., Panchenko, A., Panov, M., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.41 · Cited by 20
Han, D., Zhao, W., Yin, H., Qu, M., Zhu, J., Ma, F., Ying, Y., & Pan, A. (2025) · Journal of Asian Architecture and Building Engineering · doi:10.1080/13467581.2024.2329351 · Cited by 20
M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection
Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A., Whitehouse, C., Afzal, O. M., Mahmoud, T., Sasaki, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.14902 · Cited by 19
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Gallifant, J., Chen, S., Moreira, P. J. F., Munch, N., Gao, M., Pond, J., Celi, L. A., Aerts, H., Hartvigsen, T., & Bitterman, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.726 · Cited by 19
From Role-Play to Drama-Interaction: An LLM Solution
Wu, W., Wu, H., Jiang, L., Liu, X., Zhao, H., & Zhang, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.196 · Cited by 20
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E. P., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.14089 · Cited by 65
An Open-Source Data Contamination Report for Large Language Models
Li, Y., Guo, Y., Guerin, F., & Lin, C. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.30 · Cited by 19
A Survey on the Integration and Optimization of Large Language Models in Edge Computing Environments
Bhardwaj, S., Singh, P., & Pandit, M. K. (2024) · 2024 16th International Conference on Computer and Automation Engineering (ICCAE) · doi:10.1109/iccae59995.2024.10569285 · Cited by 18
LLM Evaluators Recognize and Favor Their Own Generations
Panickssery, A., Bowman, S. R., & Feng, S. (2024) · arXiv · doi:10.48550/arxiv.2404.13076 · Cited by 559
“To produce and evaluate LLM-generated summaries, we use instruction-tuned LLMs: Llama-2-7b-chat (Touvron et al., 2023), GPT-3.”
Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions
Gridach, M., Nanavati, J., Abidine, K. Z. E., Mendes, L., & Mack, C. (2025) · arXiv · doi:10.48550/arxiv.2503.08979 · Cited by 18
“The rapid advancements of Large Language Models (LLMs) (Touvron et al., 2023; Anil et al., 2023; Achiam et al., 2023) have opened a new era in scientific discovery, with Agentic AI systems (Kim et al., 2024; Guo et al., 2023; Wang et al., 2024; Abramovich et al., 2024) emerging as powerful tools…”
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“As instruction tuning does not necessarily require so much data (Zhou et al., 2023a; Touvron et al., 2023), we apply several (3)https://www.”
MethodUnderstanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17
Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions
Roy, S., Khatua, A., Ghoochani, F., Hadler, U., Nejdl, W., & Ganguly, N. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657882 · Cited by 17
Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 24
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., & Zhu, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02508 · Cited by 17
Self-citationAgent Lumos: Unified and Modular Training for Open-Source Language Agents
Yin, D., Brahman, F., Ravichander, A., Chandu, K., Chang, K.-W., Choi, Y., & Lin, B. Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.670 · Cited by 17
Nechakhin, V., D’Souza, J., & Eger, S. (2024) · Information · doi:10.3390/info15060328 · Cited by 17
Yang, R., Zeng, Q., You, K., Qiao, Y., Huang, L., Hsieh, C.-C., Rosand, B., Goldwasser, J., Dave, A., Keenan, T., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/60601 · Cited by 16
MAIRA-2: Grounded Radiology Report Generation
Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04449 · Cited by 19
“5 (Chiang et al., 2023), itself finetuned from Llama 2 (Touvron et al., 2023); and the adapter is a randomly initialised multilayer perceptron (MLP) with four layers.”
BackgroundWang, K., Duan, F., Wang, S., Li, P., Xian, Y., Yin, C., Rong, W., & Xiong, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.13259 · Cited by 16
Comparative Analysis of Open-Source Language Models in Summarizing Medical Text Data
Chen, Y., Wang, Z., & Zulkernine, F. (2024) · 2024 IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh62654.2024.00030 · Cited by 15
Finding Inductive Loop Invariants using Large Language Models
Kamath, A., Senthilnathan, A., Chakraborty, S., Deligiannis, P., Lahiri, S. K., Lal, A., Rastogi, A., Roy, S., & Sharma, R. (2023) · arXiv · doi:10.48550/arxiv.2311.07948 · Cited by 17
Detecting LLM-generated peer reviews
Rao, V. S., Kumar, A., Lakkaraju, H., & Shah, N. B. (2025) · PLOS One · doi:10.1371/journal.pone.0331871 · Cited by 14
“In our experimental evaluation of the cryptic prompt injection, we use two white-box LLMs, Llama 2 [TMS + 23] and Vicuna 1.5 [CLL + 23], to generate reviews for 20 examples each from two datasets, International Congress on Peer Review and Scientific Publication (PRC) 2022 abstracts and PeerRead…”
InfluentialThink Twice: Perspective-Taking Improves Large Language Models’ Theory-of-Mind Capabilities
Wilf, A., Lee, S., Liang, P. P., & Morency, L.-P. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.451 · Cited by 17
Dialogue Summarization with Mixture of Experts based on Large Language Models
Tian, Y., Xia, F., & Song, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.385 · Cited by 14
CompAct: Compressing Retrieved Documents Actively for Question Answering
Yoon, C., Lee, T., Hwang, H., Jeong, M., & Kang, J. (2024) · arXiv · doi:10.48550/arxiv.2407.09014 · Cited by 14
Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
Thalken, R., Stiglitz, E., Mimno, D., & Wilkens, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.575 · Cited by 13
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 1,068
Self-citationCOSMO: A Large-Scale E-commerce Common Sense Knowledge Generation and Serving System at Amazon
Yu, C., Liu, X., Maia, J., Li, Y., Cao, T., Gao, Y., Song, Y., Goutam, R., Zhang, H., Yin, B., et al. (2024) · Companion of the 2024 International Conference on Management of Data · doi:10.1145/3626246.3653398 · Cited by 13
RewardBench: Evaluating Reward Models for Language Modeling
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. (2025) · Findings of the Association for Computational Linguistics: NAACL 2025 · doi:10.18653/v1/2025.findings-naacl.96 · Cited by 13
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
Krumdick, M., Koncel-Kedziorski, R., Lai, V. D., Reddy, V., Lovering, C., & Tanner, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.452 · Cited by 13
Understanding the effects of human-written paraphrases in LLM-generated text detection
Lau, H. T., & Zubiaga, A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100151 · Cited by 13
Are Large Language Models Consistent over Value-laden Questions?
Moore, J., Deshpande, T., & Yang, D. (2024) · arXiv · doi:10.48550/arxiv.2407.02996 · Cited by 12
Groot, T., & Valdenegro - Toro, M. (2024) · Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024) · doi:10.18653/v1/2024.trustnlp-1.13 · Cited by 20
RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., et al. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3645056 · Cited by 12
Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B. (2023) · arXiv · doi:10.48550/arxiv.2311.04378 · Cited by 12
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
Dong, Y., Wang, Z., Sreedhar, M., Wu, X., & Kuchaiev, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.754 · Cited by 15
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145
“We tested the performance on models including Llama-7B/70B [49], Mistral-8x7B [17], LLaVA-v1.”
On the Societal Impact of Open Foundation Models
Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., Hopkins, A., Bankston, K., Biderman, S., Bogen, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07918 · Cited by 103
A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
Huang, Y., Tang, K., Chen, M., & Wang, B. (2024) · arXiv · doi:10.48550/arxiv.2404.15777 · Cited by 14
Self-citationAre Emergent Abilities in Large Language Models just In-Context Learning?
Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T., & Gurevych, I. (2023) · arXiv · doi:10.48550/arxiv.2309.01809 · Cited by 12
Investigating Agency of LLMs in Human-AI Collaboration Tasks
Sharma, A., Rao, S., Brockett, C., Malhotra, A., Jojic, N., & Dolan, B. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.119 · Cited by 13
LEGO-GraphRAG: Modularizing Graph-Based Retrieval-Augmented Generation for Design Space Exploration
Cao, Y., Gao, Z., Li, Z., Xie, X., Zhou, S. K., & Xu, J. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3748191.3748194 · Cited by 11
Beamaggr: Beam aggregation reasoning over multi-source knowledge for multi-hop question answering
Chu, Z., Chen, J., Chen, Q., Wang, H., Zhu, K., Du, X., Yu, W., Liu, M., & Qin, B. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.67 · Cited by 11
Exploring the Robustness of In-Context Learning with Noisy Labels
Cheng, C., Yu, X., Wen, H., Sun, J., Yue, G., Zhang, Y., & Wei, Z. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10889281 · Cited by 11
Self-citationAppAgent: Multimodal Agents as Smartphone Users
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G. (2023) · arXiv · doi:10.48550/arxiv.2312.13771 · Cited by 11
“Unlike earlier LLMs, these new models [32–34, 49, 51] enable multi-round conversations and have the impressive ability to follow complex instructions.”
The Linear Representation Hypothesis and the Geometry of Large Language Models
Park, K., Choe, Y. J., & Veitch, V. (2023) · arXiv · doi:10.48550/arxiv.2311.03658 · Cited by 11
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.11018 · Cited by 35
“Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least 100 × : from only a few thousand tokens in GPT-3 [8] and Llama 2 [56] to 1M tokens in Gemini 1.5 Pro [15].”
InfluentialDo Language Models Know When They’re Hallucinating References?
Agrawal, A., Suzgun, M., Mackey, L., & Kalai, A. (2024) · Findings of the Association for Computational Linguistics: EACL 2024 · doi:10.18653/v1/2024.findings-eacl.62 · Cited by 37
Hiwa, D. S., Abdalla, S. S., Muhialdeen, A. S., Hamasalih, H. M., & Karim, S. O. (2024) · Barw Medical Journal · doi:10.58742/bmj.v2i2.87 · Cited by 10
Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9
Self-citationThe Evolution of Mixture of Experts: A Survey from Basics to Breakthroughs
Vats, A., Raja, R., Jain, V., & Chadha, A. (2024) · doi:10.20944/preprints202408.0583.v2 · Cited by 10
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
Ravichander, A., Ghela, S., Wadden, D., & Choi, Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.71 · Cited by 13
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 17
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9
Self-citationA Survey on Self-Evolution of Large Language Models
Tao, Z., Lin, T.-E., Chen, X., Li, H., Wu, Y., Li, Y., Jin, Z., Huang, F., Tao, D., & Zhou, J. (2024) · arXiv · doi:10.48550/arxiv.2404.14387 · Cited by 9
Typhoon: Thai Large Language Models
Pipatanakul, K., Jirabovonvisut, P., Manakul, P., Sripaisarnmongkol, S., Patomwong, R., Chokchainant, P., & Tharnpipitchai, K. (2023) · arXiv · doi:10.48550/arxiv.2312.13951 · Cited by 9
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Hsu, T.-Y., Huang, C.-Y., Rossi, R., Kim, S., Giles, C., & Huang, T.-H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.363 · Cited by 8
RouteLLM: Learning to Route LLMs with Preference Data
Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2406.18665 · Cited by 15
“For Unify AI, we select the best-performing router configuration on the user dashboard and use it for benchmarking.”
Self-citationChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning
Wang, Z., Liu, J., Bao, Q., Rong, H., & Zhang, J. (2024) · 2024 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650138 · Cited by 8
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8
“We describe the size and composition of the four benchmarks, as well as the corresponding LLMs (see Appendix C for additional details): • HuggingFace’s Open LLM Leaderboard (Beeching et al., 2023) et al., 2023; Achiam et al., 2023; Team et al., 2023) and inclusion into several other benchmarks.”
MethodMe LLaMA: Foundation Large Language Models for Medical Applications
Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.12749 · Cited by 8
Universal Jailbreak Backdoors from Poisoned Human Feedback
Rando, J., & Tramèr, F. (2023) · arXiv · doi:10.48550/arxiv.2311.14455 · Cited by 10
Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models
Luo, L., Zhao, Z., Haffari, G., Li, Y.-F., Gong, C., & Pan, S. (2024) · arXiv · doi:10.48550/arxiv.2410.13080 · Cited by 8
Metadata
- Authors
- Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., & Scialom, T.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.09288
- OpenAlex
- W4384918448
- Cited by
- 2,604
- References
- 6
- Citations by year
- 2026: 173 · 2025: 808 · 2024: 1373 · 2023: 248
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- large language models, clinical decision support, offline deployment, context retention, allergy documentation, discrepancy detection, treatment escalation, open-weight language models, resource-constrained environments, synthetic patient data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
