CoChat
Sign Up
PreprintCited by 325Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Large Language Models: A Survey

Open access
Minaee, Shervin, Mikolov, Tomas, Nikzad, Narjes, Chenaghlu, Meysam, Socher, Richard, Amatriain, Xavier, Gao, Jianfeng·2024·arXiv
DOI 10.48550/arxiv.2402.06196arXiv 2402.06196
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.06196

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs have emerged as a transformative technology in AI, demonstrating remarkable capabilities in natural language understanding, generation, and reasoning.

    “LLMs have emerged as a transformative technology in the field of artificial intelligence, demonstrating remarkable capabilities in natural language understanding, generation, and reasoning”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Minaee et al. 2024

  • Contemporary Transformer-based LLMs cannot possess consciousness if assessed solely based on their architecture, structure, topology, or processing mechanisms.

    “Meanwhile, a widely held perspective - independent of IIT - suggests that contemporary Transformer-based LLMs cannot possess consciousness if assessed solely based on their architecture, structure, topology, or processing mechanisms.”

    From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Minaee et al., 2024

    The Dominant LLM Architectures section describes Transformer-based LLMs, and the LLM limitations section discusses memory and hallucinations. Neither addresses whether architecture or processing mechanisms rule out consciousness. Searches for consciousness and the claimed architectural criterion found no relevant passage in our copy. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 170

  1. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., "Training compute-optimal large language models," arXiv preprint arXiv:2203.15556, 2022. · arXiv
  2. C. E. Shannon, "Prediction and entropy of printed english," Bell system technical journal, vol. 30, no. 1, pp. 50-64, 1951.
  3. F. Jelinek, Statistical methods for speech recognition. MIT press, 1998.
  4. C. Manning and H. Schutze, Foundations of statistical natural lan- guage processing. MIT press, 1999.
  5. C. D. Manning, An introduction to information retrieval. Cambridge university press, 2009.
  6. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., "A survey of large language models," arXiv preprint arXiv:2303.18223, 2023. · arXiv
  7. C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He et al., "A comprehensive survey on pretrained foundation mod- els: A history from bert to chatgpt," arXiv preprint arXiv:2302.09419, 2023. · arXiv
  8. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, "Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing," ACM Computing Surveys, vol. 55, no. 9, pp. 1-35, 2023.
  9. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, "A survey for in-context learning," arXiv preprint arXiv:2301.00234, 2022. · arXiv
  10. J. Huang and K. C.-C. Chang, "Towards reasoning in large language models: A survey," arXiv preprint arXiv:2212.10403, 2022. · arXiv
  11. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling," Computer Speech & Language, vol. 13, no. 4, pp. 359-394, 1999.
  12. Y. Bengio, R. Ducharme, and P. Vincent, "A neural probabilistic language model," Advances in neural information processing systems, vol. 13, 2000.
  13. H. Schwenk, D. Déchelotte, and J.-L. Gauvain, "Continuous space language models for statistical machine translation," in Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, 2006, pp. 723-730.
  14. T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur, "Recurrent neural network based language model." in Interspeech, vol. 2, no. 3. Makuhari, 2010, pp. 1045-1048.
  15. A. Graves, "Generating sequences with recurrent neural networks," arXiv preprint arXiv:1308.0850, 2013. · arXiv
  16. P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck, "Learning deep structured semantic models for web search using clickthrough data," in Proceedings of the 22nd ACM international conference on Information & Knowledge Management, 2013, pp. 2333-2338.
  17. J. Gao, C. Xiong, P. Bennett, and N. Craswell, Neural Approaches to Conversational Information Retrieval. Springer Nature, 2023, vol. 44.
  18. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks," Advances in neural information processing systems, vol. 27, 2014.
  19. K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio, "On the properties of neural machine translation: Encoder-decoder ap- proaches," arXiv preprint arXiv:1409.1259, 2014. · arXiv
  20. H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt et al., "From captions to visual concepts and back," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1473-1482.
  21. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3156-3164.
  22. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, "Deep contextualized word representations. corr abs/1802.05365 (2018)," arXiv preprint arXiv:1802.05365, 2018. · arXiv
  23. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. · arXiv
  24. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized bert pretraining approach," arXiv preprint arXiv:1907.11692, 2019. · arXiv
  25. P. He, X. Liu, J. Gao, and W. Chen, "Deberta: Decoding-enhanced bert with disentangled attention," arXiv preprint arXiv:2006.03654, 2020. · arXiv
  26. X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, Y. Yao, A. Zhang, L. Zhang et al., "Pre-trained models: Past, present and future," AI Open, vol. 2, pp. 225-250, 2021.
  27. X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, "Pre-trained models for natural language processing: A survey," Science China Technological Sciences, vol. 63, no. 10, pp. 1872-1897, 2020.
  28. A. Gu and T. Dao, "Mamba: Linear-time sequence modeling with selective state spaces," arXiv preprint arXiv:2312.00752, 2023. · arXiv
  29. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., "Palm: Scaling language modeling with pathways," arXiv preprint arXiv:2204.02311, 2022. · arXiv
  30. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., "Llama: Open and efficient foundation language models," arXiv preprint arXiv:2302.13971, 2023. · arXiv
  31. OpenAI, "GPT-4 Technical Report," https://arxiv.org/pdf/2303. 08774v3.pdf, 2023.
  32. J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, "Chain-of-thought prompting elicits reasoning in large language models," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824-24 837. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
  33. G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyil- maz et al., "Augmented language models: a survey," arXiv preprint arXiv:2302.07842, 2023. · arXiv
  34. B. Peng, M. Galley, P. He, H. Cheng, Y. Xie, Y. Hu, Q. Huang, L. Liden, Z. Yu, W. Chen, and J. Gao, "Check your facts and try again: Improving large language models with external knowledge and automated feedback," arXiv preprint arXiv:2302.12813, 2023. · arXiv
  35. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "React: Synergizing reasoning and acting in language models," arXiv preprint arXiv:2210.03629, 2022. · arXiv
  36. J. L. Elman, "Finding structure in time," Cognitive science, vol. 14, no. 2, pp. 179-211, 1990.
  37. M. V. Mahoney, "Fast text compression with neural networks." in FLAIRS conference, 2000, pp. 230-234.
  38. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Černockỳ, "Strate- gies for training large scale neural network language models," in 2011 IEEE Workshop on Automatic Speech Recognition & Understanding. IEEE, 2011, pp. 196-201.
  39. S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, "Deep learning-based text classification: a comprehensive review," ACM computing surveys (CSUR), vol. 54, no. 3, pp. 1-40, 2021.
  40. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
  41. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "Albert: A lite bert for self-supervised learning of language represen- tations," arXiv preprint arXiv:1909.11942, 2019. · arXiv
  42. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, "Electra: Pre- training text encoders as discriminators rather than generators," arXiv preprint arXiv:2003.10555, 2020. · arXiv
  43. G. Lample and A. Conneau, "Cross-lingual language model pretrain- ing," arXiv preprint arXiv:1901.07291, 2019. · arXiv
  44. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, "Xlnet: Generalized autoregressive pretraining for language understanding," Advances in neural information processing systems, vol. 32, 2019.
  45. L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon, "Unified language model pre-training for natural language understanding and generation," Advances in neural information processing systems, vol. 32, 2019.
  46. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  47. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485-5551, 2020.
  48. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, "mt5: A massively multilingual pre-trained text-to-text transformer," arXiv preprint arXiv:2010.11934, 2020. · arXiv
  49. K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, "Mass: Masked sequence to sequence pre-training for language generation," arXiv preprint arXiv:1905.02450, 2019. · arXiv
  50. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, "Bart: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension," arXiv preprint arXiv:1910.13461, 2019. · arXiv
  51. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language mod- els are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  52. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Ka- plan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  53. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al., "Webgpt: Browser- assisted question-answering with human feedback," arXiv preprint arXiv:2112.09332, 2021. · arXiv
  54. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback," Advances in Neural Information Processing Systems, vol. 35, pp. 27 730-27 744, 2022.
  55. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  56. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, "Alpaca: A strong, replicable instruction- following model," Stanford Center for Research on Foundation Mod- els. https://crfm. stanford. edu/2023/03/13/alpaca. html, vol. 3, no. 6, p. 7, 2023.
  57. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "Qlora: Ef- ficient finetuning of quantized llms," arXiv preprint arXiv:2305.14314, 2023. · arXiv
  58. X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song, "Koala: A dialogue model for academic research," Blog post, April, vol. 1, 2023.
  59. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al., "Mistral 7b," arXiv preprint arXiv:2310.06825, 2023. · arXiv
  60. B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin et al., "Code llama: Open foundation models for code," arXiv preprint arXiv:2308.12950, 2023. · arXiv
  61. A. Pal, D. Karkhanis, M. Roberts, S. Dooley, A. Sundararajan, and S. Naidu, "Giraffe: Adventures in expanding context lengths in llms," arXiv preprint arXiv:2308.10882, 2023. · arXiv
  62. Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy et al., "How far can camels go? exploring the state of instruction tuning on open resources," arXiv preprint arXiv:2306.04751, 2023. · arXiv
  63. S. Tworkowski, K. Staniszewski, M. Pacek, Y. Wu, H. Michalewski, and P. Miłoś, "Focused transformer: Contrastive training for context scaling," arXiv preprint arXiv:2307.03170, 2023. · arXiv
  64. Y. Tay, J. Wei, H. W. Chung, V. Q. Tran, D. R. So, S. Shakeri, X. Gar- cia, H. S. Zheng, J. Rao, A. Chowdhery et al., "Transcending scaling laws with 0.1% extra compute," arXiv preprint arXiv:2210.11399, 2022. · arXiv
  65. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma et al., "Scaling instruction- finetuned language models," arXiv preprint arXiv:2210.11416, 2022. · arXiv
  66. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al., "Palm 2 technical report," arXiv preprint arXiv:2305.10403, 2023. · arXiv
  67. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl et al., "Large language models encode clinical knowledge," arXiv preprint arXiv:2212.13138, 2022. · arXiv
  68. K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal et al., "Towards expert- level medical question answering with large language models," arXiv preprint arXiv:2305.09617, 2023. · arXiv
  69. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, "Finetuned language models are zero-shot learners," arXiv preprint arXiv:2109.01652, 2021. · arXiv
  70. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., "Scaling language models: Methods, analysis & insights from training gopher," arXiv preprint arXiv:2112.11446, 2021. · arXiv
  71. V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja et al., "Multi- task prompted training enables zero-shot task generalization," arXiv preprint arXiv:2110.08207, 2021. · arXiv
  72. Y. Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao, Y. Lu et al., "Ernie 3.0: Large-scale knowledge enhanced pre- training for language understanding and generation," arXiv preprint arXiv:2107.02137, 2021. · arXiv
  73. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Mil- lican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark et al., "Improving language models by retrieving from trillions of tokens," in International conference on machine learning. PMLR, 2022, pp. 2206-2240.
  74. O. Lieber, O. Sharir, B. Lenz, and Y. Shoham, "Jurassic-1: Technical details and evaluation," White Paper. AI21 Labs, vol. 1, p. 9, 2021.
  75. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat et al., "Glam: Efficient scaling of language models with mixture-of-experts," in International Conference on Machine Learning. PMLR, 2022, pp. 5547-5569.
  76. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.- T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du et al., "Lamda: Language models for dialog applications," arXiv preprint arXiv:2201.08239, 2022. · arXiv
  77. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., "Opt: Open pre-trained transformer language models," arXiv preprint arXiv:2205.01068, 2022. · arXiv
  78. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Sar- avia, A. Poulton, V. Kerkez, and R. Stojnic, "Galactica: A large language model for science," arXiv preprint arXiv:2211.09085, 2022. · arXiv
  79. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, "Codegen: An open large language model for code with multi-turn program synthesis," arXiv preprint arXiv:2203.13474, 2022. · arXiv
  80. S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky et al., "Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model," arXiv preprint arXiv:2208.01448, 2022. · arXiv
  81. A. Glaese, N. McAleese, M. Trebacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker et al., "Improving alignment of dialogue agents via targeted human judge- ments," arXiv preprint arXiv:2209.14375, 2022. · arXiv
  82. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo et al., "Solving quantitative reasoning problems with language models," Advances in Neural Information Processing Systems, vol. 35, pp. 3843-3857, 2022.
  83. Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, D. Bahri, T. Schuster, H. S. Zheng, N. Houlsby, and D. Metzler, "Unifying language learning paradigms," arXiv preprint arXiv:2205.05131, 2022. · arXiv
  84. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé et al., "Bloom: A 176b- parameter open-access multilingual language model," arXiv preprint arXiv:2211.05100, 2022. · arXiv
  85. A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia et al., "Glm-130b: An open bilingual pre-trained model," arXiv preprint arXiv:2210.02414, 2022. · arXiv
  86. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff et al., "Pythia: A suite for analyzing large language models across train- ing and scaling," in International Conference on Machine Learning. PMLR, 2023, pp. 2397-2430.
  87. S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah, "Orca: Progressive learning from complex explanation traces of gpt-4," arXiv preprint arXiv:2306.02707, 2023. · arXiv
  88. R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., "Starcoder: may the source be with you!" arXiv preprint arXiv:2305.06161, 2023. · arXiv
  89. S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, Q. Liu et al., "Language is not all you need: Aligning perception with language models," arXiv preprint arXiv:2302.14045, 2023. · arXiv
  90. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., "Gemini: a family of highly capable multimodal models," arXiv preprint arXiv:2312.11805, 2023. · arXiv
  91. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al., "Inner monologue: Embodied reasoning through planning with language models," arXiv preprint arXiv:2207.05608, 2022. · arXiv
  92. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti et al., "Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model," arXiv preprint arXiv:2201.11990, 2022. · arXiv
  93. I. Beltagy, M. E. Peters, and A. Cohan, "Longformer: The long- document transformer," arXiv preprint arXiv:2004.05150, 2020. · arXiv
  94. S. Iyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shus- ter, T. Wang, Q. Liu, P. S. Koura et al., "Opt-iml: Scaling language model instruction meta learning through the lens of generalization," arXiv preprint arXiv:2212.12017, 2022. · arXiv
  95. Y. Hao, H. Song, L. Dong, S. Huang, Z. Chi, W. Wang, S. Ma, and F. Wei, "Language models are general-purpose interfaces," arXiv preprint arXiv:2206.06336, 2022. · arXiv
  96. Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, and C. Gan, "Principle-driven self-alignment of language mod- els from scratch with minimal human supervision," arXiv preprint arXiv:2305.03047, 2023. · arXiv
  97. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, "Pal: Program-aided language models," in International Conference on Machine Learning. PMLR, 2023, pp. 10 764-10 799.
  98. E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou, "Codegen2: Lessons for training llms on programming and natural languages," arXiv preprint arXiv:2305.02309, 2023. · arXiv
  99. L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib et al., "Zephyr: Direct distillation of lm alignment," arXiv preprint arXiv:2310.16944, 2023. · arXiv
  100. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, "Qwen-vl: A frontier large vision-language model with versatile abilities," arXiv preprint arXiv:2308.12966, 2023. · arXiv
  101. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay, "The refined- web dataset for falcon llm: outperforming curated corpora with web data, and web data only," arXiv preprint arXiv:2306.01116, 2023. · arXiv
  102. D. Hernandez, T. Brown, T. Conerly, N. DasSarma, D. Drain, S. El- Showk, N. Elhage, Z. Hatfield-Dodds, T. Henighan, T. Hume et al., "Scaling laws and interpretability of learning from repeated data," arXiv preprint arXiv:2205.10487, 2022. · arXiv
  103. P. Shaw, J. Uszkoreit, and A. Vaswani, "Self-attention with relative position representations," arXiv preprint arXiv:1803.02155, 2018. · arXiv
  104. J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu, "Roformer: En- hanced transformer with rotary position embedding," arXiv preprint arXiv:2104.09864, 2021. · arXiv
  105. O. Press, N. A. Smith, and M. Lewis, "Train short, test long: Attention with linear biases enables input length extrapolation," arXiv preprint arXiv:2108.12409, 2021. · arXiv
  106. G. Ke, D. He, and T.-Y. Liu, "Rethinking positional encoding in language pre-training," arXiv preprint arXiv:2006.15595, 2020. · arXiv
  107. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer," arXiv preprint arXiv:1701.06538, 2017. · arXiv
  108. W. Fedus, B. Zoph, and N. Shazeer, "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity," The Journal of Machine Learning Research, vol. 23, no. 1, pp. 5232-5270, 2022.
  109. S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi, "Cross-task generalization via natural language crowdsourcing instructions," arXiv preprint arXiv:2104.08773, 2021. · arXiv
  110. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, "Self-instruct: Aligning language model with self generated instructions," arXiv preprint arXiv:2212.10560, 2022. · arXiv
  111. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances in neural information processing systems, vol. 30, 2017.
  112. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, "Direct preference optimization: Your language model is secretly a reward model," arXiv preprint arXiv:2305.18290, 2023. · arXiv
  113. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, "Zero: Memory optimizations toward training trillion parameter models," in SC20: In- ternational Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020, pp. 1-16.
  114. B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV et al., "Rwkv: Reinventing rnns for the transformer era," arXiv preprint arXiv:2305.13048, 2023. · arXiv
  115. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," arXiv preprint arXiv:2106.09685, 2021. · arXiv
  116. G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," arXiv preprint arXiv:1503.02531, 2015. · arXiv
  117. J. Gou, B. Yu, S. J. Maybank, and D. Tao, "Knowledge distillation: A survey," International Journal of Computer Vision, vol. 129, pp. 1789-1819, 2021.
  118. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, "Survey of hallucination in natural language generation," ACM Comput. Surv., vol. 55, no. 12, mar 2023. [Online]. Available: https://doi.org/10.1145/3571730 · DOI · Paper page
  119. C.-Y. Lin, "ROUGE: A package for automatic evaluation of summaries," in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74-81. [Online]. Available: https://aclanthology.org/W04-1013
  120. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: a method for automatic evaluation of machine translation," in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, P. Isabelle, E. Charniak, and D. Lin, Eds. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, Jul. 2002, pp. 311- 318. [Online]. Available: https://aclanthology.org/P02-1040
  121. B. Dhingra, M. Faruqui, A. Parikh, M.-W. Chang, D. Das, and W. Cohen, "Handling divergent reference texts when evaluating table-to-text generation," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A.
  122. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4884-4895. [Online]. Available: https://aclanthology.org/P19-1483
  123. Z. Wang, X. Wang, B. An, D. Yu, and C. Chen, "Towards faithful neural table-to-text generation with content-matching constraints," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 1072-1086. [Online]. Available: https: //aclanthology.org/2020.acl-main.101
  124. H. Song, W.-N. Zhang, J. Hu, and T. Liu, "Generating persona consis- tent dialogues by exploiting natural language inference," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 8878-8885, Apr. 2020.
  125. O. Honovich, L. Choshen, R. Aharoni, E. Neeman, I. Szpektor, and O. Abend, "q 2 : Evaluating factual consistency in knowledge- grounded dialogues via question generation and question answering," in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 7856-7870. [Online]. Available: https://aclanthology.org/2021.emnlp-main.619
  126. N. Dziri, H. Rashkin, T. Linzen, and D. Reitter, "Evaluating attribution in dialogue systems: The BEGIN benchmark," Transactions of the Association for Computational Linguistics, vol. 10, pp. 1066-1083, 2022. [Online]. Available: https://aclanthology.org/2022.tacl-1.62
  127. S. Santhanam, B. Hedayatnia, S. Gella, A. Padmakumar, S. Kim, Y. Liu, and D. Z. Hakkani-Tür, "Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation," ArXiv, vol. abs/2110.05456, 2021. · arXiv
  128. D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, and M. Young, "Machine learning: The high interest credit card of technical debt," in SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop), 2014.
  129. P. S. H. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, "Retrieval-augmented generation for knowledge-intensive NLP tasks," CoRR, vol. abs/2005.11401, 2020. [Online]. Available: https://arxiv.org/abs/2005.11401 · arXiv
  130. Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, "Retrieval-augmented generation for large language models: A survey," arXiv preprint arXiv:2312.10997, 2023. · arXiv
  131. S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu, "Unifying large language models and knowledge graphs: A roadmap," arXiv preprint arXiv:2306.08302, 2023. · arXiv
  132. Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, "Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface," arXiv preprint arXiv:2303.17580, 2023. · arXiv
  133. Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al., "The rise and potential of large language model based agents: A survey," arXiv preprint arXiv:2309.07864, 2023. · arXiv
  134. L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin et al., "A survey on large language model based autonomous agents," arXiv preprint arXiv:2308.11432, 2023. · arXiv
  135. Z. Durante, Q. Huang, N. Wake, R. Gong, J. S. Park, B. Sarkar, R. Taori, Y. Noda, D. Terzopoulos, Y. Choi, K. Ikeuchi, H. Vo, L. Fei- Fei, and J. Gao, "Agent ai: Surveying the horizons of multimodal interaction," arXiv preprint arXiv:2401.03568, 2024. · arXiv
  136. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov, "Natural questions: A benchmark for question answering research," Transactions of the Association for Computational Linguistics, vol. 7, pp. 452-466, 2019. [Online]. Available: https://aclanthology.org/Q19-1026
  137. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., "Program synthesis with large language models," arXiv preprint arXiv:2108.07732, 2021. · arXiv
  138. E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer, "QuAC: Question answering in context," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 2174-2184. [Online]. Available: https://aclanthology.org/D18-1241
  139. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, "Measuring coding challenge competence with apps," NeurIPS, 2021.
  140. V. Zhong, C. Xiong, and R. Socher, "Seq2sql: Generating structured queries from natural language using reinforcement learning," arXiv preprint arXiv:1709.00103, 2017. · arXiv
  141. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer, "TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension," in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M.-Y. Kan, Eds. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1601-1611. [Online]. Available: https://aclanthology.org/P17-1147
  142. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, "RACE: Large-scale ReAding comprehension dataset from examinations," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, M. Palmer, R. Hwa, and S. Riedel, Eds. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 785-794. [Online]. Available: https://aclanthology.org/D17-1082
  143. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ questions for machine comprehension of text," in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2383-2392. [Online]. Available: https://aclanthology.org/D16-1264
  144. C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, "Boolq: Exploring the surprising difficulty of natural yes/no questions," CoRR, vol. abs/1905.10044, 2019. [Online]. Available: http://arxiv.org/abs/1905.10044 · arXiv
  145. D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth, "Looking beyond the surface:a challenge set for reading compre- hension over multiple sentences," in Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
  146. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, "Training verifiers to solve math word problems," CoRR, vol. abs/2110.14168, 2021. [Online]. Available: https: //arxiv.org/abs/2110.14168 · arXiv
  147. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, "Measuring mathematical problem solving with the MATH dataset," CoRR, vol. abs/2103.03874, 2021. [Online]. Available: https://arxiv.org/abs/2103.03874 · arXiv
  148. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, "Think you have solved question answering? try arc, the AI2 reasoning challenge," CoRR, vol. abs/1803.05457, 2018. [Online]. Available: http://arxiv.org/abs/1803.05457 · arXiv
  149. Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi, "PIQA: reasoning about physical commonsense in natural language," CoRR, vol. abs/1911.11641, 2019. [Online]. Available: http://arxiv.org/abs/ 1911.11641 · arXiv
  150. M. Sap, H. Rashkin, D. Chen, R. L. Bras, and Y. Choi, "Socialiqa: Commonsense reasoning about social interactions," CoRR, vol. abs/1904.09728, 2019. [Online]. Available: http://arxiv.org/abs/1904. 09728 · arXiv
  151. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, "Can a suit of armor conduct electricity? A new dataset for open book question answering," CoRR, vol. abs/1809.02789, 2018. [Online]. Available: http://arxiv.org/abs/1809.02789 · arXiv
  152. S. Lin, J. Hilton, and O. Evans, "Truthfulqa: Measuring how models mimic human falsehoods," arXiv preprint arXiv:2109.07958, 2021. · arXiv
  153. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, "Hotpotqa: A dataset for diverse, explainable multi-hop question answering," CoRR, vol. abs/1809.09600, 2018. [Online]. Available: http://arxiv.org/abs/1809.09600 · arXiv
  154. Y. Zhuang, Y. Yu, K. Wang, H. Sun, and C. Zhang, "Toolqa: A dataset for llm question answering with external tools," arXiv preprint arXiv:2306.13304, 2023. · arXiv
  155. D. Chen, J. Bolton, and C. D. Manning, "A thorough examination of the cnn/daily mail reading comprehension task," in Association for Computational Linguistics (ACL), 2016.
  156. R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang et al., "Abstractive text summarization using sequence-to-sequence rnns and beyond," arXiv preprint arXiv:1602.06023, 2016. · arXiv
  157. Y. Bai and D. Z. Wang, "More than reading comprehension: A survey on datasets and metrics of textual question answering," arXiv preprint arXiv:2109.12264, 2021. · arXiv
  158. H.-Y. Huang, E. Choi, and W.-t. Yih, "Flowqa: Grasping flow in history for conversational machine comprehension," arXiv preprint arXiv:1810.06683, 2018. · arXiv
  159. S. Lee, J. Lee, H. Moon, C. Park, J. Seo, S. Eo, S. Koo, and H. Lim, "A survey on evaluation metrics for machine translation," Mathematics, vol. 11, no. 4, p. 1006, 2023.
  160. J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen, "Halueval: A large-scale hallucination evaluation benchmark for large language models," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 6449-6464.
  161. J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, "Challenges and applications of large language models," arXiv preprint arXiv:2307.10169, 2023. · arXiv
  162. S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi et al., "Textbooks are all you need," arXiv preprint arXiv:2306.11644, 2023. · arXiv
  163. Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee, "Textbooks are all you need ii: phi-1.5 technical report," arXiv preprint arXiv:2309.05463, 2023. · arXiv
  164. G. J. McLachlan, S. X. Lee, and S. I. Rathnayake, "Finite mixture models," Annual review of statistics and its application, vol. 6, pp. 355-378, 2019.
  165. H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning," arXiv preprint arXiv:2304.08485, 2023. · arXiv
  166. S. Liu, H. Cheng, H. Liu, H. Zhang, F. Li, T. Ren, X. Zou, J. Yang, H. Su, J. Zhu, L. Zhang, J. Gao, and C. Li, "Llava-plus: Learning to use tools for creating multimodal agents," arXiv preprint arXiv:2311.05437, 2023. · arXiv
  167. S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, "Next-gpt: Any-to-any multimodal llm," arXiv preprint arXiv:2309.05519, 2023. · arXiv
  168. N. Alshahwan, J. Chheda, A. Finegenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, "Automated unit test improvement using large language models at meta," arXiv preprint arXiv:2402.09171, 2024. · arXiv
  169. L. Sun, Y. Huang, H. Wang, S. Wu, Q. Zhang, C. Gao, Y. Huang, W. Lyu, Y. Zhang, X. Li et al., "Trustllm: Trustworthiness in large language models," arXiv preprint arXiv:2401.05561, 2024. · arXiv
  170. M. Josifoski, L. Klein, M. Peyrard, Y. Li, S. Geng, J. P. Schnitzler, Y. Yao, J. Wei, D. Paul, and R. West, "Flows: Building blocks of reasoning and collaborating ai," arXiv preprint arXiv:2308.01285, 2023. · arXiv

Cited by

Sorted by citations

Authors
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., & Gao, J.
Affiliations
  • Amazon (United States) (US)
  • Google (United States) (US)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.06196
arXiv
2402.06196
OpenAlex
W4391766565
Cited by
325
References
2
Citations by year
2026: 39 · 2025: 172 · 2024: 114
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, GPT, llamas, Arecaceae, scaling laws, LLM training, LLM fine-tuning, LLM evaluation metrics, benchmark comparison, natural language understanding, natural language generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review