CoChat
Sign Up
PreprintCited by 1,259Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

A Comprehensive Overview of Large Language Models

Open access
Naveed, Humza, Khan, Asad Ullah, Qiu, Shi, Saqib, Muhammad, Anwar, Saeed, Usman, Muhammad, Akhtar, Naveed, Barnes, Nick +1 more·2023·arXiv
DOI 10.48550/arxiv.2307.06435arXiv 2307.06435
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the existing literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to not only provide a systematic survey but also a quick comprehensive reference for the researchers and practitioners to draw insights from extensive informative summaries of the existing works to advance the LLM research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.06435

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large language models have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks.

    “On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks.”

    From LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024) · cites this paper as Naveed et al., 2023

    Section 6 (Applications; page unavailable) describes LLMs drafting emails, generating text, answering questions, and finding patterns in text that would take humans much longer to identify. The copy does not establish that LLMs themselves perform drawing or that they significantly reduce the time required for many routine tasks generally; its discussion of image-related tools does not establish those broader claims. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 450

  1. A. Chernyavskiy, D. Ilvovsky, P. Nakov, Transformers:"the end of his- tory" for natural language processing?, in: Machine Learning and Knowledge Discovery in Databases. Research Track: European Con- ference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, Springer, 2021, pp. 677-693.
  2. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. Bowman, Superglue: A stickier benchmark for general- purpose language understanding systems, Advances in neural informa- tion processing systems 32 (2019). 1, 26, 29
  3. D. Adiwardana, M.-T. Luong, D. R. So, J. Hall, N. Fiedel, R. Thoppilan, Z. Yang, A. Kulshreshtha, G. Nemade, Y. Lu, et al., Towards a human- like open-domain chatbot, arXiv preprint arXiv:2001.09977 (2020). · arXiv
  4. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing sys- tems 33 (2020) 1877-1901. 2, 6, 7, 8, 9, 16, 18, 23, 24, 25, 34
  5. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). 2, 18, 24 · arXiv
  6. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, in: NAACL- HLT, Association for Computational Linguistics, 2018, pp. 2227-2237.
  7. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehen- sion, arXiv preprint arXiv:1910.13461 (2019). 2 · arXiv
  8. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Re- search 21 (1) (2020) 5485-5551. 2, 7, 8, 18, 19, 24, 25, 28, 30, 31
  9. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Raffel, mt5: A massively multilingual pre-trained text-to- text transformer, arXiv preprint arXiv:2010.11934 (2020). 2, 7, 8, 24, 25, 28, 30 · arXiv
  10. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al., Bloom: A 176b- parameter open-access multilingual language model, arXiv preprint arXiv:2211.05100 (2022). 2, 4, 9, 11, 23, 24, 25, 30 · arXiv
  11. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al., Opt: Open pre-trained transformer language models, arXiv preprint arXiv:2205.01068 (2022). 2, 9, 11, 24, 25 · arXiv
  12. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., Palm: Scal- ing language modeling with pathways, arXiv preprint arXiv:2204.02311 (2022). 2, 6, 9, 11, 23, 24, 25 · arXiv
  13. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al., Scaling instruction-finetuned language models, arXiv preprint arXiv:2210.11416 (2022). 2, 7, 11, 16, 17, 22, 24, 25, 28, 31 · arXiv
  14. V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al., Multitask prompted training enables zero-shot task generalization, arXiv preprint arXiv:2110.08207 (2021). 2, 11, 16, 25, 28, 31 · arXiv
  15. Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al., Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 5085-5109. 2, 7, 11, 16, 17, 24, 25, 28, 31
  16. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, H. Ha- jishirzi, Self-instruct: Aligning language model with self generated in- structions, arXiv preprint arXiv:2212.10560 (2022). 2, 16, 19, 22, 28 · arXiv
  17. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language mod- els to follow instructions with human feedback, Advances in Neural In- formation Processing Systems 35 (2022) 27730-27744. 2, 7, 11, 16, 22
  18. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 (2023). 2, 7, 10, 16, 25, 34 · arXiv
  19. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yo- gatama, M. Bosma, D. Zhou, D. Metzler, et al., Emergent abilities of large language models, arXiv preprint arXiv:2206.07682 (2022). 2 · arXiv
  20. D. A. Boiko, R. MacKnight, G. Gomes, Emergent autonomous sci- entific research capabilities of large language models, arXiv preprint arXiv:2304.05332 (2023). 2 · arXiv
  21. G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, E. Grave, Few-shot learning with retrieval augmented language models, arXiv preprint arXiv:2208.03299 (2022). 2, 18, 19, 34 · arXiv
  22. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al., Palm-e: An embodied multimodal language model, arXiv preprint arXiv:2303.03378 (2023). 2, 20, 22, 33 · arXiv
  23. A. Parisi, Y. Zhao, N. Fiedel, Talm: Tool augmented language models, arXiv preprint arXiv:2205.12255 (2022). 2, 19, 20 · arXiv
  24. B. Zhang, H. Soh, Large language models as zero-shot human models for human-robot interaction, arXiv preprint arXiv:2303.03548 (2023). 2, 33 · arXiv
  25. Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al., mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304.14178 (2023). 2, 22 · arXiv
  26. W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao, et al., Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, arXiv preprint arXiv:2305.11175 (2023). 2, 22 · arXiv
  27. R. Yang, L. Song, Y. Li, S. Zhao, Y. Ge, X. Li, Y. Shan, Gpt4tools: Teaching large language model to use tools via self-instruction, arXiv preprint arXiv:2305.18752 (2023). 2, 19, 22, 23 · arXiv
  28. A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, et al., Glm-130b: An open bilingual pre-trained model, arXiv preprint arXiv:2210.02414 (2022). 2, 10, 23, 24, 25 · arXiv
  29. Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, S. C. Hoi, Codet5+: Open code large language models for code understanding and genera- tion, arXiv preprint arXiv:2305.07922 (2023). 2, 11, 24, 25 · arXiv
  30. S. Wang, Y. Sun, Y. Xiang, Z. Wu, S. Ding, W. Gong, S. Feng, J. Shang, Y. Zhao, C. Pang, et al., Ernie 3.0 titan: Exploring larger-scale knowl- edge enhanced pre-training for language understanding and generation, arXiv preprint arXiv:2112.12731 (2021). 2, 8, 24, 25 · arXiv
  31. J. Rasley, S. Rajbhandari, O. Ruwase, Y. He, Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters, in: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 3505- 3506. 2, 5
  32. S. Rajbhandari, J. Rasley, O. Ruwase, Y. He, Zero: Memory optimiza- tions toward training trillion parameter models, in: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, IEEE, 2020, pp. 1-16. 2, 4, 24
  33. J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, G. Neubig, Towards a unified view of parameter-efficient transfer learning, arXiv preprint arXiv:2110.04366 (2021). 2, 20, 21 · arXiv
  34. Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, S. Po- ria, Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models, arXiv preprint arXiv:2304.01933 (2023). 2, 20 · arXiv
  35. B. Lester, R. Al-Rfou, N. Constant, The power of scale for parameter- efficient prompt tuning, arXiv preprint arXiv:2104.08691 (2021). 2, 8, 20, 21 · arXiv
  36. X. L. Li, P. Liang, Prefix-tuning: Optimizing continuous prompts for generation, arXiv preprint arXiv:2101.00190 (2021). 2, 20, 21 · arXiv
  37. X. Ma, G. Fang, X. Wang, Llm-pruner: On the structural pruning of large language models, arXiv preprint arXiv:2305.11627 (2023). 2, 22 · arXiv
  38. R. Xu, F. Luo, C. Wang, B. Chang, J. Huang, S. Huang, F. Huang, From dense to sparse: Contrastive pruning for better pre-trained lan- guage model compression, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, 2022, pp. 11547-11555. 2, 22
  39. G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, S. Han, Smoothquant: Accurate and efficient post-training quantization for large language models, in: ICML, Vol. 202 of Proceedings of Machine Learning Re- search, PMLR, 2023, pp. 38087-38099. 2, 21
  40. C. Tao, L. Hou, W. Zhang, L. Shang, X. Jiang, Q. Liu, P. Luo, N. Wong, Compression of generative pre-trained language models via quantiza- tion, arXiv preprint arXiv:2203.10705 (2022). 2, 21 · arXiv
  41. A. Pal, D. Karkhanis, M. Roberts, S. Dooley, A. Sundararajan, S. Naidu, Giraffe: Adventures in expanding context lengths in llms, arXiv preprint arXiv:2308.10882 (2023). 2, 17 · arXiv
  42. B. Peng, J. Quesnelle, H. Fan, E. Shippole, Yarn: Efficient con- text window extension of large language models, arXiv preprint arXiv:2309.00071 (2023). 2, 17 · arXiv
  43. M. Guo, J. Ainslie, D. Uthus, S. Ontanon, J. Ni, Y.-H. Sung, Y. Yang, Longt5: Efficient text-to-text transformer for long sequences, arXiv preprint arXiv:2112.07916 (2021). 2, 18 · arXiv
  44. S. Chen, S. Wong, L. Chen, Y. Tian, Extending context window of large language models via positional interpolation, arXiv preprint arXiv:2306.15595 (2023). 2, 17 · arXiv
  45. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al., A survey of large language models, arXiv preprint arXiv:2303.18223 (2023). 2, 3, 7 · arXiv
  46. U. Naseem, I. Razzak, S. K. Khan, M. Prasad, A comprehensive sur- vey on word representation models: From classical to state-of-the-art word representation language models, Transactions on Asian and Low- Resource Language Information Processing 20 (5) (2021) 1-35. 2, 3
  47. B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heinz, D. Roth, Recent advances in natural language pro- cessing via large pre-trained language models: A survey, arXiv preprint arXiv:2111.01243 (2021). 2, 3 · arXiv
  48. C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, et al., A comprehensive survey on pretrained foundation models: A history from bert to chatgpt, arXiv preprint arXiv:2302.09419 (2023). 2, 3 · arXiv
  49. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, Z. Sui, A survey for in-context learning, arXiv preprint arXiv:2301.00234 (2022). 2, 7, 18 · arXiv
  50. J. Huang, K. C.-C. Chang, Towards reasoning in large language models: A survey, arXiv preprint arXiv:2212.10403 (2022). 2, 7, 18 · arXiv
  51. Y. Wang, W. Zhong, L. Li, F. Mi, X. Zeng, W. Huang, L. Shang, X. Jiang, Q. Liu, Aligning large language models with human: A survey, arXiv preprint arXiv:2307.12966 (2023). 2 · arXiv
  52. X. Zhu, J. Li, Y. Liu, C. Ma, W. Wang, A survey on model compression for large language models, arXiv preprint arXiv:2308.07633 (2023). · arXiv
  53. S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, E. Chen, A survey on multi- modal large language models, arXiv preprint arXiv:2306.13549 (2023). 2, 22, 23 · arXiv
  54. J. J. Webster, C. Kit, Tokenization as the initial phase in nlp, in: COL- ING 1992 volume 4: The 14th international conference on computa- tional linguistics, 1992. 4
  55. T. Kudo, Subword regularization: Improving neural network translation models with multiple subword candidates, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), 2018, pp. 66-75. 4
  56. R. Sennrich, B. Haddow, A. Birch, Neural machine translation of rare words with subword units, in: Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp. 1715-1725. 4
  57. M. Schuster, K. Nakajima, Japanese and korean voice search, in: 2012 IEEE international conference on acoustics, speech and signal process- ing (ICASSP), IEEE, 2012, pp. 5149-5152. 4
  58. S. J. Mielke, Z. Alyafeai, E. Salesky, C. Raffel, M. Dey, M. Gallé, A. Raja, C. Si, W. Y. Lee, B. Sagot, et al., Between words and char- acters: A brief history of open-vocabulary modeling and tokenization in nlp, arXiv preprint arXiv:2112.10508 (2021). 4 · arXiv
  59. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). 4, 7
  60. O. Press, N. Smith, M. Lewis, Train short, test long: Attention with linear biases enables input length extrapolation, in: International Con- ference on Learning Representations, 2022. URL https://openreview.net/forum?id=R8sQPpGCv0 4, 17
  61. J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu, Roformer: En- hanced transformer with rotary position embedding, arXiv preprint arXiv:2104.09864 (2021). 4, 9, 17 · arXiv
  62. R. Child, S. Gray, A. Radford, I. Sutskever, Generating long sequences with sparse transformers, arXiv preprint arXiv:1904.10509 (2019). 4, 7, 23 · arXiv
  63. T. Dao, D. Fu, S. Ermon, A. Rudra, C. Ré, Flashattention: Fast and memory-efficient exact attention with io-awareness, Advances in Neural Information Processing Systems 35 (2022) 16344-16359.
  64. V. Nair, G. E. Hinton, Rectified linear units improve restricted boltz- mann machines, in: Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807-814. 4
  65. D. Hendrycks, K. Gimpel, Gaussian error linear units (gelus), arXiv preprint arXiv:1606.08415 (2016). · arXiv
  66. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research 15 (1) (2014) 1929-1958. 4
  67. D. Krueger, T. Maharaj, J. Kramár, M. Pezeshki, N. Ballas, N. R. Ke, A. Goyal, Y. Bengio, A. Courville, C. Pal, Zoneout: Regular- izing rnns by randomly preserving hidden activations, arXiv preprint arXiv:1606.01305 (2016). 4 · arXiv
  68. N. Shazeer, Glu variants improve transformer, arXiv preprint arXiv:2002.05202 (2020). 4 · arXiv
  69. Y. N. Dauphin, A. Fan, M. Auli, D. Grangier, Language modeling with gated convolutional networks, in: International conference on machine learning, PMLR, 2017, pp. 933-941. 4
  70. J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization, arXiv preprint arXiv:1607.06450 (2016). 4 · arXiv
  71. B. Zhang, R. Sennrich, Root mean square layer normalization, Advances in Neural Information Processing Systems 32 (2019). 4
  72. A. Baevski, M. Auli, Adaptive input representations for neural language modeling, arXiv preprint arXiv:1809.10853 (2018). · arXiv
  73. H. Wang, S. Ma, L. Dong, S. Huang, D. Zhang, F. Wei, Deepnet: Scaling transformers to 1,000 layers, arXiv preprint arXiv:2203.00555 (2022). · arXiv
  74. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, B. Catanzaro, Megatron-lm: Training multi-billion parameter language models using model parallelism, arXiv preprint arXiv:1909.08053 (2019). 4, 5 · arXiv
  75. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cis- tac, T. Rault, R. Louf, M. Funtowicz, et al., Transformers: State-of-the- art natural language processing, in: Proceedings of the 2020 conference on empirical methods in natural language processing: system demon- strations, 2020, pp. 38-45. 5
  76. S. Li, J. Fang, Z. Bian, H. Liu, Y. Liu, H. Huang, B. Wang, Y. You, Colossal-ai: A unified deep learning system for large-scale parallel train- ing, arXiv preprint arXiv:2110.14883 (2021). 5 · arXiv
  77. J. He, J. Qiu, A. Zeng, Z. Yang, J. Zhai, J. Tang, Fastmoe: A fast mixture-of-expert training system, arXiv preprint arXiv:2103.13262 (2021). 5 · arXiv
  78. L. Huawei Technologies Co., Huawei mindspore ai development frame- work, in: Artificial Intelligence Technology, Springer, 2022, pp. 137- 162.
  79. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imper- ative style, high-performance deep learning library, Advances in neural information processing systems 32 (2019). 5
  80. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., Tensorflow: a system for large- scale machine learning., in: Osdi, Vol. 16, Savannah, GA, USA, 2016, pp. 265-283. 5
  81. T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, Z. Zhang, Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems, arXiv preprint arXiv:1512.01274 (2015). 5 · arXiv
  82. W. Fedus, B. Zoph, N. Shazeer, Switch transformers: Scaling to tril- lion parameter models with simple and efficient sparsity, The Journal of Machine Learning Research 23 (1) (2022) 5232-5270. 5, 9
  83. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al., Glam: Efficient scaling of language models with mixture-of-experts, in: International Conference on Ma- chine Learning, PMLR, 2022, pp. 5547-5569. 5, 9, 23, 24, 25
  84. X. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, et al., Pangu-: Towards trillion parameter language model with sparse heterogeneous computing, arXiv preprint arXiv:2303.10845 (2023). 5, 10, 16, 23, 24, 25 · arXiv
  85. T. Wang, A. Roberts, D. Hesslow, T. Le Scao, H. W. Chung, I. Beltagy, J. Launay, C. Raffel, What language model architecture and pretrain- ing objective works best for zero-shot generalization?, in: International Conference on Machine Learning, PMLR, 2022, pp. 22964-22984.
  86. L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, H.-W. Hon, Unified language model pre-training for natural language understanding and generation, Advances in neural information process- ing systems 32 (2019). 6
  87. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020). 6 · arXiv
  88. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, arXiv preprint arXiv:2203.15556 (2022). 6, 9, 25, 29 · arXiv
  89. S. Iyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang, Q. Liu, P. S. Koura, et al., Opt-iml: Scaling language model in- struction meta learning through the lens of generalization, arXiv preprint arXiv:2212.12017 (2022). 7, 11, 16, 17, 22, 25, 28 · arXiv
  90. Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, C. Gan, Principle-driven self-alignment of language models from scratch with minimal human supervision, arXiv preprint arXiv:2305.03047 (2023). 7, 17 · arXiv
  91. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al., A general language assistant as a laboratory for alignment, arXiv preprint arXiv:2112.00861 (2021). 7 · arXiv
  92. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, G. Irving, Fine-tuning language models from human pref- erences, arXiv preprint arXiv:1909.08593 (2019). 7 · arXiv
  93. S. Kim, S. J. Joo, D. Kim, J. Jang, S. Ye, J. Shin, M. Seo, The cot collec- tion: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning, arXiv preprint arXiv:2305.14045 (2023). 7, 16 · arXiv
  94. Q. Liu, F. Zhou, Z. Jiang, L. Dou, M. Lin, From zero to hero: Exam- ining the power of symbolic tasks in instruction tuning, arXiv preprint arXiv:2304.07995 (2023). 7, 16 · arXiv
  95. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in Neural Information Processing Systems 35 (2022) 24824-24837. 7, 20, 23
  96. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowd- hery, D. Zhou, Self-consistency improves chain of thought reasoning in language models, arXiv preprint arXiv:2203.11171 (2022). 7, 20 · arXiv
  97. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, K. Narasimhan, Tree of thoughts: Deliberate problem solving with large language mod- els, arXiv preprint arXiv:2305.10601 (2023). 7, 20 · arXiv
  98. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, S. Gelly, Parameter-efficient transfer learn- ing for nlp, in: International Conference on Machine Learning, PMLR, 2019, pp. 2790-2799. 7, 20
  99. S. McCandlish, J. Kaplan, D. Amodei, O. D. Team, An empirical model of large-batch training, arXiv preprint arXiv:1812.06162 (2018). 7 · arXiv
  100. W. Zeng, X. Ren, T. Su, H. Wang, Y. Liao, Z. Wang, X. Jiang, Z. Yang, K. Wang, X. Zhang, et al., Pangu-α : Large-scale autoregressive pre- trained chinese language models with auto-parallel computation, arXiv preprint arXiv:2104.12369 (2021). 8, 23, 24, 25 · arXiv
  101. Y. Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao, Y. Lu, et al., Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation, arXiv preprint arXiv:2107.02137 (2021). 8, 25 · arXiv
  102. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, R. Salakhutdinov, Transformer-xl: Attentive language models beyond a fixed-length con- text, arXiv preprint arXiv:1901.02860 (2019). 8 · arXiv
  103. Y. Levine, N. Wies, O. Sharir, H. Bata, A. Shashua, Limits to depth ef- ficiencies of self-attention, Advances in Neural Information Processing Systems 33 (2020) 22640-22651. 8, 11
  104. B. Kim, H. Kim, S.-W. Lee, G. Lee, D. Kwak, D. H. Jeon, S. Park, S. Kim, S. Kim, D. Seo, et al., What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers, arXiv preprint arXiv:2109.04650 (2021). 8, 25 · arXiv
  105. S. Wu, X. Zhao, T. Yu, R. Zhang, C. Shen, H. Liu, F. Li, H. Zhu, J. Luo, L. Xu, et al., Yuan 1.0: Large-scale pre-trained language model in zero- shot and few-shot learning, arXiv preprint arXiv:2110.04725 (2021). 8, 24, 25 · arXiv
  106. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al., Scaling lan- guage models: Methods, analysis & insights from training gopher, arXiv preprint arXiv:2112.11446 (2021). 8, 9, 25, 28 · arXiv
  107. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al., Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, arXiv preprint arXiv:2201.11990 (2022). 8, 9, 24, 25 · arXiv
  108. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al., Gpt-neox-20b: An open- source autoregressive language model, arXiv preprint arXiv:2204.06745 (2022). 9, 23, 24, 25 · arXiv
  109. P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al., Mixed pre- cision training, arXiv preprint arXiv:1710.03740 (2017). 9, 23 · arXiv
  110. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hin- ton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, arXiv preprint arXiv:1701.06538 (2017). 9, 23 · arXiv
  111. S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky, et al., Alex- atm 20b: Few-shot learning using a large-scale multilingual seq2seq model, arXiv preprint arXiv:2208.01448 (2022). 9, 23, 24, 25 · arXiv
  112. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al., Palm 2 technical report, arXiv preprint arXiv:2305.10403 (2023). 9, 25 · arXiv
  113. Y. Tay, J. Wei, H. W. Chung, V. Q. Tran, D. R. So, S. Shakeri, X. Garcia, H. S. Zheng, J. Rao, A. Chowdhery, et al., Transcending scaling laws with 0.1% extra compute, arXiv preprint arXiv:2210.11399 (2022). 9, 24, 25 · arXiv
  114. Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al., Ul2: Unifying lan- guage learning paradigms, in: The Eleventh International Conference on Learning Representations, 2022. 9, 10, 24, 25
  115. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, J. Tang, Glm: Gen- eral language model pretraining with autoregressive blank infilling, in: Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 2022, pp. 320-335. 10
  116. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023). 10, 23, 25 · arXiv
  117. M. N. Rabe, C. Staats, Self-attention does not need o(n 2 ) memory, arXiv preprint arXiv:2112.05682 (2021). 10 · arXiv
  118. V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, B. Catanzaro, Reducing activation recomputation in large transformer models, Proceedings of Machine Learning and Systems 5 (2023). 10
  119. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., The llama 3 herd of models, arXiv preprint arXiv:2407.21783 (2024). 10, 25 [131] https://mistral.ai/news/mixtral-8x22b/. 10, 25 [132https://github.com/Snowflake-Labs/snowflake-arctic. 10, 25 [133] https://github.com/xai-org/grok-1. 10 [134https://x.ai/blog/grok-1.5. 10 · arXiv
  120. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al., Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 (2023). 10 · arXiv
  121. M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al., Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.05530 (2024). 10 · arXiv
  122. B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brun- dyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen, et al., Nemotron-4 340b technical report, arXiv preprint arXiv:2406.11704 (2024). 10, 25 · arXiv
  123. X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al., Deepseek llm: Scaling open-source language models with longtermism, arXiv preprint arXiv:2401.02954 (2024). 10, 25 · arXiv
  124. DeepSeek-AI, A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Yang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Chen, J. Yuan, J. Qiu, J. Song, K. Dong, K. Gao, K. Guan, L. Wang, L. Zhang, L. Xu, L. Xia, L. Zhao, L. Zhang, M. Li, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, N. Tian, P. Huang, P. Wang, P. Zhang, Q. Zhu, Q. Chen, Q. Du, R. J. Chen, R. L. Jin, R. Ge, R. Pan, R. Xu, R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Ma, S. Wang, S. Zhou, S. Yu, S. Zhou, S. Zheng, T. Wang, T. Pei, T. Yuan, T. Sun, W. L. Xiao, W. Zeng, W. An, W. Liu, W. Liang, W. Gao, W. Zhang, X. Q. Li, X. Jin, X. Wang, X. Bi, X. Liu, X. Wang, X. Shen, X. Chen, X. Chen, X. Nie, X. Sun, Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, CoRR abs/2405.04434 (2024). 10, 25 · arXiv
  125. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, C. Xiong, Codegen: An open large language model for code with multi- turn program synthesis, arXiv preprint arXiv:2203.13474 (2022). 11, 23, 25, 28 · arXiv
  126. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Ed- wards, Y. Burda, N. Joseph, G. Brockman, et al., Evaluating large lan- guage models trained on code, arXiv preprint arXiv:2107.03374 (2021). 11, 25, 29, 31 · arXiv
  127. N. Shazeer, Fast transformer decoding: One write-head is all you need, arXiv preprint arXiv:1911.02150 (2019). 11 · arXiv
  128. R. Y. Pang, H. He, Text generation by learning from demonstrations, arXiv preprint arXiv:2009.07839 (2020). 11 · arXiv
  129. R. Dabre, A. Fujita, Softmax tempering for training neural machine translation models, arXiv preprint arXiv:2009.09372 (2020). 11 · arXiv
  130. Y. Wang, W. Wang, S. Joty, S. C. Hoi, Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and genera- tion, arXiv preprint arXiv:2109.00859 (2021). 11 · arXiv
  131. R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al., Starcoder: may the source be with you!, arXiv preprint arXiv:2305.06161 (2023). 11, 25 · arXiv
  132. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, R. Stojnic, Galactica: A large language model for science, arXiv preprint arXiv:2211.09085 (2022). 11, 24, 25, 29 · arXiv
  133. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al., Lamda: Language models for dialog applications, arXiv preprint arXiv:2201.08239 (2022). 11, 25 · arXiv
  134. S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, G. Mann, Bloomberggpt: A large language model for finance, arXiv preprint arXiv:2303.17564 (2023). 11, 25, 33 · arXiv
  135. X. Zhang, Q. Yang, D. Xu, Xuanyuan 2.0: A large chinese finan- cial chat model with hundreds of billions parameters, arXiv preprint arXiv:2305.12002 (2023). 11, 17, 25 · arXiv
  136. N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, et al., Crosslingual generalization through multitask finetuning, arXiv preprint arXiv:2211.01786 (2022). 16, 25, 28, 31 · arXiv
  137. D. Yin, X. Liu, F. Yin, M. Zhong, H. Bansal, J. Han, K.-W. Chang, Dynosaur: A dynamic growth paradigm for instruction-tuning data cu- ration, arXiv preprint arXiv:2305.14327 (2023). 16 · arXiv
  138. P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue, et al., Llama-adapter v2: Parameter-efficient visual in- struction model, arXiv preprint arXiv:2304.15010 (2023). 16, 24 [157] Openai. gpt-4 technical report (2023). 16, 35 · arXiv
  139. B. Peng, C. Li, P. He, M. Galley, J. Gao, Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277 (2023). 16, 28 · arXiv
  140. T. Liu, B. K. H. Low, Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks, arXiv preprint arXiv:2305.14201 (2023). 16 · arXiv
  141. H. Wang, C. Liu, N. Xi, Z. Qiang, S. Zhao, B. Qin, T. Liu, Huatuo: Tuning llama model with chinese medical knowledge, arXiv preprint arXiv:2304.06975 (2023). 16 · arXiv
  142. C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, D. Jiang, Wizardlm: Empowering large language models to follow complex in- structions, arXiv preprint arXiv:2304.12244 (2023). 16 · arXiv
  143. Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, D. Jiang, Wizardcoder: Empowering code large language models with evol-instruct, arXiv preprint arXiv:2306.08568 (2023). 16, 25 · arXiv
  144. J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell-Gillingham, G. Irving, et al., Teach- ing language models to support answers with verified quotes, arXiv preprint arXiv:2203.11147 (2022). 17 · arXiv
  145. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al., Webgpt: Browser- assisted question-answering with human feedback, arXiv preprint arXiv:2112.09332 (2021). 17, 19, 20, 25, 31 · arXiv
  146. A. Glaese, N. McAleese, M. Tr ębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, et al., Improving alignment of dialogue agents via targeted human judgements, arXiv preprint arXiv:2209.14375 (2022). 17, 20, 25 · arXiv
  147. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, C. Finn, Direct preference optimization: Your language model is secretly a re- ward model, arXiv preprint arXiv:2305.18290 (2023). 17 · arXiv
  148. H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao, J. Zhang, K. Shum, T. Zhang, Raft: Reward ranked finetuning for generative foundation model alignment, arXiv preprint arXiv:2304.06767 (2023). 17 · arXiv
  149. Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, F. Huang, Rrhf: Rank responses to align language models with human feedback without tears, arXiv preprint arXiv:2304.05302 (2023). 17 · arXiv
  150. F. Song, B. Yu, M. Li, H. Yu, F. Huang, Y. Li, H. Wang, Preference rank- ing optimization for human alignment, arXiv preprint arXiv:2306.17492 (2023). 17 · arXiv
  151. H. Liu, C. Sferrazza, P. Abbeel, Languages are rewards: Hindsight fine- tuning using human feedback, arXiv preprint arXiv:2302.02676 (2023). 17 · arXiv
  152. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al., Constitutional ai: Harm- lessness from ai feedback, arXiv preprint arXiv:2212.08073 (2022). 17 · arXiv
  153. Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, T. B. Hashimoto, Alpacafarm: A simulation frame- work for methods that learn from human feedback, arXiv preprint arXiv:2305.14387 (2023). 17 · arXiv
  154. C. Si, Z. Gan, Z. Yang, S. Wang, J. Wang, J. Boyd-Graber, L. Wang, Prompting gpt-3 to be reliable, arXiv preprint arXiv:2210.09150 (2022). 17 · arXiv
  155. D. Ganguli, A. Askell, N. Schiefer, T. Liao, K. Lukošiūtė, A. Chen, A. Goldie, A. Mirhoseini, C. Olsson, D. Hernandez, et al., The capac- ity for moral self-correction in large language models, arXiv preprint arXiv:2302.07459 (2023). 17 · arXiv
  156. A. Wei, N. Haghtalab, J. Steinhardt, Jailbroken: How does llm safety training fail?, arXiv preprint arXiv:2307.02483 (2023). 17 · arXiv
  157. D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al., Red teaming lan- guage models to reduce harms: Methods, scaling behaviors, and lessons learned, arXiv preprint arXiv:2209.07858 (2022). 17, 28 · arXiv
  158. S. Casper, J. Lin, J. Kwon, G. Culp, D. Hadfield-Menell, Explore, estab- lish, exploit: Red teaming language models from scratch, arXiv preprint arXiv:2306.09442 (2023). 17 · arXiv
  159. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, G. Irving, Red teaming language models with language models, arXiv preprint arXiv:2202.03286 (2022). 17 · arXiv
  160. T. Scialom, T. Chakrabarty, S. Muresan, Fine-tuned language models are continual learners, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 6107-6122. 17
  161. Z. Shi, A. Lipani, Don't stop pretraining? make prompt-based fine- tuning powerful learner, arXiv preprint arXiv:2305.01711 (2023). 17 · arXiv
  162. H. Gupta, S. A. Sawant, S. Mishra, M. Nakamura, A. Mitra, S. Mashetty, C. Baral, Instruction tuned models are quick learners, arXiv preprint arXiv:2306.05539 (2023). 17 · arXiv
  163. H. Chen, Y. Zhang, Q. Zhang, H. Yang, X. Hu, X. Ma, Y. Yanggong, J. Zhao, Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning, arXiv preprint arXiv:2305.09246 (2023). 17 · arXiv
  164. C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al., Lima: Less is more for alignment, arXiv preprint arXiv:2305.11206 (2023). 17, 25, 28 · arXiv
  165. C. Han, Q. Wang, W. Xiong, Y. Chen, H. Ji, S. Wang, Lm-infinite: Sim- ple on-the-fly length generalization for large language models, arXiv preprint arXiv:2308.16137 (2023). 17, 18 · arXiv
  166. J. Ainslie, T. Lei, M. de Jong, S. Ontañón, S. Brahma, Y. Zemlyan- skiy, D. Uthus, M. Guo, J. Lee-Thorp, Y. Tay, et al., Colt5: Faster long-range transformers with conditional computation, arXiv preprint arXiv:2303.09752 (2023). 18 · arXiv
  167. J. Ding, S. Ma, L. Dong, X. Zhang, S. Huang, W. Wang, F. Wei, Longnet: Scaling transformers to 1,000,000,000 tokens, arXiv preprint arXiv:2307.02486 (2023). 18 · arXiv
  168. Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, J. Jia, Longlora: Effi- cient fine-tuning of long-context large language models, arXiv preprint arXiv:2309.12307 (2023). 18 · arXiv
  169. N. Ratner, Y. Levine, Y. Belinkov, O. Ram, I. Magar, O. Abend, E. Karpas, A. Shashua, K. Leyton-Brown, Y. Shoham, Parallel context windows for large language models, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 6383-6402. 18
  170. W. Wang, L. Dong, H. Cheng, X. Liu, X. Yan, J. Gao, F. Wei, Augmenting language models with long-term memory, arXiv preprint arXiv:2306.07174 (2023). 18 · arXiv
  171. X. Xu, Z. Gou, W. Wu, Z.-Y. Niu, H. Wu, H. Wang, S. Wang, Long time no see! open-domain conversation with long-term persona memory, arXiv preprint arXiv:2203.05797 (2022). 18 · arXiv
  172. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Milli- can, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al., Improving language models by retrieving from trillions of tokens, in: International conference on machine learning, PMLR, 2022, pp. 2206- 2240. 18, 19, 34
  173. W. Zhong, L. Guo, Q. Gao, Y. Wang, Memorybank: Enhanc- ing large language models with long-term memory, arXiv preprint arXiv:2305.10250 (2023). 18 · arXiv
  174. N. Shinn, F. Cassano, B. Labash, A. Gopinath, K. Narasimhan, S. Yao, Reflexion: Language agents with verbal reinforcement learning, arXiv preprint arXiv:2303.11366 14 (2023). 18, 20 · arXiv
  175. C. Hu, J. Fu, C. Du, S. Luo, J. Zhao, H. Zhao, Chatdb: Augment- ing llms with databases as their symbolic memory, arXiv preprint arXiv:2306.03901 (2023). 18 · arXiv
  176. Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, G. Neubig, Active retrieval augmented generation, arXiv preprint arXiv:2305.06983 (2023). 18 · arXiv
  177. O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton- Brown, Y. Shoham, In-context retrieval-augmented language models, arXiv preprint arXiv:2302.00083 (2023). 18, 34 · arXiv
  178. X. Li, X. Qiu, Mot: Pre-thinking and recalling enable chatgpt to self- improve with memory-of-thoughts, arXiv preprint arXiv:2305.05181 (2023). 18 · arXiv
  179. D. Schuurmans, Memory augmented large language models are compu- tationally universal, arXiv preprint arXiv:2301.04589 (2023). 18 · arXiv
  180. A. Modarressi, A. Imani, M. Fayyaz, H. Schütze, Ret-llm: Towards a general read-write memory for large language models, arXiv preprint arXiv:2305.14322 (2023). 18 · arXiv
  181. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, D. Zhou, Rationale-augmented ensembles in language models, arXiv preprint arXiv:2207.00747 (2022). 18 · arXiv
  182. F. Zhang, B. Chen, Y. Zhang, J. Liu, D. Zan, Y. Mao, J.-G. Lou, W. Chen, Repocoder: Repository-level code completion through iterative retrieval and generation, arXiv preprint arXiv:2303.12570 (2023). 18 · arXiv
  183. B. Wang, W. Ping, P. Xu, L. McAfee, Z. Liu, M. Shoeybi, Y. Dong, O. Kuchaiev, B. Li, C. Xiao, et al., Shall we pretrain autoregressive language models with retrieval? a comprehensive study, arXiv preprint arXiv:2304.06762 (2023). 19 · arXiv
  184. L. Wang, N. Yang, F. Wei, Learning to retrieve in-context examples for large language models, arXiv preprint arXiv:2307.07164 (2023). 19 · arXiv
  185. J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, W. Chen, What makes good in-context examples for gpt-3?, arXiv preprint arXiv:2101.06804 (2021). 19 · arXiv
  186. O. Rubin, J. Herzig, J. Berant, Learning to retrieve prompts for in- context learning, arXiv preprint arXiv:2112.08633 (2021). 19 · arXiv
  187. W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettle- moyer, W.-t. Yih, Replug: Retrieval-augmented black-box language models, arXiv preprint arXiv:2301.12652 (2023). 19 · arXiv
  188. O. Rubin, J. Berant, Long-range language modeling with self-retrieval, arXiv preprint arXiv:2306.13421 (2023). 19 · arXiv
  189. K. Guu, K. Lee, Z. Tung, P. Pasupat, M. Chang, Retrieval augmented language model pre-training, in: International conference on machine learning, PMLR, 2020, pp. 3929-3938. 19
  190. S. Hofstätter, J. Chen, K. Raman, H. Zamani, Fid-light: Efficient and ef- fective retrieval-augmented text generation, in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 1437-1447. 19
  191. M. Komeili, K. Shuster, J. Weston, Internet-augmented dialogue gener- ation, arXiv preprint arXiv:2107.07566 (2021). 19 · arXiv
  192. A. Lazaridou, E. Gribovskaya, W. Stokowiec, N. Grigorev, Internet- augmented language models through few-shot prompting for open- domain question answering, arXiv preprint arXiv:2203.05115 (2022). 19 · arXiv
  193. D. Gao, L. Ji, L. Zhou, K. Q. Lin, J. Chen, Z. Fan, M. Z. Shou, Assist- gpt: A general multi-modal assistant that can plan, execute, inspect, and learn, arXiv preprint arXiv:2306.08640 (2023). 19 · arXiv
  194. P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, J. Gao, Chameleon: Plug-and-play compositional reasoning with large language models, arXiv preprint arXiv:2304.09842 (2023). 19, 20, 23 · arXiv
  195. B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, M. T. Ribeiro, Art: Automatic multi-step reasoning and tool-use for large lan- guage models, arXiv preprint arXiv:2303.09014 (2023). 19 · arXiv
  196. C.-Y. Hsieh, S.-A. Chen, C.-L. Li, Y. Fujii, A. Ratner, C.-Y. Lee, R. Kr- ishna, T. Pfister, Tool documentation enables zero-shot tool-usage with large language models, arXiv preprint arXiv:2308.00675 (2023). 19 · arXiv
  197. Y. Song, W. Xiong, D. Zhu, C. Li, K. Wang, Y. Tian, S. Li, Restgpt: Connecting large language models with real-world applications via rest- ful apis, arXiv preprint arXiv:2306.06624 (2023). 19 · arXiv
  198. S. Hao, T. Liu, Z. Wang, Z. Hu, Toolkengpt: Augmenting frozen lan- guage models with massive tools via tool embeddings, arXiv preprint arXiv:2305.11554 (2023). 19 · arXiv
  199. S. G. Patil, T. Zhang, X. Wang, J. E. Gonzalez, Gorilla: Large language model connected with massive apis, arXiv preprint arXiv:2305.15334 (2023). 19 · arXiv
  200. Q. Xu, F. Hong, B. Li, C. Hu, Z. Chen, J. Zhang, On the tool manipu- lation capability of open-source large language models, arXiv preprint arXiv:2305.16504 (2023). 19 · arXiv

200 of 450 shown

Cited by

Sorted by citations

Authors
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A.
Affiliations
  • The University of Sydney (AU)
  • University of Engineering and Technology Lahore (PK)
  • Chinese University of Hong Kong (HK)
  • University of Technology Sydney (AU)
  • King Fahd University of Petroleum and Minerals (SA)
  • Ontario Tech University (CA)
  • The University of Melbourne (AU)
  • Australian National University (AU)
  • The University of Western Australia (AU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
16
Issue
5
Pages
1-72
ISSN
2157-6904 (ISSN-L)
DOI
10.48550/arxiv.2307.06435
arXiv
2307.06435
OpenAlex
W4411403346
Cited by
1,259
References
0
Citations by year
2026: 529 · 2025: 488 · 2024: 225 · 2023: 17
FWCI
406.47
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Intelligent Systems and Technology (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • UWA Profiles and Research Repository (UWA) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
King Fahd University of Petroleum and Minerals
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, LLM architecture, context length, fine-tuning, multimodal large language models, LLM benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review