CoChat
Sign Up
Cited by 1,257Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its Crossref record; 270 of the 270 references it cites were found in a registry or an index.

A Comprehensive Overview of Large Language Models

Open access
Naveed, Humza, Khan, Asad Ullah, Qiu, Shi, Saqib, Muhammad, Anwar, Saeed, Usman, Muhammad, Akhtar, Naveed, Barnes, Nick, Mian, Ajmal·2025·ACM Transactions on Intelligent Systems and Technology
DOI 10.1145/3744746
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multimodal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to provide not only a systematic survey but also a quick, comprehensive reference for the researchers and practitioners to draw insights from extensive, informative summaries of the existing works to advance the LLM research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3744746

Cites

270 of 270 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 257

  1. D. Adiwardana M.-T. Luong D. R. So J. Hall N. Fiedel R. Thoppilan Z. Yang A. Kulshreshtha G. Nemade Y. Lu et al. 2020. Towards a human-like open-domain chatbot. arXiv:2001.09977. Retrieved from https://arxiv.org/abs/2001.09977 · arXiv
  2. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
  3. 10.5555/3455716.3455856 · DOI
  4. T. L. Scao A. Fan C. Akiki E. Pavlick S. Ilić D. Hesslow R. Castagné A. S. Luccioni F. Yvon M. Gallé et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https://arxiv.org/abs/2211.05100 · arXiv
  5. S. Zhang S. Roller N. Goyal M. Artetxe M. Chen S. Chen C. Dewan M. Diab X. Li X. V. Lin et al. 2022. OPT: Open pre-trained transformer language models. arXiv:2205.01068. Retrieved from https://arxiv.org/abs/2205.01068 · arXiv
  6. A. Chowdhery S. Narang J. Devlin M. Bosma G. Mishra A. Roberts P. Barham H. W. Chung C. Sutton S. Gehrmann et al. 2022. PaLM: Scaling language modeling with pathways. arXiv:2204.02311. Retrieved from https://arxiv.org/abs/2204.02311 · arXiv
  7. V. Sanh A. Webson C. Raffel S. H. Bach L. Sutawika Z. Alyafeai A. Chaffin A. Stiegler T. L. Scao A. Raja et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv:2110.08207. Retrieved from https://arxiv.org/abs/2110.08207 · arXiv
  8. H. Touvron L. Martin K. Stone P. Albert A. Almahairi Y. Babaei N. Bashlykov S. Batra P. Bhargava S. Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https://arxiv.org/abs/2307.09288 · arXiv
  9. G. Izacard P. Lewis M. Lomeli L. Hosseini F. Petroni T. Schick J. Dwivedi-Yu A. Joulin S. Riedel and E. Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv:2208.03299. Retrieved from https://arxiv.org/abs/2208.03299 · arXiv
  10. A. Parisi Y. Zhao and N. Fiedel. 2022. TALM: Tool augmented language models. arXiv:2205.12255. Retrieved from https://arxiv.org/abs/2205.12255 · arXiv
  11. Q. Ye H. Xu G. Xu J. Ye M. Yan Y. Zhou J. Wang A. Hu P. Shi Y. Shi et al. 2023. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv:2304.14178. Retrieved from https://arxiv.org/abs/2304.14178 · arXiv
  12. W. Wang Z. Chen X. Chen J. Wu X. Zhu G. Zeng P. Luo T. Lu J. Zhou Y. Qiao et al. 2023. VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks. arXiv:2305.11175. Retrieved from https://arxiv.org/abs/2305.11175 · arXiv
  13. R. Yang L. Song Y. Li S. Zhao Y. Ge X. Li and Y. Shan. 2023. GPT4Tools: Teaching large language model to use tools via self-instruction. arXiv:2305.18752. Retrieved from https://arxiv.org/abs/2305.18752 · arXiv
  14. A. Zeng X. Liu Z. Du Z. Wang H. Lai M. Ding Z. Yang Y. Xu W. Zheng X. Xia et al. 2022. GLM-130B: An open bilingual pre-trained model. arXiv:2210.02414. Retrieved from https://arxiv.org/abs/2210.02414 · arXiv
  15. S. Wang Y. Sun Y. Xiang Z. Wu S. Ding W. Gong S. Feng J. Shang Y. Zhao C. Pang et al. 2021. ERNIE 3.0 Titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation. arXiv:2112.12731. Retrieved from https://arxiv.org/abs/2112.12731 · arXiv
  16. 10.1109/SC41405.2020.00024 · DOI
  17. J. He C. Zhou X. Ma T. Berg-Kirkpatrick and G. Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv:2110.04366. Retrieved from https://arxiv.org/abs/2110.04366 · arXiv
  18. X. Ma G. Fang and X. Wang. 2023. LLM-Pruner: On the structural pruning of large language models. arXiv:2305.11627. Retrieved from https://arxiv.org/abs/2305.11627 · arXiv
  19. 10.1609/aaai.v36i10.21408 · DOI
  20. G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. 2023. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the International Conference on Machine Learning, Vol. 202, PMLR, 38087–38099.
  21. A. Pal D. Karkhanis M. Roberts S. Dooley A. Sundararajan and S. Naidu. 2023. Giraffe: Adventures in expanding context lengths in LLMS. arXiv:2308.10882. Retrieved from https://arxiv.org/abs/2308.10882 · arXiv
  22. B. Peng J. Quesnelle H. Fan and E. Shippole. 2023. YaRN: Efficient context window extension of large language models. arXiv:2309.00071. Retrieved from https://arxiv.org/abs/2309.00071 · arXiv
  23. W. X. Zhao K. Zhou J. Li T. Tang X. Wang Y. Hou Y. Min B. Zhang J. Zhang Z. Dong et al. 2023. A survey of large language models. arXiv:2303.18223. Retrieved from https://arxiv.org/abs/2303.18223 · arXiv
  24. Q. Dong L. Li D. Dai C. Zheng Z. Wu B. Chang X. Sun J. Xu and Z. Sui. 2022. A survey for in-context learning. arXiv:2301.00234. Retrieved from https://arxiv.org/abs/2301.00234 · arXiv
  25. J. Huang and K. C.-C. Chang. 2022. Towards reasoning in large language models: A survey. arXiv:2212.10403. Retrieved from https://arxiv.org/abs/2212.10403 · arXiv
  26. Y. Wang W. Zhong L. Li F. Mi X. Zeng W. Huang L. Shang X. Jiang and Q. Liu. 2023. Aligning large language models with human: A survey. arXiv:2307.12966. Retrieved from https://arxiv.org/abs/2307.12966 · arXiv
  27. S. Yin C. Fu S. Zhao K. Li X. Sun T. Xu and E. Chen. 2023. A survey on multimodal large language models. arXiv:2306.13549. Retrieved from https://arxiv.org/abs/2306.13549 · arXiv
  28. 10.3115/992424.992434 · DOI
  29. 10.18653/v1/P16-1162 · DOI · Paper page
  30. 10.1109/ICASSP.2012.6289079 · DOI · Paper page
  31. S. J. Mielke Z. Alyafeai E. Salesky C. Raffel M. Dey M. Gallé A. Raja C. Si W. Y. Lee B. Sagot et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arXiv:2112.10508. Retrieved from https://arxiv.org/abs/2112.10508 · arXiv
  32. O. Press, N. Smith, and M. Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In Proceedings of the International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=R8sQPpGCv0
  33. R. Child S. Gray A. Radford and I. Sutskever. 2019. Generating long sequences with sparse transformers. arXiv:1904.10509. Retrieved from https://arxiv.org/abs/1904.10509 · arXiv
  34. 10.5555/3104322.3104425 · DOI
  35. D. Hendrycks and K. Gimpel. 2016. Gaussian error linear units (GELUs). arXiv:1606.08415. Retrieved from https://arxiv.org/abs/1606.08415 · arXiv
  36. 10.5555/2627435.2670313 · DOI
  37. D. Krueger T. Maharaj J. Kramár M. Pezeshki N. Ballas N. R. Ke A. Goyal Y. Bengio A. Courville and C. Pal. 2016. Zoneout: Regularizing RNNS by randomly preserving hidden activations. arXiv:1606.01305. Retrieved from https://arxiv.org/abs/1606.01305 · arXiv
  38. N. Shazeer. 2020. GLU variants improve transformer. arXiv:2002.05202. Retrieved from https://arxiv.org/abs/2002.05202 · arXiv
  39. Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the International Conference on Machine Learning. PMLR, 933–941.
  40. J. L. Ba J. R. Kiros and G. E. Hinton. 2016. Layer normalization. arXiv:1607.06450. Retrieved from https://arxiv.org/abs/1607.06450 · arXiv
  41. A. Baevski and M. Auli. 2018. Adaptive input representations for neural language modeling. arXiv:1809.10853. Retrieved from https://arxiv.org/abs/1809.10853 · arXiv
  42. H. Wang S. Ma L. Dong S. Huang D. Zhang and F. Wei. 2022. DeepNet: Scaling transformers to 1 000 layers. arXiv:2203.00555. Retrieved from https://arxiv.org/abs/2203.00555 · arXiv
  43. M. Shoeybi M. Patwary R. Puri P. LeGresley J. Casper and B. Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053. Retrieved from https://arxiv.org/abs/1909.08053 · arXiv
  44. S. Li J. Fang Z. Bian H. Liu Y. Liu H. Huang B. Wang and Y. You. 2021. Colossal-AI: A unified deep learning system for large-scale parallel training. arXiv:2110.14883. Retrieved from https://arxiv.org/abs/2110.14883 · arXiv
  45. J. He J. Qiu A. Zeng Z. Yang J. Zhai and J. Tang. 2021. FastMoE: A fast mixture-of-expert training system. arXiv:2103.13262. Retrieved from https://arxiv.org/abs/2103.13262 · arXiv
  46. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the Operating Systems Design and Implementation, Vol. 16, 265–283.
  47. T. Chen M. Li Y. Li M. Lin N. Wang M. Wang T. Xiao B. Xu C. Zhang and Z. Zhang. 2015. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv:1512.01274. Retrieved from https://arxiv.org/abs/1512.01274 · arXiv
  48. 10.5555/3586589.3586709 · DOI
  49. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al. 2022. GLaM: Efficient scaling of language models with mixture-of-experts. In Proceedings of the International Conference on Machine Learning. PMLR, 5547–5569.
  50. X. Ren P. Zhou X. Meng X. Huang Y. Wang W. Wang P. Li X. Zhang A. Podolskiy G. Arshinov et al. 2023. Pangu- \(\sum\) : Towards trillion parameter language model with sparse heterogeneous computing. arXiv:2303.10845. Retrieved from https://arxiv.org/abs/2303.10845 · arXiv
  51. T. Wang, A. Roberts, D. Hesslow, T. Le Scao, H. W. Chung, I. Beltagy, J. Launay, and C. Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization? In Proceedings of the International Conference on Machine Learning. PMLR, 22964–22984.
  52. J. Hoffmann S. Borgeaud A. Mensch E. Buchatskaya T. Cai E. Rutherford D. D. L. Casas L. A. Hendricks J. Welbl A. Clark et al. 2022. Training compute-optimal large language models. arXiv:2203.15556. Retrieved from https://arxiv.org/abs/2203.15556 · DOI · Paper page
  53. S. Iyer X. V. Lin R. Pasunuru T. Mihaylov D. Simig P. Yu K. Shuster T. Wang Q. Liu P. S. Koura et al. 2022. OPT-IML: Scaling language model instruction meta learning through the lens of generalization. arXiv:2212.12017. Retrieved from https://arxiv.org/abs/2212.12017 · arXiv
  54. Z. Sun Y. Shen Q. Zhou H. Zhang Z. Chen D. Cox Y. Yang and C. Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv:2305.03047. Retrieved from https://arxiv.org/abs/2305.03047 · arXiv
  55. A. Askell Y. Bai A. Chen D. Drain D. Ganguli T. Henighan A. Jones N. Joseph B. Mann N. DasSarma et al. 2021. A general language assistant as a laboratory for alignment. arXiv:2112.00861. Retrieved from https://arxiv.org/abs/2112.00861 · arXiv
  56. D. M. Ziegler N. Stiennon J. Wu T. B. Brown A. Radford D. Amodei P. Christiano and G. Irving. 2019. Fine-tuning language models from human preferences. arXiv:1909.08593. Retrieved from https://arxiv.org/abs/1909.08593 · arXiv
  57. Q. Liu F. Zhou Z. Jiang L. Dou and M. Lin. 2023. From zero to hero: Examining the power of symbolic tasks in instruction tuning. arXiv:2304.07995. Retrieved from https://arxiv.org/abs/2304.07995 · arXiv
  58. X. Wang J. Wei D. Schuurmans Q. Le E. Chi S. Narang A. Chowdhery and D. Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv:2203.11171. Retrieved from https://arxiv.org/abs/2203.11171 · arXiv
  59. S. Yao D. Yu J. Zhao I. Shafran T. L. Griffiths Y. Cao and K. Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv:2305.10601. Retrieved from https://arxiv.org/abs/2305.10601 · arXiv
  60. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning. PMLR, 2790–2799.
  61. S. McCandlish J. Kaplan D. Amodei and OpenAI Dota Team. 2018. An empirical model of large-batch training. arXiv:1812.06162. Retrieved from https://arxiv.org/abs/1812.06162 · arXiv
  62. OpenAI S. Adler S. Agarwal L. Ahmad I. Akkaya F. L. Alemen D. Almeida J. Altenschmidt S. Altman S. Anadkat et al. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
  63. A. Hurst A. Lerer A. P. Goucher A. Perelman A. Ramesh A. Clark A. Ostrow A. Welihinda A. Hayes A. Radford et al. 2024. GPT-4o system card. arXiv:2410.21276. Retrieved from https://arxiv.org/abs/2410.21276 · arXiv
  64. W. Zeng X. Ren T. Su H. Wang Y. Liao Z. Wang X. Jiang Z. Yang K. Wang X. Zhang et al. 2021. PanGu- \(\alpha\) : Large-scale autoregressive pretrained Chinese language models with auto-parallel computation. arXiv:2104.12369. Retrieved from https://arxiv.org/abs/2104.12369 · arXiv
  65. Y. Sun S. Wang S. Feng S. Ding C. Pang J. Shang J. Liu X. Chen Y. Zhao Y. Lu et al. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv:2107.02137. Retrieved from https://arxiv.org/abs/2107.02137 · arXiv
  66. O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. 2021. Jurassic-1: Technical Details and Evaluation. White Paper. AI21 Labs.
  67. S. Wu X. Zhao T. Yu R. Zhang C. Shen H. Liu F. Li H. Zhu J. Luo L. Xu et al. 2021. Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning. arXiv:2110.04725. Retrieved from https://arxiv.org/abs/2110.04725 · arXiv
  68. J. W. Rae S. Borgeaud T. Cai K. Millican J. Hoffmann F. Song J. Aslanides S. Henderson R. Ring S. Young et al. 2021. Scaling language models: Methods analysis & insights from training gopher. arXiv:2112.11446. Retrieved from https://arxiv.org/abs/2112.11446 · arXiv
  69. S. Smith M. Patwary B. Norick P. LeGresley S. Rajbhandari J. Casper Z. Liu S. Prabhumoye G. Zerveas V. Korthikanti et al. 2022. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B a large-scale generative language model. arXiv:2201.11990. Retrieved from https://arxiv.org/abs/2201.11990 · arXiv
  70. P. Micikevicius S. Narang J. Alben G. Diamos E. Elsen D. Garcia B. Ginsburg M. Houston O. Kuchaiev G. Venkatesh et al. 2017. Mixed precision training. arXiv:1710.03740. Retrieved from https://arxiv.org/abs/1710.03740 · arXiv
  71. N. Shazeer A. Mirhoseini K. Maziarz A. Davis Q. Le G. Hinton and J. Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv:1701.06538. Retrieved from https://arxiv.org/abs/1701.06538 · arXiv
  72. S. Soltan S. Ananthakrishnan J. FitzGerald R. Gupta W. Hamza H. Khan C. Peris S. Rawls A. Rosenbaum A. Rumshisky et al. 2022. AlexaTM 20B: Few-shot learning using a large-scale multilingual Seq2Seq model. arXiv:2208.01448. Retrieved from https://arxiv.org/abs/2208.01448 · arXiv
  73. R. Anil A. M. Dai O. Firat M. Johnson D. Lepikhin A. Passos S. Shakeri E. Taropa P. Bailey Z. Chen et al. 2023. PaLM 2 technical report. arXiv:2305.10403. Retrieved from https://arxiv.org/abs/2305.10403 · arXiv
  74. Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al. 2022. UL2: Unifying language learning paradigms. In Proceedings of the 11th International Conference on Learning Representations.
  75. M. N. Rabe and C. Staats. 2021. Self-attention does not need o(n \({}^{2}\) ) memory. arXiv:2112.05682. Retrieved from https://arxiv.org/abs/2112.05682 · arXiv
  76. A. Dubey A. Jauhri A. Pandey A. Kadian A. Al-Dahle A. Letman A. Mathur A. Schelten A. Yang A. Fan et al. 2024. The Llama 3 Herd of models. arXiv:2407.21783. Retrieved from https://arxiv.org/abs/2407.21783 · arXiv
  77. G. Team R. Anil S. Borgeaud Y. Wu J.-B. Alayrac J. Yu R. Soricut J. Schalkwyk A. M. Dai A. Hauth et al. 2023. Gemini: A family of highly capable multimodal models arXiv:2312.11805. Retrieved from https://arxiv.org/abs/2312.11805 · arXiv
  78. M. Reid N. Savinov D. Teplyashin D. Lepikhin T. Lillicrap J.-B. Alayrac R. Soricut A. Lazaridou O. Firat J. Schrittwieser et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530. Retrieved from https://arxiv.org/abs/2403.05530 · arXiv
  79. B. Adler N. Agarwal A. Aithal D. H. Anh P. Bhattacharya A. Brundyn J. Casper B. Catanzaro S. Clay J. Cohen et al. 2024. Nemotron-4 340B technical report. arXiv:2406.11704. Retrieved from https://arxiv.org/abs/2406.11704 · arXiv
  80. X. Bi D. Chen G. Chen S. Chen D. Dai C. Deng H. Ding K. Dong Q. Du Z. Fu et al. 2024. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954. Retrieved from https://arxiv.org/abs/2401.02954 · arXiv
  81. A. DeepSeek-AI B. Liu B. Feng B. Wang B. Wang B. Liu C. Zhao C. Deng C. Ruan D. Dai et al. 2024. DeepSeek-V2: A strong economical and efficient mixture-of-experts language model. arXiv:2405.04434. Retrieved from https://arxiv.org/abs/2405.04434 · arXiv
  82. A. Liu B. Feng B. Xue B. Wang B. Wu C. Lu C. Zhao C. Deng C. Zhang C. Ruan et al. 2024. DeepSeek-V3 technical report. arXiv:2412.19437. Retrieved from https://arxiv.org/abs/2412.19437 · arXiv
  83. E. Nijkamp B. Pang H. Hayashi L. Tu H. Wang Y. Zhou S. Savarese C. Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. Retrieved from https://arxiv.org/abs/2203.13474 · arXiv
  84. M. Chen J. Tworek H. Jun Q. Yuan H. P. O. Pinto J. Kaplan H. Edwards Y. Burda N. Joseph G. Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https://arxiv.org/abs/2107.03374 · arXiv
  85. N. Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv:1911.02150. Retrieved from https://arxiv.org/abs/1911.02150 · arXiv
  86. R. Y. Pang and H. He. 2020. Text generation by learning from demonstrations. arXiv:2009.07839. Retrieved from https://arxiv.org/abs/2009.07839 · arXiv
  87. R. Dabre and A. Fujita. 2020. Softmax tempering for training neural machine translation models. arXiv:2009.09372. Retrieved from https://arxiv.org/abs/2009.09372 · arXiv
  88. R. Li L. B. Allal Y. Zi N. Muennighoff D. Kocetkov C. Mou M. Marone C. Akiki J. Li J. Chim et al. 2023. StarCoder: May the source be with you! arXiv:2305.06161. Retrieved from https://arxiv.org/abs/2305.06161 · arXiv
  89. R. Taylor M. Kardas G. Cucurull T. Scialom A. Hartshorn E. Saravia A. Poulton V. Kerkez and R. Stojnic. 2022. Galactica: A large language model for science. arXiv:2211.09085. Retrieved from https://arxiv.org/abs/2211.09085 · arXiv
  90. R. Thoppilan D. De Freitas J. Hall N. Shazeer A. Kulshreshtha H.-T. Cheng A. Jin T. Bos L. Baker Y. Du et al. 2022. LaMDA: Language models for dialog applications. arXiv:2201.08239. Retrieved from https://arxiv.org/abs/2201.08239 · arXiv
  91. S. Wu O. Irsoy S. Lu V. Dabravolski M. Dredze S. Gehrmann P. Kambadur D. Rosenberg and G. Mann. 2023. BloombergGPT: A large language model for finance. arXiv:2303.17564. Retrieved from https://arxiv.org/abs/2303.17564 · arXiv
  92. X. Zhang, Q. Yang, and D. Xu. 2023. XuanYuan 2.0: A large Chinese financial chat model with hundreds of billions parameters. arXiv:2305.12002. Retrieved from https://arxiv.org/abs/2305.12002 · arXiv
  93. P. Gao J. Han R. Zhang Z. Lin S. Geng A. Zhou W. Zhang P. Lu C. He X. Yue et al. 2023. LLaMA-Adapter V2: Parameter-efficient visual instruction model. arXiv:2304.15010. Retrieved from https://arxiv.org/abs/2304.15010 · arXiv
  94. B. Peng C. Li P. He M. Galley and J. Gao. 2023. Instruction tuning with GPT-4. arXiv:2304.03277. Retrieved from https://arxiv.org/abs/2304.03277 · arXiv
  95. T. Liu and B. K. H. Low. 2023. Goat: Fine-tuned llama outperforms GPT-4 on arithmetic tasks. arXiv:2305.14201. Retrieved from https://arxiv.org/abs/2305.14201 · arXiv
  96. C. Xu Q. Sun K. Zheng X. Geng P. Zhao J. Feng C. Tao and D. Jiang. 2023. WizardLM: Empowering large language models to follow complex instructions. arXiv:2304.12244. Retrieved from https://arxiv.org/abs/2304.12244 · arXiv
  97. Z. Luo C. Xu P. Zhao Q. Sun X. Geng W. Hu C. Tao J. Ma Q. Lin and D. Jiang. 2023. WizardCoder: Empowering code large language models with Evol-Instruct. arXiv:2306.08568. Retrieved from https://arxiv.org/abs/2306.08568 · arXiv
  98. J. Menick M. Trebacz V. Mikulik J. Aslanides F. Song M. Chadwick M. Glaese S. Young L. Campbell-Gillingham G. Irving et al. 2022. Teaching language models to support answers with verified quotes. arXiv:2203.11147. Retrieved from https://arxiv.org/abs/2203.11147 · arXiv
  99. R. Nakano J. Hilton S. Balaji J. Wu L. Ouyang C. Kim C. Hesse S. Jain V. Kosaraju W. Saunders et al. 2021. WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332. Retrieved from https://arxiv.org/abs/2112.09332 · arXiv
  100. A. Glaese N. McAleese M. Trębacz J. Aslanides V. Firoiu T. Ewalds M. Rauh L. Weidinger M. Chadwick P. Thacker et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv:2209.14375. Retrieved from https://arxiv.org/abs/2209.14375 · arXiv
  101. R. Rafailov A. Sharma E. Mitchell S. Ermon C. D. Manning and C. Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv:2305.18290. Retrieved from https://arxiv.org/abs/2305.18290 · arXiv
  102. H. Dong W. Xiong D. Goyal R. Pan S. Diao J. Zhang K. Shum and T. Zhang. 2023. RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv:2304.06767. Retrieved from https://arxiv.org/abs/2304.06767 · arXiv
  103. Z. Yuan H. Yuan C. Tan W. Wang S. Huang and F. Huang. 2023. RRHF: Rank responses to align language models with human feedback without tears. arXiv:2304.05302. Retrieved from https://arxiv.org/abs/2304.05302 · arXiv
  104. F. Song B. Yu M. Li H. Yu F. Huang Y. Li and H. Wang. 2023. Preference ranking optimization for human alignment. arXiv:2306.17492. Retrieved from https://arxiv.org/abs/2306.17492 · arXiv
  105. H. Liu C. Sferrazza and P. Abbeel. 2023. Languages are rewards: Hindsight finetuning using human feedback. arXiv:2302.02676. Retrieved from https://arxiv.org/abs/2302.02676 · arXiv
  106. Y. Bai S. Kadavath S. Kundu A. Askell J. Kernion A. Jones A. Chen A. Goldie A. Mirhoseini C. McKinnon et al. 2022. Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073. Retrieved from https://arxiv.org/abs/2212.08073 · arXiv
  107. Y. Dubois X. Li R. Taori T. Zhang I. Gulrajani J. Ba C. Guestrin P. Liang and T. B. Hashimoto. 2023. AlpacaFarm: A simulation framework for methods that learn from human feedback. arXiv:2305.14387. Retrieved from https://arxiv.org/abs/2305.14387 · arXiv
  108. C. Si Z. Gan Z. Yang S. Wang J. Wang J. Boyd-Graber and L. Wang. 2022. Prompting GPT-3 to be reliable. arXiv:2210.09150. Retrieved from https://arxiv.org/abs/2210.09150 · arXiv
  109. D. Ganguli A. Askell N. Schiefer T. Liao K. Lukošiūtė A. Chen A. Goldie A. Mirhoseini C. Olsson D. Hernandez et al. 2023. The capacity for moral self-correction in large language models. arXiv:2302.07459. Retrieved from https://arxiv.org/abs/2302.07459 · arXiv
  110. D. Ganguli L. Lovitt J. Kernion A. Askell Y. Bai S. Kadavath B. Mann E. Perez N. Schiefer K. Ndousse et al. 2022. Red teaming language models to reduce harms: Methods scaling behaviors and lessons learned. arXiv:2209.07858. Retrieved from https://arxiv.org/abs/2209.07858 · arXiv
  111. S. Casper J. Lin J. Kwon G. Culp and D. Hadfield-Menell. 2023. Explore establish exploit: Red teaming language models from scratch. arXiv:2306.09442. Retrieved from https://arxiv.org/abs/2306.09442 · arXiv
  112. Z. Shi and A. Lipani. 2023. Don’t stop pretraining? Make prompt-based fine-tuning powerful learner. arXiv:2305.01711. Retrieved from https://arxiv.org/abs/2305.01711 · arXiv
  113. H. Gupta S. A. Sawant S. Mishra M. Nakamura A. Mitra S. Mashetty and C. Baral. 2023. Instruction tuned models are quick learners. arXiv:2306.05539. Retrieved from https://arxiv.org/abs/2306.05539 · arXiv
  114. H. Chen Y. Zhang Q. Zhang H. Yang X. Hu X. Ma Y. Yanggong and J. Zhao. 2023. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning. arXiv:2305.09246. Retrieved from https://arxiv.org/abs/2305.09246 · arXiv
  115. C. Zhou P. Liu P. Xu S. Iyer J. Sun Y. Mao X. Ma A. Efrat P. Yu L. Yu et al. 2023. LIMA: Less is more for alignment. arXiv:2305.11206. Retrieved from https://arxiv.org/abs/2305.11206 · arXiv
  116. C. Han Q. Wang W. Xiong Y. Chen H. Ji and S. Wang. 2023. LM-infinite: Simple on-the-fly length generalization for large language models. arXiv:2308.16137. Retrieved from https://arxiv.org/abs/2308.16137 · arXiv
  117. Y. Chen S. Qian H. Tang X. Lai Z. Liu S. Han and J. Jia. 2023. LongLoRA: Efficient fine-tuning of long-context large language models. arXiv:2309.12307. Retrieved from https://arxiv.org/abs/2309.12307 · arXiv
  118. W. Wang L. Dong H. Cheng X. Liu X. Yan J. Gao and F. Wei. 2023. Augmenting language models with long-term memory. arXiv:2306.07174. Retrieved from https://arxiv.org/abs/2306.07174 · arXiv
  119. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of the International Conference on Machine Learning. PMLR, 2206–2240.
  120. W. Zhong L. Guo Q. Gao and Y. Wang. 2023. MemoryBank: Enhancing large language models with long-term memory. arXiv:2305.10250. Retrieved from https://arxiv.org/abs/2305.10250 · arXiv
  121. N. Shinn F. Cassano B. Labash A. Gopinath K. Narasimhan and S. Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv:2303.11366. Retrieved from https://arxiv.org/abs/2303.11366 · DOI · Paper page
  122. C. Hu J. Fu C. Du S. Luo J. Zhao and H. Zhao. 2023. ChatDB: Augmenting LLMs with databases as their symbolic memory. arXiv:2306.03901. Retrieved from https://arxiv.org/abs/2306.03901 · arXiv
  123. X. Li and X. Qiu. 2023. MoT: Pre-thinking and recalling enable ChatGPT to self-improve with memory-of-thoughts. arXiv:2305.05181. Retrieved from https://arxiv.org/abs/2305.05181 · arXiv
  124. D. Schuurmans. 2023. Memory augmented large language models are computationally universal. arXiv:2301.04589. Retrieved from https://arxiv.org/abs/2301.04589 · arXiv
  125. A. Modarressi A. Imani M. Fayyaz and H. Schütze. 2023. RET-LLM: Towards a general read-write memory for large language models. arXiv:2305.14322. Retrieved from https://arxiv.org/abs/2305.14322 · arXiv
  126. X. Wang J. Wei D. Schuurmans Q. Le E. Chi and D. Zhou. 2022. Rationale-augmented ensembles in language models. arXiv:2207.00747. Retrieved from https://arxiv.org/abs/2207.00747 · arXiv
  127. L. Wang N. Yang and F. Wei. 2023. Learning to retrieve in-context examples for large language models. arXiv:2307.07164. Retrieved from https://arxiv.org/abs/2307.07164 · arXiv
  128. O. Rubin and J. Berant. 2023. Long-range language modeling with self-retrieval. arXiv:2306.13421. Retrieved from https://arxiv.org/abs/2306.13421 · arXiv
  129. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the International Conference on Machine Learning. PMLR, 3929–3938.
  130. M. Komeili K. Shuster and J. Weston. 2021. Internet-augmented dialogue generation. arXiv:2107.07566. Retrieved from https://arxiv.org/abs/2107.07566 · arXiv
  131. A. Lazaridou E. Gribovskaya W. Stokowiec and N. Grigorev. 2022. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv:2203.05115. Retrieved from https://arxiv.org/abs/2203.05115 · arXiv
  132. D. Gao L. Ji L. Zhou K. Q. Lin J. Chen Z. Fan and M. Z. Shou. 2023. AssistGPT: A general multi-modal assistant that can plan execute inspect and learn. arXiv:2306.08640. Retrieved from https://arxiv.org/abs/2306.08640 · arXiv
  133. P. Lu B. Peng H. Cheng M. Galley K.-W. Chang Y. N. Wu S.-C. Zhu and J. Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv:2304.09842. Retrieved from https://arxiv.org/abs/2304.09842 · arXiv
  134. B. Paranjape S. Lundberg S. Singh H. Hajishirzi L. Zettlemoyer and M. T. Ribeiro. 2023. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv:2303.09014. Retrieved from https://arxiv.org/abs/2303.09014 · arXiv
  135. C.-Y. Hsieh S.-A. Chen C.-L. Li Y. Fujii A. Ratner C.-Y. Lee R. Krishna and T. Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models. arXiv:2308.00675. Retrieved from https://arxiv.org/abs/2308.00675 · arXiv
  136. Y. Song W. Xiong D. Zhu C. Li K. Wang Y. Tian and S. Li. 2023. RestGPT: Connecting large language models with real-world applications via RESTful APIs. arXiv:2306.06624. Retrieved from https://arxiv.org/abs/2306.06624 · arXiv
  137. S. Hao T. Liu Z. Wang and Z. Hu. 2023. ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings. arXiv:2305.11554. Retrieved from https://arxiv.org/abs/2305.11554 · DOI
  138. S. G. Patil T. Zhang X. Wang and J. E. Gonzalez. 2023. Gorilla: Large language model connected with massive APIs. arXiv:2305.15334. Retrieved from https://arxiv.org/abs/2305.15334 · arXiv
  139. Q. Xu F. Hong B. Li C. Hu Z. Chen and J. Zhang. 2023. On the tool manipulation capability of open-source large language models. arXiv:2305.16504. Retrieved from https://arxiv.org/abs/2305.16504 · arXiv
  140. Y. Qin S. Liang Y. Ye K. Zhu L. Yan Y. Lu Y. Lin X. Cong X. Tang B. Qian et al. 2023. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. arXiv:2307.16789. Retrieved from https://arxiv.org/abs/2307.16789 · arXiv
  141. Y. Shen K. Song X. Tan D. Li W. Lu and Y. Zhuang. 2023. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. arXiv:2303.17580. Retrieved from https://arxiv.org/abs/2303.17580 · arXiv
  142. S. Hong X. Zheng J. Chen Y. Cheng J. Wang C. Zhang Z. Wang S. K. S. Yau Z. Lin L. Zhou et al. 2023. MetaGPT: Meta programming for multi-agent collaborative framework. arXiv:2308.00352. Retrieved from https://arxiv.org/abs/2308.00352 · arXiv
  143. L. Wang C. Ma X. Feng Z. Zhang H. Yang J. Zhang Z. Chen J. Tang X. Chen Y. Lin et al. 2023. A survey on large language model based autonomous agents. arXiv:2308.11432. Retrieved from https://arxiv.org/abs/2308.11432 · arXiv
  144. W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the International Conference on Machine Learning. PMLR, 9118–9147.
  145. W. Yao S. Heinecke J. C. Niebles Z. Liu Y. Feng L. Xue R. Murthy Z. Chen J. Zhang D. Arpit et al. 2023. Retroformer: Retrospective large language agents with policy gradient optimization. arXiv:2308.02151. Retrieved from https://arxiv.org/abs/2308.02151 · arXiv
  146. C. Jin W. Tan J. Yang B. Liu R. Song L. Wang and J. Fu. 2023. AlphaBlock: Embodied finetuning for vision-language reasoning in robot manipulation. arXiv:2305.18898. Retrieved from https://arxiv.org/abs/2305.18898 · arXiv
  147. W. Yu N. Gileadi C. Fu S. Kirmani K.-H. Lee M. G. Arenas H.-T. L. Chiang T. Erez L. Hasenclever J. Humplik et al. 2023. Language to rewards for robotic skill synthesis. arXiv:2306.08647. Retrieved from https://arxiv.org/abs/2306.08647 · arXiv
  148. A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. 2023. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning. PMLR, 287–318.
  149. H. Ha P. Florence and S. Song. 2023. Scaling up and distilling down: Language-guided robot skill acquisition. arXiv:2307.14535. Retrieved from https://arxiv.org/abs/2307.14535 · arXiv
  150. A. Rajvanshi K. Sikka X. Lin B. Lee H.-P. Chiu and A. Velasquez. 2023. SayNav: Grounding large language models for dynamic planning to navigation in new environments. arXiv:2309.04077. Retrieved from https://arxiv.org/abs/2309.04077 · arXiv
  151. V. S. Dorbala J. F. Mullen Jr and D. Manocha. 2023. Can an embodied agent find your “cat-shaped mug”? LLM-based zero-shot object navigation. arXiv:2303.03480. Retrieved from https://arxiv.org/abs/2303.03480 · arXiv
  152. 10.1109/ICRA48891.2023.10160969 · DOI · Paper page
  153. X. Liu Y. Zheng Z. Du M. Ding Y. Qian Z. Yang and J. Tang. 2021. GPT understands too. arXiv:2103.10385. Retrieved from https://arxiv.org/abs/2103.10385 · arXiv
  154. A. Razdaibiedina Y. Mao R. Hou M. Khabsa M. Lewis and A. Almahairi. 2023. Progressive prompts: Continual learning for language models. arXiv:2301.12314. Retrieved from https://arxiv.org/abs/2301.12314 · arXiv
  155. Z.-R. Zhang C. Tan H. Xu C. Wang J. Huang and S. Huang. 2023. Towards adaptive prefix tuning for parameter-efficient language model fine-tuning. arXiv:2305.15212. Retrieved from https://arxiv.org/abs/2305.15212 · arXiv
  156. T. Dettmers M. Lewis Y. Belkada and L. Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. arXiv:2208.07339. Retrieved from https://arxiv.org/abs/2208.07339 · arXiv
  157. C. Lee J. Jin T. Kim H. Kim and E. Park. 2023. OWQ: Lessons learned from activation outliers for weight quantization in large language models. arXiv:2306.02272. Retrieved from https://arxiv.org/abs/2306.02272 · arXiv
  158. T. Dettmers A. Pagnoni A. Holtzman and L. Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. arXiv:2305.14314. Retrieved from https://arxiv.org/abs/2305.14314 · DOI · Paper page
  159. 10.1109/CVPR.2017.430 · DOI · Paper page
  160. J. Kim J. H. Lee S. Kim J. Park K. M. Yoo S. J. Kwon and D. Lee. 2023. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv:2305.14152. Retrieved from https://arxiv.org/abs/2305.14152 · DOI
  161. M. Sun Z. Liu A. Bair and J. Z. Kolter. 2023. A simple and effective pruning approach for large language models. arXiv:2306.11695. Retrieved from https://arxiv.org/abs/2306.11695 · arXiv
  162. L. Yin Y. Wu Z. Zhang C.-Y. Hsieh Y. Wang Y. Jia M. Pechenizkiy Y. Liang Z. Wang and S. Liu. 2023. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. arXiv:2310.05175. Retrieved from https://arxiv.org/abs/2310.05175 · arXiv
  163. J. Li D. Li S. Savarese and S. Hoi. 2023. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597. Retrieved from https://arxiv.org/abs/2301.12597 · arXiv
  164. H. Liu C. Li Q. Wu and Y. J. Lee. 2023. Visual instruction tuning. arXiv:2304.08485. Retrieved from https://arxiv.org/abs/2304.08485 · arXiv
  165. K. Li Y. He Y. Wang Y. Li W. Wang P. Luo Y. Wang L. Wang and Y. Qiao. 2023. VideoChat: Chat-centric video understanding. arXiv:2305.06355. Retrieved from https://arxiv.org/abs/2305.06355 · arXiv
  166. M. Maaz H. Rasheed S. Khan and F. S. Khan. 2023. Video-ChatGPT: Towards detailed video understanding via large vision and language models. arXiv:2306.05424. Retrieved from https://arxiv.org/abs/2306.05424 · arXiv
  167. C. Lyu M. Wu L. Wang X. Huang B. Liu Z. Du S. Shi and Z. Tu. 2023. Macaw-LLM: Multi-modal language modeling with image audio video and text integration. arXiv:2306.09093. Retrieved from https://arxiv.org/abs/2306.09093 · arXiv
  168. D. Zhu J. Chen X. Shen X. Li and M. Elhoseiny. 2023. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592. Retrieved from https://arxiv.org/abs/2304.10592 · arXiv
  169. A. Dosovitskiy L. Beyer A. Kolesnikov D. Weissenborn X. Zhai T. Unterthiner M. Dehghani M. Minderer G. Heigold S. Gelly et al. 2020. An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv:2010.11929. Retrieved from https://arxiv.org/abs/2010.11929 · arXiv
  170. W. Dai J. Li D. Li A. M. H. Tiong J. Zhao W. Wang B. Li P. Fung and S. Hoi. 2023. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500. Retrieved from https://arxiv.org/abs/2305.06500 · DOI
  171. Z. Zhao L. Guo T. Yue S. Chen S. Shao X. Zhu Z. Yuan and J. Liu. 2023. ChatBridge: Bridging modalities with large language model as a language catalyst. arXiv:2305.16103. Retrieved from https://arxiv.org/abs/2305.16103 · arXiv
  172. L. Li Y. Yin S. Li L. Chen P. Wang S. Ren M. Li Y. Yang J. Xu X. Sun et al. 2023. M3IT: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv:2306.04387. Retrieved from https://arxiv.org/abs/2306.04387 · arXiv
  173. G. Luo Y. Zhou T. Ren S. Chen X. Sun and R. Ji. 2023. Cheap and quick: Efficient vision-language instruction tuning for large language models. arXiv:2305.15023. Retrieved from https://arxiv.org/abs/2305.15023 · arXiv
  174. R. Zhang J. Han A. Zhou X. Hu S. Yan P. Lu H. Li P. Gao and Y. Qiao. 2023. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention. arXiv:2303.16199. Retrieved from https://arxiv.org/abs/2303.16199 · arXiv
  175. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the International Conference on Machine Learning. PMLR, 28492–28518.
  176. Z. Zhang A. Zhang M. Li H. Zhao G. Karypis and A. Smola. 2023. Multimodal chain-of-thought reasoning in language models. arXiv:2302.00923. Retrieved from https://arxiv.org/abs/2302.00923 · arXiv
  177. J. Ge H. Luo S. Qian Y. Gan J. Fu and S. Zhan. 2023. Chain of thought prompt tuning in vision language models. arXiv:2304.07919. Retrieved from https://arxiv.org/abs/2304.07919 · arXiv
  178. C. Wu S. Yin W. Qi X. Wang Z. Tang and N. Duan. 2023. Visual ChatGPT: Talking drawing and editing with visual foundation models. arXiv:2303.04671. Retrieved from https://arxiv.org/abs/2303.04671 · arXiv
  179. Z. Yang L. Li J. Wang K. Lin E. Azarnasab F. Ahmed Z. Liu C. Liu M. Zeng and L. Wang. 2023. MM-REACT: Prompting ChatGPT for multimodal reasoning and action. arXiv:2303.11381. Retrieved from https://arxiv.org/abs/2303.11381 · arXiv
  180. T. Wang J. Zhang J. Fei Y. Ge H. Zheng Y. Tang Z. Li M. Gao S. Zhao Y. Shan et al. 2023. Caption anything: Interactive image description with diverse multimodal controls. arXiv:2305.02677. Retrieved from https://arxiv.org/abs/2305.02677 · arXiv
  181. T. Q. Nguyen and J. Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. arXiv:1910.05895. Retrieved from https://arxiv.org/abs/1910.05895 · arXiv
  182. Y. Liu M. Ott N. Goyal J. Du M. Joshi D. Chen O. Levy M. Lewis L. Zettlemoyer and V. Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv:1907.11692. Retrieved from https://arxiv.org/abs/1907.11692 · arXiv
  183. L. Gao S. Biderman S. Black L. Golding T. Hoppe C. Foster J. Phang H. He A. Thite N. Nabeshima et al. 2020. The pile: An 800GB dataset of diverse text for language modeling. arXiv:2101.00027. Retrieved from https://arxiv.org/abs/2101.00027 · arXiv
  184. Y. Bai A. Jones K. Ndousse A. Askell A. Chen N. DasSarma D. Drain S. Fort D. Ganguli T. Henighan et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. Retrieved from https://arxiv.org/abs/2204.05862 · arXiv
  185. D. Hendrycks C. Burns S. Basart A. Zou M. Mazeika D. Song and J. Steinhardt. 2020. Measuring massive multitask language understanding. arXiv:2009.03300. Retrieved from https://arxiv.org/abs/2009.03300 · arXiv
  186. A. Srivastava A. Rastogi A. Rao A. A. M. Shoeb A. Abid A. Fisch A. R. Brown A. Santoro A. Gupta A. Garriga-Alonso et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. Retrieved from https://arxiv.org/abs/2206.04615 · arXiv
  187. Y. Yao Q. Dong J. Guan B. Cao Z. Zhang C. Xiao X. Wang F. Qi J. Bao J. Nie et al. 2021. CUGE: A Chinese language understanding and generation evaluation benchmark. arXiv:2112.13610. Retrieved from https://arxiv.org/abs/2112.13610 · arXiv
  188. L. Xu X. Lu C. Yuan X. Zhang H. Xu H. Yuan G. Wei X. Pan X. Tian L. Qin et al. 2021. FewCLUE: A Chinese few-shot learning evaluation benchmark. arXiv:2107.07498. Retrieved from https://arxiv.org/abs/2107.07498 · arXiv
  189. P. Liang R. Bommasani T. Lee D. Tsipras D. Soylu M. Yasunaga Y. Zhang D. Narayanan Y. Wu A. Kumar et al. 2022. Holistic evaluation of language models. arXiv:2211.09110. Retrieved from https://arxiv.org/abs/2211.09110 · arXiv
  190. S. Park J. Moon S. Kim W. I. Cho J. Han J. Park C. Song J. Kim Y. Song T. Oh et al. 2021. KLUE: Korean language understanding evaluation. arXiv:2105.09680. Retrieved from https://arxiv.org/abs/2105.09680 · arXiv
  191. M. T. Pilehvar and J. Camacho-Collados. 2018. WiC: 10 000 example pairs for evaluating context-sensitive representations. arXiv:1808.09121. Retrieved from https://arxiv.org/abs/1808.09121 · arXiv
  192. S. Merity C. Xiong J. Bradbury and R. Socher. 2016. Pointer sentinel mixture models. arXiv:1609.07843. Retrieved from https://arxiv.org/abs/1609.07843 · arXiv
  193. J. W. Rae A. Potapenko S. M. Jayakumar and T. P. Lillicrap. 2019. Compressive transformers for long-range sequence modelling. arXiv:1911.05507. Retrieved from https://arxiv.org/abs/1911.05507 · arXiv
  194. J. Xu J. Wen X. Sun and Q. Su. 2017. A discourse-level named entity recognition and relation extraction dataset for Chinese literature text. arXiv:1711.07010. Retrieved from https://arxiv.org/abs/1711.07010 · arXiv
  195. P. Li W. Li Z. He X. Wang Y. Cao J. Zhou and W. Xu. 2016. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. arXiv:1607.06275. Retrieved from https://arxiv.org/abs/1607.06275 · arXiv
  196. R. Weischedel, S. Pradhan, L. Ramshaw, M. Palmer, N. Xue, M. Marcus, A. Taylor, C. Greenberg, E. Hovy, R. Belvin, et al. 2011. Ontonotes Release 4.0, LDC2011T03. Linguistic Data Consortium, Philadelphia, PA.
  197. Z. Shao M. Huang J. Wen W. Xu and X. Zhu. 2019. Long and diverse text generation with planning-based hierarchical variational model. arXiv:1908.06605. Retrieved from https://arxiv.org/abs/1908.06605 · arXiv
  198. P. Clark I. Cowhey O. Etzioni T. Khot A. Sabharwal C. Schoenick and O. Tafjord. 2018. Think you have solved question answering? Try ARC the AI2 reasoning challenge. arXiv:1803.05457. Retrieved from https://arxiv.org/abs/1803.05457 · arXiv
  199. M. Roemmele, C. A. Bejan, and A. S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Proceedings of the AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 90–95.
  200. H. Levesque, E. Davis, and L. Morgenstern. 2012. The Winograd schema challenge. In Proceedings of the 13th International Conference on the Principles of Knowledge Representation and Reasoning.

200 of 257 shown

Cited by

Sorted by citations

See all 1,257

Related

See all

Metadata

Authors
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A.
Affiliations
  • The University of Sydney (AU)
  • University of Engineering and Technology Lahore (PK)
  • Chinese University of Hong Kong (HK)
  • University of Technology Sydney (AU)
  • King Fahd University of Petroleum and Minerals (SA)
  • Ontario Tech University (CA)
  • The University of Melbourne (AU)
  • Australian National University (AU)
  • The University of Western Australia (AU)
ORCID
Venue
ACM Transactions on Intelligent Systems and Technology
Publisher
Association for Computing Machinery (ACM)
Published
2025-08-18
Volume
16
Issue
5
Pages
1-72
ISSN
  • 2157-6904 (ISSN-L)
  • 2157-6912
DOI
10.1145/3744746
OpenAlex
W4411403346
Cited by
1,257
References
245
Citations by year
2026: 527 · 2025: 488
FWCI
407.12
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • ACM Transactions on Intelligent Systems and Technology (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • UWA Profiles and Research Repository (UWA) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
King Fahd University of Petroleum and Minerals
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, LLM architecture, context length, fine-tuning, multimodal large language models, LLM benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review