CoChat
Sign Up
PreprintCited by 6Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

MiniMax-01: Scaling Foundation Models with Lightning Attention

Open access
MiniMax, Li, Aonian, Gong, Bangwei, Yang, Bo, Shan, Boji, Liu, Chang, Zhu, Cheng, Zhang, Chunhao +82 more·2025·arXiv
DOI 10.48550/arxiv.2501.08313arXiv 2501.08313
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.08313

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most models mix linear attention with full attention at a fixed ratio, typically 3:1.

    “As a result, most models, such as MiniMax-M1 [94], Qwen3-Next [95], and Kimi Linear [92], mix linear attention with full attention at a fixed ratio, typically 3:1.”

    From A Survey of Large Language Models (2026) · cites this paper as [94]

    The Model Architecture section describes MiniMax-Text-01 as mixing linear and softmax attention at a fixed ratio, but specifies seven linear-attention blocks for every one softmax-attention block—not 3:1. The paper does not establish what ratio most models use. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 189

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024. · arXiv
  2. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024. · arXiv
  3. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895-4901, 2023.
  4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. · arXiv
  5. Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018. · arXiv
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. · arXiv
  7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022b. · arXiv
  8. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024. · arXiv
  9. Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. · arXiv
  10. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
  11. Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21-29. IEEE, 1997.
  12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
  13. Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. MEGA-Bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint arXiv:2410.10563, 2024a. · arXiv
  14. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  15. Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. MindSearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183, 2024b. · arXiv
  16. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. · arXiv
  17. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
  18. Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning (ICML), pages 4057-4086.
  19. PMLR, 2022.
  20. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, 2019.
  21. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. · arXiv
  22. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
  23. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=mZn2Xyh9Ec.
  24. Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. · arXiv
  25. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344-16359, 2022.
  26. Alexandre de Brébisson and Pascal Vincent. A cheap linear attention mechanism with fast lookups and fixed-size representations. arXiv preprint arXiv:1609.05866, 2016. · arXiv
  27. DeepSeek-AI. DeepSeek-V3 technical report, 2024. URL https://arxiv.org/abs/2412.19437. · arXiv
  28. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107-1128, 2024.
  29. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/ forum?id=YicbFdNTTy.
  30. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, 2019.
  31. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. · arXiv
  32. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
  33. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. · arXiv
  34. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b SSM hybrid model. arXiv preprint arXiv:2405.16712, 2024. MiniMax-01: Scaling Foundation Models with Lightning Attention Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010. · arXiv
  35. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id= tEYskw1VY2.
  36. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. · arXiv
  37. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. OlympiadBench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024a. · arXiv
  38. Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. Chinese simpleQA: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140, 2024b. · arXiv
  39. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  40. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL https://openreview.net/forum?id=7Bywt2mQsCe.
  41. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. · arXiv
  42. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022. · arXiv
  43. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 30016-30030, 2022.
  44. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. · arXiv
  45. Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning, pages 9099-9117. PMLR, 2022.
  46. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. · arXiv
  47. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. · arXiv
  48. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, 2017.
  49. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. · arXiv
  50. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
  51. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7: 453-466, 2019.
  52. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424-8445, 2022.
  53. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional compu- tation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb.
  54. Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024a. · arXiv
  55. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
  56. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
  57. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. arXiv preprint arXiv:2406.11939, 2024b. · arXiv
  58. Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=WsRHpHH4s0. MiniMax-01: Scaling Foundation Models with Lightning Attention
  59. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 21558-21572, 2023.
  60. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. OCRBench: On the hidden mystery of OCR in large multimodal models. Science China Information Sciences, 67(12):220102, 2024b.
  61. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  62. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. · arXiv
  63. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523, 2024. · arXiv
  64. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022. · arXiv
  65. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021.
  66. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. · arXiv
  67. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, 2018.
  68. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50376, 2023.
  69. Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don't retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv:2407.07263, 2024. · arXiv
  70. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessan- dro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=kM5eGcdCzq.
  71. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=n6SCkn2QaG.
  72. Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025-7041, 2022a.
  73. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosFormer: Rethinking softmax in attention. In International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?id=Bl8CQrx2Up4.
  74. Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id= IxmWsm4xrua.
  75. Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 33202-33221, 2023b.
  76. Zhen Qin, Yuxin Mao, Xuyang Shen, Dong Li, Jing Zhang, Yuchao Dai, and Yiran Zhong. You only scan once: Efficient multi-dimension sequential modeling with lightnet. arXiv preprint arXiv:2405.21022, 2024a. · arXiv
  77. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401.04658, 2024b. · arXiv
  78. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In International conference on machine learning, pages 41517-41535.
  79. PMLR, 2024c.
  80. Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. HGRN2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024d. · arXiv
  81. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021. · arXiv
  82. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 53728-53741, 2023.
  83. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
  84. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id= Ti67584b98.
  85. Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522, 2024. MiniMax-01: Scaling Foundation Models with Lightning Attention · arXiv
  86. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
  87. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473, 2019.
  88. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. · arXiv
  89. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. · arXiv
  90. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. SCROLLS: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12007-12021, 2022.
  91. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. · arXiv
  92. Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16377-16426, 2024.
  93. Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Linear attention sequence parallelism. arXiv preprint arXiv:2404.02882, 2024. · arXiv
  94. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=tbVWug9f2h.
  95. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024a. · arXiv
  96. Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid Transformer-Mamba models at scale. arXiv preprint arXiv:2408.12570, 2024b. · arXiv
  97. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  98. Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024. · arXiv
  99. Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
  100. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, 2019.
  101. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understand- ing benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b. URL https://openreview.net/forum?id=y10DM6R2r3.
  102. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
  103. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024. · arXiv
  104. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. · arXiv
  105. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=Ee277P3AYC.
  106. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024a.
  107. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024b. · arXiv
  108. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big Bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020.
  109. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
  110. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Références · arXiv
  111. Zhang J. Unité de base des neurones du cerveau : neu- rones, synapses et potentiel d'action. arXiv preprint arXiv:190601703. 2019.
  112. Stein BE, Stanford TR, Rowland BA. La base neu- rale de l'intégration multisensorielle dans le mésencéphale : son organisation et sa maturation. Hear Res. 2009;258(1-2):4- 15.
  113. Shigihara Y, Zeki S. Traitement par- allèle dans le système visuel de la forme du cerveau : une étude fMRI. Front Hum Neurosci. 2014;8:506.
  114. Egorova N, Shty- rov Y, Pulvermüller F. Traitement précoce et parallèle de l'information pragmatique et sémantique dans les actes de parole : preuves neurophysiologiques. Front Hum Neurosci. 2013;7:86.
  115. Lang EW, Tome AM, Keck IR, Gorriz-Saez J, Puntonet CG. Analyse de la connectivité cérébrale : une courte enquête. Comput Intell Neurosci. 2012;2012:8.
  116. Demarin V, MOROVIC. Pe- riodicum Biologorum. vol. 116. 2014:209- 211.
  117. S. Neuroplasticité. 9. Funahashi S. Mémoire de travail dans le cortex préfrontal. Brain Sci. 2017;7(5):49.
  118. Teffer K, Semende- feri K. Cortex préfrontal humain : évolu- tion, développement et pathologie. Prog Brain Res. 2012;195:191-218.
  119. Tur- ing AM. Computing Machinery and Intel- ligence. Springer; 2009.
  120. McCulloch WS, Pitts W. Un calcul logique des idées imma- nentes dans l'activité nerveuse. Bull Math Biophys. 1943;5:115-133.
  121. Werbos P. Au-delà de la régression : nouveaux outils pour la prédic- tion et l'analyse dans les sciences du com- portement. PhD Thesis, Committee on Ap- plied Mathematics. Cambridge, MA: Har- vard University; 1974.
  122. LeCun Y, Bengio Y. Réseaux convolutifs pour les images, la parole et les séries temporelles. Le manuel de la théorie du cerveau et des réseaux neuronaux. 1995;3361(10):1995.
  123. Hubel DH, Wiesel TN. Champs ré- cepteurs, interaction binoculaire et archi- tecture fonctionnelle dans le cortex visuel du chat. J Physiol. 1962;160(1):106.
  124. Posner MI, Petersen SE. Le système d'attention du cerveau humain. Annu Rev Neurosci. 1990;13(1):25-42.
  125. Devlin J, Chang MW, Lee K, Toutanova K. Bert : pré-formation de transformateurs bidirec- tionnels profonds pour la compréhension du langage. arXiv preprint arXiv:181004805. 2018.
  126. Radford A, Narasimhan K, Sali- mans T, Sutskever I. Amélioration de la com- préhension du langage par la pré-formation générative. Open. 2018. 22. Dosovit- skiy A, Beyer L, Kolesnikov A, et al. Une image vaut 16x16 mots : transformateurs pour la reconnaissance d'images à grande échelle. arXiv preprint arXiv:201011929. 2020.
  127. Bassett DS, Bullmore E. Réseaux cérébraux petit-mondevol. 12. The neu- roscientist; 2006:512523.
  128. Bullmore -E, Sporns O. Réseaux cérébraux complexes : analyse théorique des systèmes struc- turels et fonctionnels. Nat Rev Neurosci. 2009;10(3):186-198.
  129. Bassett DS, Bull- more ET. Réseaux cérébraux petit-monde revisités. Neuro-scientist. 2017; 23(5):499- 516.
  130. Xie S, Kiril-lov A, Girshick R, He K. Exploration de réseaux neuronaux connectés aléatoirement pour la reconnais- sance d'images. Dans : Proceedings of the IEEE/CVF International Conference on Com- puter Vision. 2019:1284-1293.
  131. Tol- stikhin IO, Houlsby N, Ko-lesnikov A, et al. Mlp-mixer : une architecture tout-MLP pour la vision. Adv Neural Inf Process Syst. 2021;34:24261-24272.
  132. You J, Leskovec J, He K, Xie S. Graph structure of n-eural networks. Dans : International Conference on Machine Learning. PMLR; 2020:10881- 10891.
  133. Chen Y, Du Y, Xiao Z, et al. Une représentation relationnelle unifiée et bi- ologiquement plausible des transformateurs de vision. arXiv preprint arXiv:220611073. 2022.
  134. Zhao L , L, Dai H, Wu Z, et al. Couplage de la sémantique visuelle des réseaux neuronaux artificiels et de la fonction cérébrale humaine via des activa- tions synchronisées. arXiv preprint arXiv: 220610821. 2022.
  135. Liu X, Zhou M, Shi G, et al. Couplage des neurones artifi- ciels dans BERT et des neurones biologiques dans le cerveau humain. arXiv preprint arXiv:230314871. 2023.
  136. Huang H, Zhao L, Hu X , X, et al. BI avan : réseau d'attention visuelle antagoniste inspiré du cerveau. arXiv preprint arXiv:221015790. 2022.
  137. Yu X, Zhang L, Dai H, et al. Redéf- inition de l'auto-attention dans les trans- formateurs guidée par le principe cœur- périphérie. arXiv preprint arXiv:230315569. 2023.
  138. Zhao L, Dai H, Wu Z, Zhu D, Liu T, Cnn CP-. Réseaux de neurones convolutifs guidés par le principe cœur- périphérie. arXiv preprint arXiv:230410515. 2023.
  139. Ghosh-Dastidar S, Adeli H. Réseaux neuronaux à pointes. Int J Neural Syst. 2009;19(4):295308.
  140. Kasabov NK. NeuCube : une architecture de réseau neuronal à pointes pour le map- page, l'apprentissage et la compréhension des données cérébrales spatio-temporelles. Neural Network. 2014;52:62-76.
  141. Ku- marasinghe K, Kasabov N, Taylor D. Réseaux neuronaux à pointes inspirés du cerveau pour décoder et comprendre l'activité mus- culaire et la cinématique à partir des sig- naux d'électroencéphalographie pendant les mouvements de la main. Sci Rep. 2021;11(1):2486.
  142. Dethier J, Nuyu- jukian P, Ryu SI, Shenoy KV, Boahen K. Con- ception et validation d'un décodeur en temps réel de réseau neuronal à pointes pour les interfaces cerveau-machine. J Neural Eng. 2013;10(3):036008.
  143. Kumarasinghe K, Kasabov N, Taylor D. Apprentissage pro- fond et représentation profonde des connais- sances dans les réseaux neuronaux à pointes pour les interfaces cerveau-ordinateur. Neu- ral Network. 2020;121:169-185.
  144. Merolla PA, Arthur JV, Alvarez-Icaza R, et al. Un circuit intégré d'un million de neu- rones à pointes avec un réseau de commu- nication et une interface évolutifs. Science. 2014;345(6197):668-673.
  145. Benjamin BV, Gao P, McQuinn E, et al. Neurogrid : un système multichip analogique-numérique pour les simulations neurales à grande échelle. Proc IEEE. 2014;102(5):699-716.
  146. Zhang B, Shi L, Song S. Créer des robots plus intelligents grâce au cal- cul inspiré du cerveau. Science Robotics. 2016;354(6318):1445.
  147. Davies M, Srini- vasa N, Lin TH, et al. Loihi : un processeur neuromorphique multicœur avec apprentis- sage intégré. Ieee Micro. 2018;38(1):82-99.
  148. Roy K, Jaiswal A, Panda P. Vers une intelligence machine basée sur les pointes avec le calcul neuromorphique. Nature. 2019;575(7784):607-617.
  149. Pei J, Deng L, Song S, et al. Vers l'intelligence générale ar- tificielle avec l'architecture de puce hybride Tianjic. Nature. 2019;572(7767):106-111.
  150. Akopyan F, Sawada J, Cassidy A, et al. TrueNorth : conception et flux de travail d'une puce neurosynaptique pro- grammable d'un million de neurones de 65 mw. IEEE Trans Comput Aided Des Integrated Circ Syst. 2015;34(10):1537- 1557.
  151. Indiveri G, Douglas R. Cap- teurs de vision neuromorphiques. Science. 2000;288(5469):1189-1190.
  152. Viale A, Marchisio A, Martina M, Masera G, Shafique M. LaneSNNs : réseaux neu- ronaux à pointes pour la détection des voies sur le processeur neuromorphique Loihi. Dans : 2022 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS). IEEE; 2022:79-86.
  153. Schafer W. Systèmes nerveux des nématodes. Curr Biol. 2016;26(20):R955-R959.
  154. Schef- fer LK, Xu CS, Januszewski M, et al. Un connectome et une analyse du cerveau cen- tral de la mouche drosophile adulte. Elife. 2020;9:e57443.
  155. Er€o C, Gewaltig MO, Keller D, Markram H. Un atlas cellulaire pour le cerveau de la souris. Front Neu- roinf. 2018;12:84.
  156. Christensen JR, Larsen KB, Lisanby SH, et al. Nombre de neurones et de cellules gliales néocorticales et hippocampiques chez le macaque rhésus. Anat Rec: Advances in Integrative Anatomy and Evolutionary Biology: Advances in In- tegrative Anatomy and Evolutionary Biol- ogy. 2007;290(3):330-340.
  157. Dicke U, Roth G. Facteurs neuronaux déterminant l'intelligence élevée. Phil Trans Biol Sci. 2016;371(1685):20150180.
  158. Stanley KO, D'Ambrosio DB, Gauci J. Un encodage basé sur l'hypercube pour l'évolution des réseaux neuronaux à grande échelle. Ar- tif Life. 2009;15(2):185-212.
  159. Hutten- locher PR. Densité synaptique dans le cortex frontal humain -changements développe- mentaux et effets du vieillissement. Brain Res. 1979;163(2):195-205.
  160. Rakic P. Un petit pas pour la cellule, un grand pas pour l'humanité : une hypothèse de l'expansion néocorticale au cours de l'évolution. Trends Neurosci. 1995;18(9):383-388.
  161. Sporns O. Le connectome humain : un réseau com- plexe. Ann N Y Acad Sci. 2011;1224(1):109- 125.
  162. Devlin J, Cha-ng MW, Lee K, Toutanova K. BERT : pré-formation de trans- formateurs bidirectionnels profonds pour la compréhension du langage. Dans : NAACL HLT 2019 -2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies -Proceedings of the Con- ference. vol. 1. 2019:4171-4186. Mlm.
  163. Radford A, Narasimhan K, Salimans T, Sutskever I, et al. Amélioration de la com- préhension du langage par la pré-formation générative. CoRR; 2018.
  164. Liu Y, Ott M, Goyal N, et al. Roberta : une approche de pré-formation BERT robuste et optimisée. 2019. arXiv preprint arXiv:1907.11692. · arXiv
  165. Sanh V, Debut L, Chaumond J, Wolf T. Dis- tilBERT, une version distillée de BERT : plus petit, plus rapide, moins cher et plus léger. 2019. arXiv preprint arXiv:1910.01108. · arXiv
  166. Lepikhin D, Lee H, Xu Y, et al. Gshard : Scal- ing Giant Models with Conditional Compu- tation and Automatic Sharding. 2020. arXiv preprint arXiv:2006.16668. · arXiv
  167. Zhang Z, Han X, Liu Z, Jiang X, Sun M, Liu Q. ERNIE : Amélioration de la représentation du lan- gage avec des entités informatives. 2019. arXiv preprint arXiv:1905.07129. · arXiv
  168. Lewis M, Liu Y, Goyal N, et al. Bart : Denois- ing Sequence-To-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. 2019. arXiv preprint arXiv:1910.13461. · arXiv
  169. Raffel C, Shazeer N, Roberts A, et al. Exploration des limites du transfert learning avec un transforma- teur texte-vers-texte unifié. J Mach Learn Res. 2020;21(1):5485-5551.
  170. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdi- nov RR, Le QV. Xlnet : pré-formation au- torégressive généralisée pour la compréhen- sion du langage. Adv Neural Inf Process Syst. 2019;32. 70. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Les modèles de langage sont des apprenants multitâches non supervisés. OpenAI blog. 2019;1(8):9.
  171. Clark K, Luong MT, Le QV, Manning CD. Electra : pré-formation des encodeurs de texte comme discrimina- teurs plutôt que comme générateurs. 2020. arXiv preprint arXiv:2003.10555. · arXiv
  172. He P, Liu X, Gao J, Chen W. Deberta : Decoding- Enhanced Bert with Disentangled Attention. 2020. arXiv preprint arXiv:2006.03654. · arXiv
  173. Nakano R, Hilton J, Balaji S, et al. Webgpt : question-answering assisté par navigateur avec retour humain. 2021. arXiv preprint arXiv:2112.09332. · arXiv
  174. Wei J, Bosma M, Zhao VY, et al. Finetuned Language Mod- els Are Zero-Shot Learners. 2021. arXiv preprint arXiv:2109.01652. · arXiv
  175. Zhang Z, Gu Y, Han X, et al. Cpm-2 : modèles de langage pré-entraînés à grande échelle et rentables. AI Open. 2021;2:216-224.
  176. Xue L, Constant N, Roberts A, et al. mT5 : Un transformateur pré-entraîné texte-vers-texte multilingue massif. 2020. arXiv preprint arXiv:2010.11934. · arXiv
  177. Sanh V, Webson A, Raffel C, et al. Multitask Prompted Train- ing Enables Zero-Shot Task Generalization. 2021. arXiv preprint arXiv:2110.08207. · arXiv
  178. Brown T, Mann B, Ryder N, et al. Les modèles de langage sont des apprenants few-shot. Adv Neural Inf Process Syst. 2020;33:1877-1901.
  179. Nijkamp E, Pang B, Hayashi H, et al. Codegen : Un modèle de langage génératif open large pour le code avec syn- thèse de programme multi-tour. 2022. arXiv preprint arXiv:2203.13474. · arXiv
  180. Ganguli D, Hernandez D, Lovitt L, et al. Prédictabil- ité et surprise dans les grands modèles génératifs. Dans : Proceedings of the 2022 ACM Conference on Fairness, Accountabil- ity, and Transparency. 2022:1747-1764.
  181. Smith S, Patwary M, Norick B, et al. Using DeepSpeed and Megatron to Train MegatronTuring Nlg 530b, a Large-Scale Generative Language Model. 2022. arXiv preprint arXiv:2201.11990. · arXiv
  182. Biderman S, Schoelkopf H, Anthony QG, et al. Pythia : une suite pour analyser les grands modèles de langage à travers la formation et la mise à l'échelle. Dans : International Conference on Machine Learning. PMLR; 2023:2397-2430.
  183. Muennighoff N, Wang T, Sutawika L, et al. Généralisation interlinguistique grâce au multitâche finetuning. 2022. arXiv preprint arXiv:2211.01786. · arXiv
  184. Du N, Huang Y, Dai AM, et al. Glam : mise à l'échelle efficace des modèles de langage avec mélange d'experts. Dans : International Conference on Ma- chine Learning. PMLR; 2022:5547-5569.
  185. Lieber O, Sharir O, Lenz B, Shoham Y. Jurassic-1 : Détails techniques et éval- uation. White Paper. AI21 Labs; 2021:1.
  186. Rae JW, Borgeaud S, Cai T, et al. Scal- ing Language Models : Methods, Analy- sis & Insights from Training Gopher. 2021. arXiv preprint arXiv:2112.11446. · arXiv
  187. Sun Y, Wang S, Feng S, et al. Ernie 3.0 : Large- Scale Knowledge Enhanced Pretraining for Language Understanding and Generation. 2021. arXiv preprint arXiv:2107.02137. · arXiv
  188. Black S, Biderman S, Hallahan E, et al. Gpt-neox-20b : Un modèle de lan- gage autorégressif open-source. 2022. arXiv preprint arXiv:2204.06745. · arXiv
  189. Zoph B, Bello I, Kumar S, et al. St-moe : Designing Stable and Transferable Sparse Expert Mod- els. 2022. arXiv preprint arXiv:2202.08906. · arXiv

Cited by

Sorted by citations

Related

Authors
MiniMax, Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., Jiao, E., Li, G., Zhang, G., Sun, H., Dong, H., Zhu, J., Zhuang, J., Song, J., Zhu, J., Han, J., Li, J., Xie, J., Xu, J., Yan, J., Zhang, K., Xiao, K., Kang, K., Han, L., Wang, L., Yu, L., Feng, L., Zheng, L., Chai, L., Xing, L., Ju, M., Chi, M., Zhang, M., Huang, P., Niu, P., Li, P., Zhao, P., Yang, Q., Xu, Q., Wang, Q., Wang, Q., Li, Q., Leng, R., Shi, S., Yu, S., Li, S., Zhu, S., Huang, T., Liang, T., Sun, W., Sun, W., Cheng, W., Li, W., Song, X., Su, X., Han, X., Zhang, X., Hou, X., Min, X., Zou, X., Shen, X., Gong, Y., Zhu, Y., Zhou, Y., Zhong, Y., Hu, Y., Fan, Y., Yu, Y., Yang, Y., Li, Y., Huang, Y., Li, Y., Huang, Y., Xu, Y., Mao, Y., Li, Z., Li, Z., Tao, Z., Ying, Z., Cong, Z., Qin, Z., Fan, Z., Yu, Z., Jiang, Z., & Wu, Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.08313
arXiv
2501.08313
OpenAlex
W4406465766
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
mixture of experts, foundation models, long-context modeling, context window, vision-language models, communication-computation overlap, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review