The DOI matches its DataCite record; 25 of the 25 references it cites were found in a registry or an index.
Mixtral of Experts
Open accessAbstract
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Our selection includes GPT-4o and GPT-4o-mini (OpenAI), Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B,enabling analysis across proprietary and openweight model families.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Jiang et al., 2024
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MoE models typically activate a relatively small number of routed experts (e.g., 8 experts in Mixtral [111]) per layer, with each expert being an FFN.”
From A Survey of Large Language Models (2026) · cites this paper as [111]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For this purpose, an auxiliary loss is commonly introduced to achieve load balancing among experts, typically minimizing the product of activation frequency and average gating scores across all experts [111].”
From A Survey of Large Language Models (2026) · cites this paper as [111]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Mixtral 8×22B (Jiang et al., 2024) represents open-weight mixture-of-experts models, allowing us to examine rubric generation and application under sparse activation settings (64K context).”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Jiang et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Minor design variations have been shown to significantly affect both efficiency and specialization ability (Dai et al., 2024; Jiang et al., 2024; Liu et al., 2024b; Qiu et al., 2025; Team, 2024).”
From ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution (2025) · cites this paper as Jiang et al., 2024
Experimental — accuracy not yet measured
The cited source says what the claim attributes to it.
In the citing paper
“Notably, early large MoE models, like Mixtral [111], typically chose topexperts and employed the softmax function to compute the final gating values.”
From A Survey of Large Language Models (2026) · cites this paper as [111]
The cited source says what the claim attributes to it.
In the citing paper
“which has demonstrated its ability to either surpass or equal the performance of other models like llama-2-70B and gpt-3.5 across a range of benchmarks.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“While they do not report their precise evaluation set-up, the 'MMLU-style' prompt appears to produce scores consistent with those reported in Jiang et al. (2024) for 25-shot ARC Challenge on Mixtral-8x7B and Llama 2-70B, while Cloze-style or other prompts we tested do not.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Jiang et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Our latest results with FRESHEVAL suggest that opensource LLMS such as MIXTRAL (Jiang et al., 2024), when combined with FRESHPROMPT, are competitive with closed-source and commercial systems on search-augmented QA.”
From FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2024) · cites this paper as Jiang et al., 2024
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Currently, Mixtral-8 × 7B28 and Llama 2-70B-Chat29 do not report training on the MedQA test or train set.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“the model with the lowest per-character log-likelihood (Mixtral-8x22b) and the model with the highest per-character log-likelihood (Mixtral-8x22b-Instruct) are not only variations of the same model, but also have similar levels of overfit (Jiang et al. [2024]).”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Jiang et al. [2024]
Experimental — accuracy not yet measuredIn the citing paper
“We chose these popular models because they are recent and highperforming general models also capable of code generation, and they are freely available and open-source.”
From Helping LLMs improve code generation using feedback from testing and static analysis (2026) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 10
- Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432-7439, 2020.
- Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pages 4057-4086. PMLR, 2022.
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872, 2021.
- Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335-29347, 2021.
- Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. · arXiv
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, pages 453-466, 2019.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, pages 99-106, 2021.
- Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
Cited by
Sorted by citations
Items 101–180 of 180
Generative AI as a New Platform for Applications DevelopmentPreprint
Cusumano, M. A., Farias, V. F., & Ramakrishnan, R. (2024) · doi:10.21428/e4baedd9.f189351f · Cited by 7
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.335 · Cited by 4
Let’s learn step by step: Enhancing in-context learning ability with curriculum learning
Liu, Y., Liu, J., Shi, X., Cheng, Q., Huang, Y., & Lu, W. (2024) · arXiv · doi:10.48550/arxiv.2402.10738 · Cited by 4
Deductive machine learning models for product identification
Jin, T., Zhao, Q., Schofield, A. B., & Savoie, B. M. (2024) · Chemical Science · doi:10.1039/d3sc04909d · Cited by 4
Accelerating Medical Record Data Abstraction and Analysis in Muscular Dystrophy
Zhou, H., Rajamani, G., Huang, J., Jorand‐Fletcher, M., Mohamed, Y., DeGolier, K. A., Xenopoulos-Oddsson, A., Cui, E., Zingariello, C. D., Zhang, R., et al. (2025) · Neurology Clinical Practice · doi:10.1212/cpj.0000000000200542 · Cited by 4
Predicting concrete compressive strength using optimized deep learning and large language models
Zaman, S., Eid, M. M., Mattar, E. A., Khafaga, D. S., & Towfek, S. K. (2026) · Scientific Reports · doi:10.1038/s41598-026-41072-7 · Cited by 4
Agarwal, S., Wood, D. M., Murray, B. A. K., Wei, Y., Busaidi, A. A., Kafiabadi, S., Guilhem, E., Lynch, J., Townend, M., Mazumder, A., et al. (2025) · European Radiology · doi:10.1007/s00330-025-11500-9 · Cited by 7
Survey and improvement strategies for gene prioritization with large language models
Neeley, M. B., Qi, G., Wang, G., Tang, R., Mao, D., Liu, C., Pasupuleti, S., Yuan, B., Xia, F., Liu, P., et al. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf148 · Cited by 9
CeRTS: certainty retrieval token search in large language model clinical information extraction
Schimmelpfennig, L. E., Bhattarai, K., Oh, I. Y., Lever, J., Griffith, O. L., Griffith, M., Lai, A. M., & Abrams, Z. B. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104900 · Cited by 4
Alboody, A., & Slama, R. (2024) · Proceedings of the World Congress on Electrical Engineering and Computer Systems and Science · doi:10.11159/mvml24.105 · Cited by 6
Building UniGPT: A Customizable On-Premise LLM-Solution for Universities
Radas, J., Risse, B., & Vogl, R. (2025) · EPiC series in computing · doi:10.29007/jv1l · Cited by 6
Mou, Y., Lehmkuhl, J., Sauerbrunn, N., Köchel, A., Panse, J., Truh, D., Sowe, S. K., Brümmendorf, T. H., & Decker, S. (2024) · Studies in health technology and informatics · doi:10.3233/shti240507 · Cited by 4
Quantification of Large Language Model Distillation
Lee, S., Zhou, J., Ao, C., Li, K., Du, X., He, S., Wu, H., Liu, T., Liu, J., Alinejad-Rokny, H., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.248 · Cited by 4
On the Impacts of Contexts on Repository-Level Code Generation
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11927 · Cited by 3
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3
AAAR-1.0: Assessing AI's Potential to Assist Research
Lou, R., Xu, H., Wang, S., Du, J., Kamoi, R., Lu, X., Xie, J., Sun, Y., Zhang, Y., Ahn, J. J., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.22394 · Cited by 3
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
Zhang, Y., & He, Z. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.691 · Cited by 3
PILs of Knowledge: A Synthetic Benchmark for Evaluating Question Answering Systems in Healthcare
Lunardi, R., Soprano, M., Coppola, P., Mea, V. D., Mizzaro, S., & Roitero, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730283 · Cited by 3
MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
Ding, R., Xu, T., Shen, X., Ding, A. A., & Fei, Y. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765174 · Cited by 4
Large language models in 6G from standard to on-device networks
Zou, H., Zhao, Q., Lasaulce, S., Zhang, C., Tian, Y., Bariah, L., Bader, F., & Debbah, M. A. (2026) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-025-00239-6 · Cited by 6
Sarwar, T., Yepes, A., & Cavedon, L. (2025) · Journal of Medical Internet Research · doi:10.2196/73325 · Cited by 3
Volokhovskyi, V. (2024) · Innovative technologies and scientific solutions for industries · doi:10.30837/2522-9818.2024.2.048 · Cited by 4
Greek Wikipedia: A Study on Abstractive Summarization
Giarelis, N., Mastrokostas, C., & Karacapilidis, N. I. (2024) · Hellenic Conference on Artificial Intelligence · doi:10.1145/3688671.3688769 · Cited by 3
On Generative Artificial Intelligence: Open-Source is the WayPreprint
Merilehto, J. (2024) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/jnmzg · Cited by 5
Can LLMs evaluate items measuring collaborative problem-solving?Preprint
Anghel, E., Wang, Y., Gopalakrishnan, M., Mansukhani, P., & Bergner, Y. (2024) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/tuvks · Cited by 3
SkillAggregation: Reference-free LLM-Dependent Aggregation
Sun, G., Kagrecha, A., Manakul, P., Woodland, P., & Gales, M. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.756 · Cited by 3
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 24
“For example, mixture-of-experts (MoE) relies on the routing of queries to various neural sub-components, leveraging the specialized expertise of one model (Masoudnia & Ebrahimpour, 2014; Roller et al., 2021; Pfeiffer et al., 2022; Jiang et al., 2024).”
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
“We begin by discussing a number of proposed self-improvement strategies, with examples detailed in Appendix B and visualized in Figure 2.”
MethodUnderstanding and Optimizing Agentic Workflows via Shapley value
Yang, Y., Huang, B., Qi, S., Feng, C., Hu, H., Zhu, Y., Hu, J., Zhao, H., He, Z., Liu, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.00510 · Cited by 2
Observational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3
SESAME - self-supervised framework for extractive question answering over document collections
Batista, V. A., Gomes, D. D. S. M., & Evsukoff, A. G. (2024) · Journal of Intelligent Information Systems · doi:10.1007/s10844-024-00869-6 · Cited by 2
Lost but Not Only in the Middle
Hutter, J., Rau, D., Marx, M., & Kamps, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_16 · Cited by 3
Large Language Models as Narrative-Driven Recommenders
Eberhard, L., Ruprechter, T., & Helić, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714668 · Cited by 4
TacTalk: Personalizing Haptics Through Conversation
Mishra, A., & Schneider, O. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736638 · Cited by 2
Adaptive-expert-weight-based load balance scheme for dynamic routing of MoE
Wen, J., Li, X., Yao, J., Kong, X., & Cheng, P. (2025) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2025.1590994 · Cited by 2
Diff-MoE: Efficient Batched MoE Inference with Priority-Driven Differential Expert Caching
Li, K., Huang, W., Wang, Q., Zheng, L., Liao, X., Jin, H., & Xue, J. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759903 · Cited by 7
Digital Prompting in Education: A Design Framework and Bibliometric Analysis
Breitwieser, J., Bittermann, A., Theobald, M., Lauer, T., Schaaf, M., & Brod, G. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10164-1 · Cited by 3
Deciphering Molecular Embeddings with Centered Kernel Alignment
Welsch, M., Hirte, S., & Kirchmair, J. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00837 · Cited by 3
GenDetect: Generative Large Language Model Usage in Smart Contract Vulnerability Detection
Ince, P., Yu, J., Liu, J. K., Du, X., & Luo, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-2961-2_22 · Cited by 3
A Large Language Model based legal assistant for governance applicationsPreprint
Mamalis, M. E., Kalampokis, E., Fitsilis, F., Theodorakopoulos, G., & Tarabanis, K. A. (2024) · doi:10.31219/osf.io/94fmq · Cited by 2
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.09605 · Cited by 2
Berges, V.-P., Oğuz, B., Haziza, D., Yih, W.-T., Zettlemoyer, L., & Ghosh, G. (2024) · arXiv · doi:10.48550/arxiv.2412.09764 · Cited by 46
Diversity of Thought Elicits Stronger Reasoning Capabilities in Multi-Agent Debate Frameworks
University of Montreal, M. Q. A. I., & Hegazy, M. (2024) · Journal of Robotics and Automation Research · doi:10.33140/jrar.05.03.06 · Cited by 1
STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
Dycke, N., Zečević, M., Kuznetsov, I., Suess, B., Kersting, K., & Gurevych, I. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1107 · Cited by 1
Efficient large language model with analog in-memory computing
Subramoney, A. (2025) · Nature Computational Science · doi:10.1038/s43588-024-00760-y · Cited by 1
Comparison of Approaches to the Extraction of Mathematical Methods from Scientific Texts
Ismagulov, Z. S., Kosyakov, D. V., & Гуськов, А. Е. (2024) · Automatic Documentation and Mathematical Linguistics · doi:10.3103/s0005105524700328 · Cited by 1
Transforming role classification in scientific teams using LLMs and advanced predictive analytics
Seo, W., & Bu, Y. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00360 · Cited by 3
CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory
Suo, J., Liao, X., Xiao, L., Ruan, L., Wang, J., Su, X., & Huo, Z. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3715986 · Cited by 1
Relative Value Encoding in Large Language Models: A Multi-Task, Multi-Model Investigation
Hayes, W. M., Yax, N., & Palminteri, S. (2025) · Open Mind · doi:10.1162/opmi_a_00209 · Cited by 3
MTAOS: Aspect-Level Opinion Summarization with Opinion Phrase Masking
Kong, Y., Wang, H., & Wang, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_8 · Cited by 1
Lu, K., Mehdinia, S., Man, K., Wong, C. W., Mao, A., & Eftekhari, Z. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00011 · Cited by 2
Almonacid, J. E. R., Lombardo, C., Romano, M., Quiroga, A., Bianchi, P. C., Hualpa, M., Giai, C., Oviedo, X. M. A., Mansur, R. A. S., Vallejo, M. G., et al. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1633382 · Cited by 1
Cajal’s legacy in the digital era: from neuroscience foundations to deep learning
García, M., Herreras, Ó., & DeFelipe, J. (2025) · Frontiers in Neuroanatomy · doi:10.3389/fnana.2025.1672016 · Cited by 3
GraphVideoAgent: Enhancing Long-form Video Understanding with Entity Relation Graphs
Chu, M., Li, Y., & Chua, T. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755537 · Cited by 3
Bridging the mentorship divide: how large language models could reshape medical workforce equity
Nguyen, N. P., & Tran, P. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02167-z · Cited by 2
Kokate, R., & Onkar, P. S. (2026) · Design Science · doi:10.1017/dsj.2025.10044 · Cited by 4
Enabling Efficient SpMM for Sparse Attention on GEMM-Optimized Hardware with Block Aggregation
Ji, T., Balasubramanian, N., Ferdman, M., & Milder, P. A. (2026) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3748173.3779187 · Cited by 2
Li, D., Li, Y., Chen, X., & Zhang, W. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3607 · Cited by 1
Bolaños-Burgos, F., Salatino, A. A., Osborne, F., & Motta, E. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3921 · Cited by 6
AI Tools and Technologies for Academic Research
Szendi, A., Kuttor, D., & Pál, Z. (2025) · Springer eBooks · doi:10.1007/978-3-031-94809-1_3 · Cited by 1
Cache Management for Mixture-of-Experts LLMs
Angelopoulos, S., Marchal, L., Obrecht, A., & Simon, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99872-0_2 · Cited by 1
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2401.13649 · Cited by 1
Will LLMs Sink or Swim? Exploring Decision-Making Under Pressure
Kim, K., Jeon, H., Ryu, J., & Suh, B. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.668 · Cited by 1
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Koepke, A. S., Zverev, D., Ginosar, S., & Efros, A. A. (2026) · arXiv · doi:10.48550/arxiv.2604.18572
“[42]: BLOOMZ (560M–7.1B) [71], OpenLlama (3B–13B) [26], LLaMA (7B–65B) [93], OLMo (1B, 7B) [30], Gemma (2B, 7B) [23], Mistral-7B [45], Mixtral-8 × 7B [46], and Meta-Llama-3-70B [29].”
Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
Li, W., Lin, Y., Xia, M., & Jin, C. (2025) · arXiv · doi:10.48550/arxiv.2502.00674 · Cited by 62
“1 (Jiang et al., 2024a), and dbrx-instruct (Team et al., 2024b).”
SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
Li, D., Tan, Z., Qian, P., Li, Y., Chaudhary, K. S., Hu, L., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.03284 · Cited by 35
Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
Wu, F., Silwal, S., Qiuyi, & Zhang (2026) · arXiv · doi:10.48550/arxiv.2601.18999
Aruna, K. B., Kaliraj, V., Sudha, I., & Sureka, V. (2026) · EAI Endorsed Transactions on Internet of Things · doi:10.4108/eetiot.13349
Bhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · arXiv
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Johnson, W. (2026) · arXiv · Cited by 1
Does Spatial Cognition Emerge in Frontier Models?
Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., & Koltun, V. (2024) · arXiv · doi:10.48550/arxiv.2410.06468 · Cited by 49
ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
Lange, R. T., Imajuku, Y., & Cetin, E. (2025) · arXiv · doi:10.48550/arxiv.2509.19349
Zhang, J., Nolte, N., Sadhukhan, R., Chen, B., & Bottou, L. (2024) · arXiv · doi:10.48550/arxiv.2405.06394 · Cited by 10
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Bogin, B., Yang, K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., & Khot, T. (2024) · arXiv · doi:10.48550/arxiv.2409.07440
Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis
Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E. (2024) · arXiv · doi:10.48550/arxiv.2412.14841 · Cited by 8
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citationGemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Self-citation
Metadata
- Authors
- Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., & Sayed, W. E.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2401.04088
- OpenAlex
- W4390723197
- Cited by
- 143
- References
- 0
- Citations by year
- 2026: 47 · 2025: 57 · 2024: 39
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- sparse mixture-of-experts, Mixtral-8x7B, Mistral-7B, expert routing, large language models, instruction tuning, code generation, multilingual benchmark, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
