The DOI matches its DataCite record; 25 of the 25 references it cites were found in a registry or an index.
Mixtral of Experts
Open accessAbstract
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Our selection includes GPT-4o and GPT-4o-mini (OpenAI), Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B,enabling analysis across proprietary and openweight model families.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Jiang et al., 2024
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MoE models typically activate a relatively small number of routed experts (e.g., 8 experts in Mixtral [111]) per layer, with each expert being an FFN.”
From A Survey of Large Language Models (2026) · cites this paper as [111]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For this purpose, an auxiliary loss is commonly introduced to achieve load balancing among experts, typically minimizing the product of activation frequency and average gating scores across all experts [111].”
From A Survey of Large Language Models (2026) · cites this paper as [111]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Mixtral 8×22B (Jiang et al., 2024) represents open-weight mixture-of-experts models, allowing us to examine rubric generation and application under sparse activation settings (64K context).”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Jiang et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Minor design variations have been shown to significantly affect both efficiency and specialization ability (Dai et al., 2024; Jiang et al., 2024; Liu et al., 2024b; Qiu et al., 2025; Team, 2024).”
From ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution (2025) · cites this paper as Jiang et al., 2024
Experimental — accuracy not yet measured
The cited source says what the claim attributes to it.
In the citing paper
“Notably, early large MoE models, like Mixtral [111], typically chose topexperts and employed the softmax function to compute the final gating values.”
From A Survey of Large Language Models (2026) · cites this paper as [111]
The cited source says what the claim attributes to it.
In the citing paper
“which has demonstrated its ability to either surpass or equal the performance of other models like llama-2-70B and gpt-3.5 across a range of benchmarks.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [28]
The cited source says what the claim attributes to it.
In the citing paper
“While they do not report their precise evaluation set-up, the 'MMLU-style' prompt appears to produce scores consistent with those reported in Jiang et al. (2024) for 25-shot ARC Challenge on Mixtral-8x7B and Llama 2-70B, while Cloze-style or other prompts we tested do not.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Jiang et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Our latest results with FRESHEVAL suggest that opensource LLMS such as MIXTRAL (Jiang et al., 2024), when combined with FRESHPROMPT, are competitive with closed-source and commercial systems on search-augmented QA.”
From FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2024) · cites this paper as Jiang et al., 2024
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Currently, Mixtral-8 × 7B28 and Llama 2-70B-Chat29 do not report training on the MedQA test or train set.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [28]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“the model with the lowest per-character log-likelihood (Mixtral-8x22b) and the model with the highest per-character log-likelihood (Mixtral-8x22b-Instruct) are not only variations of the same model, but also have similar levels of overfit (Jiang et al. [2024]).”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Jiang et al. [2024]
Experimental — accuracy not yet measuredIn the citing paper
“We chose these popular models because they are recent and highperforming general models also capable of code generation, and they are freely available and open-source.”
From Helping LLMs improve code generation using feedback from testing and static analysis (2026) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 10
- Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432-7439, 2020.
- Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pages 4057-4086. PMLR, 2022.
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872, 2021.
- Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335-29347, 2021.
- Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. · arXiv
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, pages 453-466, 2019.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, pages 99-106, 2021.
- Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203
Sentiment Analysis in the Age of Generative AI
Krugmann, J. O., & Hartmann, J. (2024) · Customer Needs and Solutions · doi:10.1007/s40547-024-00143-4 · Cited by 189
Metadata
- Authors
- Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., & Sayed, W. E.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2401.04088
- OpenAlex
- W4390723197
- Cited by
- 143
- References
- 0
- Citations by year
- 2026: 47 · 2025: 57 · 2024: 39
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- sparse mixture-of-experts, Mixtral-8x7B, Mistral-7B, expert routing, large language models, instruction tuning, code generation, multilingual benchmark, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
