Mixtral of Experts
Open accessAbstract
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The evaluated LLMs include GPT-4o, GPT-4o-mini, Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B, covering proprietary and open-weight model families.
“Our selection includes GPT-4o and GPT-4o-mini (OpenAI), Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B,enabling analysis across proprietary and openweight model families.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Jiang et al., 2024
The Introduction identifies the model presented in this paper as Mixtral 8x7B, not Mixtral-8×22B. The Results likewise evaluate Mixtral 8x7B alongside Llama 2 and GPT-3.5; they do not identify the cited paper as a source for the named Mixtral-8×22B model. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredMoE models typically activate a relatively small number of routed experts per layer, with each expert being an FFN; Mixtral uses 8 experts.
“MoE models typically activate a relatively small number of routed experts (e.g., 8 experts in Mixtral [111]) per layer, with each expert being an FFN.”
From A Survey of Large Language Models (2026) · cites this paper as [111]
The Introduction says Mixtral has eight expert parameter groups and that a router chooses two per token at every layer; it also identifies an expert as a feedforward block. Thus the FFN description and eight-expert total are supported, but the sentence is misleading if it means Mixtral activates eight experts per token: it activates two. (LLM, conf=0.90)
Experimental — accuracy not yet measuredAn auxiliary loss is commonly introduced to achieve load balancing among experts, typically minimizing the product of activation frequency and average gating scores across all experts.
“For this purpose, an auxiliary loss is commonly introduced to achieve load balancing among experts, typically minimizing the product of activation frequency and average gating scores across all experts [111].”
From A Survey of Large Language Models (2026) · cites this paper as [111]
Section 2.1 (page unavailable) says expert parallelism requires an even distribution of work across GPUs, supporting the need for load balancing. It does not describe an auxiliary loss or a product of activation frequency and average gating scores; searches for those terms found no such passage. (LLM, conf=0.76)
Experimental — accuracy not yet measuredMixtral 8×22B represents open-weight mixture-of-experts models, allowing examination of rubric generation and application under sparse activation settings (64K context).
“Mixtral 8×22B (Jiang et al., 2024) represents open-weight mixture-of-experts models, allowing us to examine rubric generation and application under sparse activation settings (64K context).”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Jiang et al., 2024
The Introduction identifies Mixtral as an open-weight, sparsely activated mixture-of-experts model, supporting the general model description. But the Introduction and Abstract specify Mixtral 8×7B with a 32K-token context, not Mixtral 8×22B with a 64K context. Rubric generation and application are the citing authors’ proposed uses, not findings attributed to this source. (LLM, conf=0.90)
Experimental — accuracy not yet measuredMinor design variations have been shown to significantly affect both efficiency and specialization ability.
“Minor design variations have been shown to significantly affect both efficiency and specialization ability (Dai et al., 2024; Jiang et al., 2024; Liu et al., 2024b; Qiu et al., 2025; Team, 2024).”
From ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution (2025) · cites this paper as Jiang et al., 2024
The Size and Efficiency section reports that Mixtral uses fewer active parameters per token than Llama 2 70B while outperforming it across most categories, supporting an efficiency difference between architectures. But the paper does not test minor design variations, and its Routing analysis (§5; page unavailable) finds no obvious topic-based expert specialization. It therefore does not show that minor variations significantly affect both efficiency and specialization ability. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Early large MoE models, like Mixtral, typically chose top-k experts and employed the softmax function to compute the final gating values.
“Notably, early large MoE models, like Mixtral [111], typically chose topexperts and employed the softmax function to compute the final gating values.”
From A Survey of Large Language Models (2026) · cites this paper as [111]
Mixtral 8 × 7B has demonstrated the ability to either surpass or equal the performance of other models like llama-2-70B and gpt-3.5 across a range of benchmarks.
“which has demonstrated its ability to either surpass or equal the performance of other models like llama-2-70B and gpt-3.5 across a range of benchmarks.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [28]
The 'MMLU-style' prompt appears to produce scores consistent with those reported in Jiang et al. (2024) for 25-shot ARC Challenge on Mixtral-8x7B and Llama 2-70B, while Cloze-style or other prompts we tested do not.
“While they do not report their precise evaluation set-up, the 'MMLU-style' prompt appears to produce scores consistent with those reported in Jiang et al. (2024) for 25-shot ARC Challenge on Mixtral-8x7B and Llama 2-70B, while Cloze-style or other prompts we tested do not.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Jiang et al., 2024
Open-source LLMs such as MIXTRAL, when combined with FRESHPROMPT, are competitive with closed-source and commercial systems on search-augmented QA.
“Our latest results with FRESHEVAL suggest that opensource LLMS such as MIXTRAL (Jiang et al., 2024), when combined with FRESHPROMPT, are competitive with closed-source and commercial systems on search-augmented QA.”
From FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2024) · cites this paper as Jiang et al., 2024
Mixtral-8 × 7B and Llama 2-70B-Chat do not report training on the MedQA test or train set.
“Currently, Mixtral-8 × 7B28 and Llama 2-70B-Chat29 do not report training on the MedQA test or train set.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [28]
The Results section lists benchmarks used to evaluate Mixtral and Llama 2, but does not address whether either model was trained on MedQA. Searches for MedQA and medical question-answering training data found no relevant passage in our copy; the absence of a report does not establish the claim under this rubric. (LLM, conf=0.70)
Experimental — accuracy not yet measuredThe selected models are recent, high-performing general models capable of code generation, freely available and open-source.
“We chose these popular models because they are recent and highperforming general models also capable of code generation, and they are freely available and open-source.”
From Helping LLMs improve code generation using feedback from testing and static analysis (2026) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 10
- Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432-7439, 2020.
- Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pages 4057-4086. PMLR, 2022.
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862-872, 2021.
- Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335-29347, 2021.
- Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. · arXiv
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, pages 453-466, 2019.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, pages 99-106, 2021.
- Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310 · see citing works
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203 · see citing works
Sentiment Analysis in the Age of Generative AI
Krugmann, J. O., & Hartmann, J. (2024) · Customer Needs and Solutions · doi:10.1007/s40547-024-00143-4 · Cited by 189 · see citing works
Ferber, D., El Nahhas, O. S. M., Wölflein, G., Wiest, I. C., Clusmann, J., Leßmann, M.-E., Foersch, S., Lammert, J., Tschochohei, M., Jäger, D., et al. (2025) · Nature Cancer · doi:10.1038/s43018-025-00991-6 · Cited by 175 · see citing works
The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges
Bulut, O., Beiting-Parrish, M., Casabianca, J., Slater, S. C., Jiao, H., Song, D., Ormerod, C., Fabiyi, D. G., Ivan, R., Walsh, C., et al. (2024) · Chinese/English Journal of Educational Measurement and Evaluation · doi:10.59863/miql7785 · Cited by 83 · see citing works
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 113 · see citing works
Maharjan, J., Garikipati, A., Singh, N. P., Cyrus, L., Sharma, M., Ciobanu, M., Barnes, G., Thapa, R., Mao, Q., & Das, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-64827-6 · Cited by 116 · see citing works
On the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113 · see citing works
Evaluation of large language models for discovery of gene set function
Hu, M., Alkhairy, S., Lee, I., Pillich, R. T., Fong, D., Smith, K., Bachelder, R., Ideker, T., & Pratt, D. (2025) · Nature Methods · doi:10.1038/s41592-024-02525-x · Cited by 104 · see citing works
Generative language models exhibit social identity biases
Hu, T., Kyrychenko, Y., Rathje, S., Collier, N., van der Linden, S., & Roozenbeek, J. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00741-1 · Cited by 100 · see citing works
Learning from models beyond fine-tuning
Zheng, H., Shen, L., Tang, A., Luo, Y., Hu, H., Du, B., Wen, Y., & Tao, D. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00961-0 · Cited by 114 · see citing works
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98 · see citing works
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81 · see citing works
Knowledge sharing in manufacturing using LLM-powered tools: user study and model benchmarking
Freire, S. K., Wang, C., Foosherian, M., Wellsandt, S., Ruiz-Arenas, S., & Niforatos, E. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1293084 · Cited by 101 · see citing works
Zhang, G., Jin, Q., Zhou, Y., Wang, S., Idnay, B., Luo, Y., Park, E., Nestor, J. G., Spotnitz, M. E., Soroush, A., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01239-w · Cited by 87 · see citing works
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 124 · see citing works
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.813 · Cited by 73 · see citing works
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74 · see citing works
Evaluation and mitigation of cognitive biases in medical language models
Schmidgall, S., Harris, C., Essien, I., Olshvang, D., Rahman, T., Kim, J. W., Ziaei, R., Eshraghian, J., Abadir, P., & Chellappa, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01283-6 · Cited by 68 · see citing works
A review of LLMs and their applications in the architecture, engineering and construction industry
Kampelopoulos, D., Tsanousa, A., Vrochidis, S., & Kompatsiaris, I. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11241-7 · Cited by 69 · see citing works
MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
Tang, L., Laban, P., & Durrett, G. (2024) · arXiv · doi:10.48550/arxiv.2404.10774 · Cited by 53 · see citing works
Gupta, S., Basu, A., Nievas, M., Thomas, J., Wolfrath, N., Ramamurthi, A., Taylor, B., Kothari, A. N., Schwind, R., Miller, T. M., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01274-7 · Cited by 52 · see citing works
Testing AI on language comprehension tasks reveals insensitivity to underlying meaning
Dentella, V., Günther, F., Murphy, E., Marcus, G., & Leivada, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-79531-8 · Cited by 54 · see citing works
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations
Wei, Z., Wang, Y., Li, A., Mo, Y., & Wang, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3660147 · Cited by 51 · see citing works
VisualWebArena: evaluating multimodal agents on realistic visual web tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.50 · Cited by 50 · see citing works
Conversing with business process-aware large language models: the BPLLM framework
Bernardi, M. L., Casciani, A., Cimitile, M., & Marrella, A. (2024) · Journal of Intelligent Information Systems · doi:10.1007/s10844-024-00898-1 · Cited by 55 · see citing works
Multimodal learning enables chat-based exploration of single-cell data
Schaefer, M., Peneder, P., Malzl, D., Lombardo, S. D., Peycheva, M., Burton, J., Hakobyan, A., Sharma, V., Krausgruber, T., Sin, C., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02857-9 · Cited by 53 · see citing works
Long-form factuality in large language models
Wei, J., Yang, C., Song, X., Lu, Y., Hu, N., Huang, J., Tran, D., Peng, D., Liu, R., Huang, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.18802 · Cited by 36 · see citing works
Baek, J., Jauhar, S. K., Cucerzan, S., & Hwang, S. J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.342 · Cited by 46 · see citing works
AI–AI bias: Large language models favor communications generated by large language models
Laurito, W., Davis, B., Grietzer, P., Gavenčiak, T., Böhm, A., & Kulveit, J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415697122 · Cited by 32 · see citing works
A psychometric framework for evaluating and shaping personality traits in large language models
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01115-6 · Cited by 30 · see citing works
A survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 35 · see citing works
Drowzee: Metamorphic Testing for Fact-Conflicting Hallucination Detection in Large Language Models
Li, N., Li, Y., Liu, Y., Shi, L., Wang, K., & Wang, H. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689776 · Cited by 32 · see citing works
The Platonic Representation Hypothesis
Huh, M., Cheung, B., Wang, T., & Isola, P. (2024) · arXiv · doi:10.48550/arxiv.2405.07987 · Cited by 35 · see citing works
Self-citationMixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 25 · see citing works
“1 (Jiang et al., 2024), dbrx-instruct (The Mosaic Research Team, 2024).”
OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale
Li, H., Wu, S., Zhang, X., Huang, X., Zhang, J., Jiang, F., Wang, S., Zhang, T., Chen, J., Shi, R., et al. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749723 · Cited by 56 · see citing works
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · arXiv · doi:10.48550/arxiv.2402.13178 · Cited by 312 · see citing works
Use of large language models as a scalable approach to understanding public health discourse
Espinosa, L., & Salathé, M. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000631 · Cited by 24 · see citing works
The changing landscape of text mining: a review of approaches for ecology and evolution
Farrell, M. J., Guillarme, N. L., Brierley, L., Hunter, S. B., Scheepens, D. R., Willoughby, A. R., Yates, A., & Mideo, N. (2024) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2024.0423 · Cited by 23 · see citing works
Harder Task Needs More Experts: Dynamic Routing in MoE Models
Huang, Q., An, Z., Zhuang, N., Tao, M., Zhang, C., Jin, Y., Xu, K., Xu, K., Chen, L., Huang, S., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.696 · Cited by 23 · see citing works
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Gallifant, J., Chen, S., Moreira, P. J. F., Munch, N., Gao, M., Pond, J., Celi, L. A., Aerts, H., Hartvigsen, T., & Bitterman, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.726 · Cited by 19 · see citing works
Transforming agency: On the mode of existence of large language models
Barandiaran, X. E., & Almendros, L. S. (2025) · Phenomenology and the Cognitive Sciences · doi:10.1007/s11097-025-10094-3 · Cited by 24 · see citing works
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24 · see citing works
VeriScore: Evaluating the factuality of verifiable claims in long-form text generation
Song, Y., Kim, Y., & Iyyer, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.552 · Cited by 15 · see citing works
Gaebler, J. D., Goel, S., Huq, A., & Tambe, P. (2024) · Behavioral Science & Policy · doi:10.1177/23794607251320229 · Cited by 15 · see citing works
DirectMultiStep: Direct Route Generation for Multistep Retrosynthesis
Shee, Y., Morgunov, A., Li, H., & Batista, V. S. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01982 · Cited by 15 · see citing works
COFFE: A Code Efficiency Benchmark for Code Generation
Peng, Y., Wan, J., Li, Y., & Ren, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715727 · Cited by 15 · see citing works
Schaefer, M., Peneder, P., Malzl, D., Peycheva, M., Burton, J., Hakobyan, A., Sharma, V. S., Krausgruber, T., Menche, J., Tomazou, E. M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.15.618501 · Cited by 15 · see citing works
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 13 · see citing works
Johnson, B., Bath, T., Huang, X., Lamm, M., Earles, A., Eddington, H., Dornisch, A. M., Program, V. M. V., Jih, L., Gupta, S. N., et al. (2025) · BMJ Open Gastroenterology · doi:10.1136/bmjgast-2025-001896 · Cited by 13 · see citing works
Academic Advising Chatbot Powered with AI Agent
Tamascelli, M., Bunch, O., Fowler, B., Taeb, M., & Cohen, A. (2025) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/3696673.3723065 · Cited by 13 · see citing works
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
Krumdick, M., Koncel-Kedziorski, R., Lai, V. D., Reddy, V., Lovering, C., & Tanner, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.452 · Cited by 13 · see citing works
Bhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · 2024 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec62836.2024.10938485 · Cited by 13 · see citing works
Large Language Models for Combinatorial Optimization: A Systematic Review
Ros, F. D., Soprano, M., Gaspero, L. D., & Roitero, K. (2026) · ACM Computing Surveys · doi:10.1145/3801961 · Cited by 24 · see citing works
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 11 · see citing works
“Mixtral 8 × 7B is a language model that employs a Sparse Mixture of Experts (SMoE) architecture 28 .”
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“We tested the performance on models including Llama-7B/70B [49], Mistral-8x7B [17], LLaVA-v1.”
Debating with More Persuasive LLMs Leads to More Truthful Answers
Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., & Perez, E. (2024) · arXiv · doi:10.48550/arxiv.2402.06782 · Cited by 13 · see citing works
OLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00838 · Cited by 10 · see citing works
Large language models for PHM: a review of optimization techniques and applications
Yu, T., Tang, J., Yu, Q., Li, L., Liu, Y., & Poler, R. (2025) · Autonomous Intelligent Systems · doi:10.1007/s43684-025-00100-5 · Cited by 12 · see citing works
NewsPolyML: Multi-lingual European News Fake Assessment Dataset
Mohtaj, S., Nizamoglu, A., Sahitaj, P., Schmitt, V., Jakob, C., & Möller, S. (2024) · ACM International Workshop on Multimedia AI against Disinformation (MAD) · doi:10.1145/3643491.3660290 · Cited by 11 · see citing works
Knott, A., Pedreschi, D., Jitsuzumi, T., Leavy, S., Eyers, D. M., Chakraborti, T., Trotman, A., Sundareswaran, S., Baeza-Yates, R. A., Biecek, P., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09795-1 · Cited by 13 · see citing works
Comparative analysis of generative LLMs for labeling entities in clinical notes
del Moral-González, R., Gómez-Adorno, H., & Ramos-Flores, O. (2025) · Genomics & Informatics · doi:10.1186/s44342-024-00036-x · Cited by 12 · see citing works
The Evolution of Mixture of Experts: A Survey from Basics to Breakthroughs
Vats, A., Raja, R., Jain, V., & Chadha, A. (2024) · doi:10.20944/preprints202408.0583.v2 · Cited by 10 · see citing works
Large Language Models Can Self-Correct with Key Condition Verification
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., & Jiang, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.714 · Cited by 10 · see citing works
Qiu, Z., Huang, Z., Zheng, B., Wen, K., Wang, Z., Men, R., Titov, I., Liu, D., Zhou, J., & Lin, J. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.249 · Cited by 10 · see citing works
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
Ravichander, A., Ghela, S., Wadden, D., & Choi, Y. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.71 · Cited by 13 · see citing works
MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
Huang, Q., Vora, J., Liang, P., & Leskovec, J. (2023) · arXiv · doi:10.48550/arxiv.2310.03302 · Cited by 9 · see citing works
Prashanthan, A., Roshan, R., & Maduranga, M. (2025) · Journal of Future Artificial Intelligence and Technologies · doi:10.62411/faith.3048-3719-110 · Cited by 11 · see citing works
Helping LLMs improve code generation using feedback from testing and static analysis
Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01009-5 · Cited by 8 · see citing works
“For our experiments we consider four LLMs: Llama 3 8B and 70B [5], Gemma 7B [6], and Mixtral 8X7B [7].”
InfluentialRouteLLM: Learning to Route LLMs with Preference Data
Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2406.18665 · Cited by 15 · see citing works
“We opt to use gpt-4-1106-preview [20] as a representative model in M strong , and Mixtral 8x7B [17] as a representative model in M weak , to concretely evaluate router performance.”
InfluentialDorémus, O., Russon, D., Contrand, B., Guerra-Adames, A., Avalos, M., Gil‐Jardine, C., & Lagarde, E. (2024) · JMIR AI · doi:10.2196/57828 · Cited by 10 · see citing works
Exploring the expertise of large language models in materials science and metallurgical engineering
Olivier, B. C. M., & Lambard, G. (2025) · Digital Discovery · doi:10.1039/d4dd00319e · Cited by 8 · see citing works
Question Answering models for information extraction from perovskite materials science literature
Sipilä, M., Mehryary, F., Pyysalo, S., Ginter, F., & Todorović, M. (2025) · Communications Materials · doi:10.1038/s43246-025-00979-w · Cited by 12 · see citing works
Zhou, H., Chow, L. S., Harnack, L., Panda, S., Manoogian, E. N. C., Li, M., Xiao, Y., & Zhang, R. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag003 · Cited by 8 · see citing works
General-Purpose Models for the Chemical Sciences: LLMs and Beyond
Alampara, N., Aneesh, A., Ríos-García, M., Mirza, A., Schilling-Wilhelmi, M., Aghajani, A. A., Sun, M., Prastalo, G., & Jablonka, K. M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00583 · Cited by 12 · see citing works
Chen, Z., & Sivaraman, J. (2025) · Advanced Science · doi:10.1002/advs.202413689 · Cited by 8 · see citing works
LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection
Abassy, M., Elozeiri, K., Aziz, A., Ta, M. N., Tomar, R. V., Adhikari, B., Ahmed, S. E. D., Wang, Y., Mohammed Afzal, O., Xie, Z., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2024.emnlp-demo.35 · Cited by 7 · see citing works
Hao, C., Wang, Z., Mao, X., He, S., Li, B., Liu, H., Peng, F., & Li, W. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104302 · Cited by 7 · see citing works
Large language model performance versus human expert ratings in automated suicide risk assessment
Thomas, J., Elyoseph, Z., Kuchinke, L., & Meinlschmidt, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-22402-7 · Cited by 7 · see citing works
The Evolution of Mixture of Experts: A Survey from Basics to BreakthroughsPreprint
Vats, A., Raja, R., Jain, V., & Chadha, A. R. (2024) · Preprints.org · doi:10.20944/preprints202408.0583.v1 · Cited by 7 · see citing works
DG Comics: Semi-Automatically Authoring Graph Comics for Dynamic Graphs
Kim, J., Lee, H., Nguyen, D. M., Shin, M., Kwon, B. C., Ko, S., & Elmqvist, N. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456340 · Cited by 7
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19 · see citing works
“…downloads from the Hugging Face platform in our evaluation such as LLama-3 (AI@Meta, 2024); 32k-64k – Mistral (Jiang et al., 2023), Mixtral (Jiang et al., 2024); 128k – ChatGLM3 (Du et al., 2022), Phi-3 (Abdin et al., 2024), Command-R (Cohere, 2024); 200k – Yi (Young et al., 2024); including…”
Tokenization efficiency of current foundational large language models for the Ukrainian language
Maksymenko, D., & Turuta, O. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1538165 · Cited by 6 · see citing works
Assessing LLMs Suitability for Knowledge Graph Completion
Iga, V. I. R., & Silaghi, G. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_22 · Cited by 7 · see citing works
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5 · see citing works
“Large language models (LLMs) (Touvron et al., 2023a;b; Jiang et al., 2024) have emerged as a promising solution to this challenge, offering the potential to revolutionize the way we interact with and discover knowledge in the biomedical domain(Wang et al., 2023a; Taylor et al., 2022).”
BackgroundLessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. D., & Pretorius, A. (2023) · arXiv · doi:10.48550/arxiv.2311.17371 · Cited by 5 · see citing works
Kommineni, V. K., Ahmed, W., König‐Ries, B., & Samuel, S. (2024) · Biodiversity Information Science and Standards · doi:10.3897/biss.8.136735 · Cited by 5 · see citing works
LAOS: Large Language Model-Driven Adaptive Operator Selection for Evolutionary Algorithms
Zhang, Y., & Yi, G. (2025) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3712256.3726450 · Cited by 5 · see citing works
Yao, M. S., Chae, A., Saraiya, P. V., Kahn, C. E., Witschey, W. R. T., Gee, J. C., Sagreiya, H., & Bastani, O. (2025) · Communications Medicine · doi:10.1038/s43856-025-01061-9 · Cited by 7 · see citing works
Online Disinformation and Generative Language Models: Motivations, Challenges, and Mitigations
Guo, Z. (2024) · Companion Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589335.3651254 · Cited by 5 · see citing works
Große Sprachmodelle von OpenAI, Google, Meta, X und Co.
Nowak, S., & Sprinkart, A. M. (2024) · Die Radiologie · doi:10.1007/s00117-024-01327-8 · Cited by 5 · see citing works
Shopfloor layout generation method based on large language models
Hu, Y., Sun, Y., Wen, X., Wang, S., & Bao, J. (2025) · International Journal of Computer Integrated Manufacturing · doi:10.1080/0951192x.2025.2482258 · Cited by 5 · see citing works
Scrambled text: fine-tuning language models for OCR error correction using synthetic data
Bourne, J. (2025) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-025-00522-0 · Cited by 7 · see citing works
Training-efficient density quantum machine learning
Coyle, B., Raj, S., Mathur, N., Cherrat, E. A., Jain, N., Kazdaghli, S., & Kerenidis, I. (2025) · npj Quantum Information · doi:10.1038/s41534-025-01099-6 · Cited by 8 · see citing works
Generative AI as a New Platform for Applications DevelopmentPreprint
Cusumano, M. A., Farias, V. F., & Ramakrishnan, R. (2024) · doi:10.21428/e4baedd9.f189351f · Cited by 5 · see citing works
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10 · see citing works
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.335 · Cited by 4 · see citing works
- Authors
- Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., & Sayed, W. E.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2401.04088
- OpenAlex
- W4390723197
- Cited by
- 143
- References
- 0
- Citations by year
- 2026: 47 · 2025: 57 · 2024: 39
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- sparse mixture-of-experts, Mixtral-8x7B, Mistral-7B, expert routing, large language models, instruction tuning, code generation, multilingual benchmark, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
