CoChat
Sign Up
PreprintCited by 112Multimodal Machine Learning ApplicationsTopic ModelingNatural Language Processing Techniques

Gemma 4 Technical Report

Open access
Gemma Team, Abd, Sherif El, Aggarwal, Vaibhav, Algayres, Robin, Andreev, Alek, Bachem, Olivier, Ballantyne, Ian, Brick, Cormac +315 more·2026·arXiv
DOI 10.48550/arxiv.2607.02770arXiv 2607.02770
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.02770

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Gemma 4 has layers 15-34 that share K/V projections with earlier layers; the weights are untrained placeholders, and inference reads from source layers' cache via DynamicCache.

    “Gemma 4 also exercises a feature Llama doesn't have: layers 15-34 share K/V projections with earlier layers (the weights are untrained placeholders, and inference reads from source layers' cache via DynamicCache ).”

    From Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets (2026) · cites this paper as [5]

    The “Dense and MoE” section (page unavailable) says Gemma 4 shares the KV cache in its E2B and E4B models, supporting the general cache-sharing point. It does not identify layers 15–34, say their K/V projection weights are untrained placeholders, or describe inference through DynamicCache; the cited passages also do not establish the comparison with Llama. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Gemma 4's Per-Layer Embeddings (PLE) are a dense 262K × 8960 matrix-not a standard lookup table.

    “Gemma 4's Per-Layer Embeddings (PLE) are a dense 262K × 8960 matrix-not a standard lookup table.”

    From Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets (2026) · cites this paper as [5]

    The “Dense and MoE” section says that Gemma 4 E2B and E4B use per-layer embeddings. It does not describe those embeddings as a dense 262K × 8960 matrix or distinguish them from a standard lookup table. The 262,000 figure in “Efficient MTP Decoding” refers to a vocabulary projection for the drafters, not PLE dimensions; no page numbers are available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer nor- malization. arXiv preprint arXiv:1607.06450, 2016. · arXiv
  2. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axel- rod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna. Fleurs: Few-shot learning evaluation of univer- sal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798-805. IEEE, 2023.
  3. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkor- eit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  4. C. for AI Safety et al. A benchmark of expert-level academic questions to assess ai capabilities. Na- ture, 649(8099):1139-1146, 2026.
  5. Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabili- ties. arXiv preprint arXiv:2507.06261, 2025. · arXiv
  6. Gemma Team. Gemma 2: Improving open lan- guage models at a practical size. arXiv preprint arXiv:2408.00118, 2024b. · arXiv
  7. Gemma Team. Gemma 3: Technical report. arXiv preprint arXiv:2503.19786, 2025a. · arXiv
  8. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020. · arXiv
  9. A. Henry, P. R. Dachapally, S. S. Pawar, and Y. Chen. Query-key normalization for trans- formers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246-4253, 2020.
  10. B. Heo, S. Park, D. Han, and S. Yun. Rotary po- sition embedding for vision transformer. In European Conference on Computer Vision, pages 289-305. Springer, 2024.
  11. C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. · arXiv
  12. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR, 2018.
  13. N. Jain, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Live- codebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Repre- sentations, volume 2025, pages 58791-58831, 2025.
  14. A. Kayyam, A. M. Gopal, and M. A. Lewis. Do transformers need three projections? system- atic study of qkv variants. arXiv preprint arXiv:2606.04032, 2026. · arXiv
  15. M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V. Mehta, L. K. Jain, V. Aglietti, D. Jindal, P. Chen, et al. Big-bench extra hard. arXiv preprint arXiv:2502.19187, 2025. · arXiv
  16. J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. R. Arnold, V. Perot, S. Dalmia, H. Hu, X. Lin, P. Pasupat, A. Amini, J. R. Cole, S. Riedel, I. Naim, M.-W. Chang, and K. Guu. Can long-context language models subsume retrieval, rag, sql, and more? ArXiv, 2024.
  17. Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th Interna- tional Conference on Machine Learning, ICML'23. JMLR.org, 2023.
  18. Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: Speculative sampling requires rethinking fea- ture uncertainty. In International Conference on Machine Learning, 2024.
  19. M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Val- veny, and C. Jawahar. Infographicvqa. In WACV, 2022.
  20. OpenAI. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. · arXiv
  21. L. Ouyang, Y. Qu, H. Zhou, J. Zhu, R. Zhang, Q. Lin, B. Wang, Z. Zhao, M. Jiang, X. Zhao, et al. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive an- notations. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni- tion, pages 24838-24848, 2025.
  22. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025. · arXiv
  23. V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi. Gen- eralizing verifiable instruction following. Ad- vances in Neural Information Processing Systems, 38, 2026.
  24. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bow- man. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv, abs/2311.12022, 2023. · arXiv
  25. J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He. Zero-offload: Democratizing billion- scale model training. In USENIX, 2021.
  26. A. Roberts, H. W. Chung, G. Mishra, A. Levskaya, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, et al. Scaling up models and data with t5x and seqio. JMLR, 2023.
  27. N. Shazeer. Fast transformer decoding: One write- head is all you need. CoRR, abs/1911.02150, 2019. · arXiv
  28. G. Tanzer, M. Suzgun, E. Visser, D. Jurafsky, and L. Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations, 2024.
  29. K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. · arXiv
  30. Q. Team. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804, 2026. · arXiv
  31. M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624-30650, 2024.
  32. K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024. · arXiv
  33. C. Wang, J. Pino, A. Wu, and J. Gu. Covost: A diverse multilingual speech-to-text transla- tion corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4197-4203, 2020.
  34. K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li. Measuring multi- modal mathematical reasoning with math- vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024a.
  35. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In NeurIPS, 2024b.
  36. B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026. · arXiv
  37. A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: Towards highly efficient million- token context intelligence. arXiv preprint arXiv:2606.19348, 2026. · arXiv
  38. X. Yue, T. Zheng, Y. Ni, Y. Wang, K. Zhang, S. Tong, Y. Sun, B. Yu, G. Zhang, H. Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Pro- ceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 15134-15186, 2025.
  39. A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. · arXiv
  40. Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023. · arXiv
  41. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction- following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. · arXiv

Cited by

Sorted by citations

Authors
Gemma Team, Abd, S. E., Aggarwal, V., Algayres, R., Andreev, A., Bachem, O., Ballantyne, I., Brick, C., Cărbune, V., Casbon, M., Chaturvedi, M., Chawla, A., Cotruta, V., Coucke, A., Culliton, P., Dadashi, R., Dixon, L., Elhawaty, M., Evci, U., Farabet, C., Ferret, J., Galgani, F., Girgin, S., Grill, J.-B., Grootendorst, M., Guo, J., Hardin, C., He, Y., Hernandez, S. M., Homburger, O., Hussenot, L., Ji, J., Joulin, A., Kamath, A., Kassraie, P., Lacombe, O., Lahoti, P., Liu, G., Martins, G., Martins, L., Matejovicova, T., Merhej, R., Momchev, N., Mondal, S., Mullins, R., Panyam, S. R., Pathak, S., Perrin, S., Pinto, A. S., Pot, E., Pouget, A., Ramé, A., Ramos, S., Reid, D., Rim, D., Rivière, M., Roth, K., Rouillard, L., Sanseviero, O., Sessa, P. G., Settle, S., Sinopalnikov, D., Smoot, S., Stanczyk, P., Steiner, A., Stewart, L., Tolstikhin, I., Tschannen, M., Tsitsulin, A., Vieillard, N., Wu, R., Xu, P., Yang, H., Yvinec, E., Zhang, B., Zhang, L., Zou, J., Aagnes, N., Abdelhamed, A., Adamek, J., Agrawal, S., Agrawal, S., Alabdulmohsin, I., Alayrac, J. B., Alon, U., Amarnath, C., Anand, A., Anastasiou, C., Ariafar, S., Aubet, F.-X., Axiotis, K., Barbero, F., Barral, J., Bendebury, A., Bergmann, U., Bileschi, S., Black, K., Blondel, M., Borgeaud, S., Bražinskas, A., Burnell, R., Busa-Fekete, R., Cai, M., Calandriello, D., Cameron, G., Caucheteux, C., Chaabouni, R., Chadha, G., Chan, J., Chen, B. J., Chen, J., Chen, L., Chen, X., Cheng, D., Chien, T.-H., Chinaev, N., Chou, Y., Chu, Z., Coleman, B., Consul, P., Conway-Rahman, S., Crowell, S., Cutler, D., Dani, V., Daruki, S., Das, A., Deutsch, D., Dikkala, N., Ding, L., Ding, Q., Dodhia, S., Donhauser, K., Doshi, T., Dragan, A., Druinsky, A., Dua, S., Egyed, Z., Eisenbud, D., Eppens, D., Fan, C., Fatemi, B., Fathullah, Y., Feinberg, V., Ferev, M., Flennerhag, S., Fujimoto, T., Oliveira, J. G., Galatzer-Levy, I., Gante, J., Geisler, S., Ghosal, S., Girgis, A. M., von Glehn, T., Go, A., Gokhale, A., Grills, A., Gu, Y., Gupta, M., Gupta, P., Guruganesh, G., Hadsell, R., Harkous, H., Harlalka, J., Hassabis, D., Hauth, A., Heyward, J., Hosseini, A., Hsia, C.-Y., Hsu, I.-H., Huang, X., Huang, Y., Hui, K., Hutter, A., I, T., Iliopoulos, F., Jain, A., Jawahar, G., Ji, Z., Jin, Q., Johnson, M., Joshi, K., Kandoor, A., Kang, W.-C., Kavukcuoglu, K., Kazemi, M., Kenealy, K., Khalifa, A., Kirk, P., Korotkov, I., Kothawade, S., Kovalev, V., Kovelamudi, N., Kraft, A., Kumar, R., Kumar, V., Kuppam, H., Lannin, J., Lee, C.-Y., Lee, S., Lepikhin, D., Levkovitch, A., Li, D., Li, Q., Liévin, V., Lin, E., Lin, Z., Liu, C., Liu, T., Liu, T., Liu, X., Lobov, I., Lunayach, M., Ma, M., Madan, G., Maksai, A., Malmi, E., Matuszak, M., McDuff, D., Menghani, G., Mikuła, M., Mirylenka, D., Misiunas, K., Misra, V., Mitran, A., Mohamed, K., Mukha, M., Noland, E., O'Donnell, J., O'Donoghue, B., Olszewska, K., Orlando, B., Pan, W., Panigrahy, R., Parekh, U., Perez-Nieves, N., Park, C., Paskie, E., Peng, L., Petrini, B., Petrov, S., Pfeiffer, J., Piot, B., Plomecka, M., Poder, S., Ponce, O., Pramanik, A., Racz, D., Rajan, A., Ramanovich, M., Rao, A., Ritter, M., Rodrigues, V., Rosen, E., Rybiński, M., Sachdeva, N., Sander, M. E., Sathyanarayana, R., Savla, S., Schmidgall, S., Schuster, T., Scrivener, G., Seguin, B., Sellergren, A., Severyn, A., Shafran, I., Shah, D., Shahriari, B., Shangguan, Y., Shenoy, A., Shenoy, P., Shivanna, R., Sho, P., Spangher, L., Stokowiec, W., Strother, T., Su, Y., Sun, Y., Sundararajan, M., Tacchetti, A., Taege, M. H., Tafti, P., Tarbouriech, J., Tekur, C., Thakoor, S., Thapa, R., Traverse, M., Treven, L., Tu, T., Tung, C. T., Ünlü, Ç., Veličković, P., Venkat, M. P., Venkatesh, S. G., Venkiteswaran, V., Visin, F., Vitvitskyi, A., Vodrahalli, K., Wang, W., Wang, X., Warkentin, T., Wassenberg, J., Wieting, J., Wu, C., Xiao, L., Xu, H., Xu, Y., Xue, F., Yadav, A., Yan, J., Yang, A., Yang, L., Yang, M.-H., Ying, Z., Yoo, J. H., Zadimoghaddam, M., Zafar, S., Zhang, F., Zhang, J., Zhang, J., Zhang, X., Zhao, C., Zhou, D., & Zou, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.02770
arXiv
2607.02770
OpenAlex
W7167604932
Cited by
112
References
2
Citations by year
2026: 112
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
Gemma 4, open-weight language models, multimodal language models, mixture of experts, reasoning traces, long-context understanding, computational efficiency, inference efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review