CoChat
Sign Up
PreprintCited by 1Topic ModelingAdvanced Neural Network ApplicationsNatural Language Processing Techniques

Residual Matrix Transformers: Scaling the Size of the Residual Stream

Open access
Mak, Brian, Flanigan, Jeffrey·2025·arXiv
DOI 10.48550/arxiv.2506.22696arXiv 2506.22696
Open this paper in CoChat — free accountOpen PDF

Abstract

The residual stream acts as a memory bus where transformer layers both store and access features (Elhage et al., 2021). We consider changing the mechanism for retrieving and storing information in the residual stream, and replace the residual stream of the transformer with an outer product memory matrix (Kohonen, 1972, Anderson, 1972). We call this model the Residual Matrix Transformer (RMT). We find that the RMT enjoys a number of attractive properties: 1) the size of the residual stream can be scaled independently of compute and model size, improving performance, 2) the RMT can achieve the same loss as the transformer with 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens tokens, and 3) the RMT outperforms the transformer on downstream evaluations. We theoretically analyze the transformer and the RMT, and show that the RMT allows for more efficient scaling of the residual stream, as well as improved variance propagation properties. Code for this project can be found at https://github.com/bmac3/residual-matrix-transformer.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.22696

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Residual Matrix Transformer (RMT) replaces the transformer's residual vector with an outer-product memory matrix: each layer retrieves its input from the matrix through learned read keys and stores its output back as outer products of learned write keys and values.

    “The Residual Matrix Transformer (RMT; Mak and Flanigan, 2025) replaces the transformer's residual vector with an outer-product memory matrix: each layer retrieves its input from the matrix through learned read keys and stores its output back as outer products of learned write keys and values.”

    From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Mak and Flanigan, 2025

  • The RMT's outer-product memory matrix lets the residual stream be scaled independently of parameters and compute.

    “This lets the residual stream be scaled independently of parameters and compute.”

    From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Mak and Flanigan, 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works

    “Introduction Kaplan et al. (2020) presented a new way forward for the field of NLP and AI, showing that simply scaling model size, data size, and computational budget is sufficient to advance model capabilities at an unprecedented rate.”

Flagged first, then by citations

Research works printed but not linked: 46

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. Anderson, J. A. A simple neural network generating an in- teractive memory. Mathematical Biosciences, 14(3):197- 220, 1972. ISSN 0025-5564. doi: https://doi.org/10.1016/ 0025-5564(72)90075-2. URL https://www.sciencedirect. com/science/article/pii/0025556472900752. · DOI · Paper page
  3. Bachlechner, T., Majumder, B. P., Mao, H. H., Cottrell, G. W., and McAuley, J. J. Rezero is all you need: Fast convergence at large depth. CoRR, abs/2003.04887, 2020. URL https://arxiv.org/abs/2003.04887. · arXiv
  4. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
  5. Chai, Y., Jin, S., and Hou, X. Highway transformer: Self-gating enhanced self-attentive networks. In Juraf- sky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 6887-6900, Online, July 2020. Association for Computational Lin- guistics. doi: 10.18653/v1/2020.acl-main.616. URL https://aclanthology.org/2020.acl-main.616. · DOI
  6. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. · arXiv
  7. Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 10041-10071. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/ dao24a.html.
  8. Dou, Z.-Y., Tu, Z., Wang, X., Shi, S., and Zhang, T. Exploit- ing deep representations for neural machine translation. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pp. 4253- 4262, Brussels, Belgium, October-November 2018. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/ D18-1457. URL https://aclanthology.org/D18-1457. · DOI
  9. Dou, Z.-Y., Tu, Z., Wang, X., Wang, L., Shi, S., and Zhang, T. Dynamic layer aggregation for neural machine trans- lation with routing-by-agreement. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):86- 93, Jul. 2019. doi: 10.1609/aaai.v33i01.330186. URL https://ojs.aaai.org/index.php/AAAI/article/view/3772. · DOI
  10. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M. P., Zhou, Z., Wang, T., Wang, E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q., Wu, Y., Chen, Z., and Cui, C. GLaM: Effi- cient scaling of language models with mixture-of-experts. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Re- search, pp. 5547-5569. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/du22c.html.
  11. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. · arXiv
  12. ElNokrashy, M., AlKhamissi, B., and Diab, M. Depth- wise attention (DWAtt): A layer fusion method for data- efficient classification. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Confer- ence on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 4665-4674, Torino, Italia, May 2024. ELRA and ICCL. URL https: //aclanthology.org/2024.lrec-main.417.
  13. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
  14. Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.- t. (eds.), Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Repub- lic, November 2021. Association for Computational Lin- guistics. doi: 10.18653/v1/2021.emnlp-main.446. URL https://aclanthology.org/2021.emnlp-main.446. · DOI · Paper page
  15. Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Teh, Y. W. and Titterington, M. (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https://proceedings. mlr.press/v9/glorot10a.html.
  16. Gmitro, A. F., Keller, P. E., and Gindi, G. R. Statistical performance of outer-product associative memory models. Appl. Opt., 28(10):1940-1948, May 1989. doi: 10.1364/ AO.28.001940. URL https://opg.optica.org/ao/abstract. cfm?URI=ao-28-10-1940. · DOI
  17. Godin, F., Dambre, J., and De Neve, W. Improving lan- guage modeling using densely connected recurrent neural networks. In Blunsom, P., Bordes, A., Cho, K., Cohen, S., Dyer, C., Grefenstette, E., Hermann, K. M., Rimell, L., Weston, J., and Yih, S. (eds.), Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 186- 190, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-2622. URL https://aclanthology.org/W17-2622. · DOI
  18. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L. Training compute-optimal large lan- guage models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024.
  19. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, 2017. doi: 10.1109/ CVPR.2017.243. · DOI · Paper page
  20. Huang, X. S., Perez, F., Ba, J., and Volkovs, M. Improv- ing transformer optimization through better initializa- tion. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pp. 4475-4483. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/huang20f.html.
  21. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral of experts, 2024. URL https://arxiv.org/ abs/2401.04088. · arXiv
  22. Kedia, A., Zaidi, M. A., Khyalia, S., Jung, J., Goka, H., and Lee, H. Transformers get stable: an end-to-end signal propagation theory for language models. In Proceed- ings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
  23. Kidger, P. and Garcia, C. Equinox: neural networks in JAX via callable PyTrees and filtered transformations. Differ- entiable Programming workshop at Neural Information Processing Systems 2021, 2021.
  24. Kohonen, T. Correlation matrix memories. IEEE Trans- actions on Computers, C-21(4):353-359, 1972. doi: 10.1109/TC.1972.5008975. · DOI · Paper page
  25. Luccioni, A. S., Viguier, S., and Ligozat, A.-L. Estimating the carbon footprint of bloom, a 176b parameter lan- guage model. Journal of Machine Learning Research, 24(253):1-15, 2023. URL http://jmlr.org/papers/v24/23- 0069.html.
  26. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17359-17372.
  27. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper files/paper/2022/ file/6f1d43d5a82a37e89b0665b33bf3a182-Paper- Conference.pdf.
  28. Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 50358-50376.
  29. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper files/paper/ 2023/file/9d89448b63ce1e2e8dc7af72c984c196-Paper- Conference.pdf.
  30. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Der- czynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K. S. I., Mom, F., Saito, A., Song, G., Tang, X., Wind, J., Woźniak, S., Zhang, Z., Zhou, Q., Zhu, J., and Zhu, R.-J. RWKV: Reinventing RNNs for the transformer era. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048-14077, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2023.findings-emnlp.936. URL https://aclanthology.org/2023.findings-emnlp.936/. · DOI · Paper page
  31. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019. · arXiv
  32. Shen, Y., Tan, X., He, D., Qin, T., and Liu, T.-Y. Dense in- formation flow for neural machine translation. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1294-1303, New Orleans, Louisiana, June 2018. Association for Com- putational Linguistics. doi: 10.18653/v1/N18-1117. URL https://aclanthology.org/N18-1117. · DOI
  33. Shleifer, S., Weston, J., and Ott, M. Normformer: Improved transformer pretraining with extra normalization. CoRR, abs/2110.09456, 2021. URL https://arxiv.org/abs/2110. 09456. · arXiv
  34. Srivastava, R. K., Greff, K., and Schmidhuber, J. Highway networks, 2015. URL https://arxiv.org/abs/1505.00387. · arXiv
  35. Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in NLP. In Ko- rhonen, A., Traum, D., and Màrquez, L. (eds.), Pro- ceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 3645-3650, Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10.18653/v1/P19-1355. URL https://aclanthology.org/P19-1355. · DOI · Paper page
  36. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models, 2023. URL https://arxiv.org/abs/2307.08621. · arXiv
  37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 6000-6010, Red Hook, NY, USA, 2017.
  38. Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., and Hobbhahn, M. Will we run out of data? limits of llm scaling based on human-generated data, 2024. URL https://arxiv.org/abs/2211.04325. · arXiv
  39. Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., and Wei, F. DeepNet: Scaling Transformers to 1,000 Layers . IEEE Transactions on Pattern Analy- sis & Machine Intelligence, 46(10):6761-6774, October 2024. ISSN 1939-3539. doi: 10.1109/TPAMI.2024. 3386927. URL https://doi.ieeecomputersociety.org/10. 1109/TPAMI.2024.3386927. · DOI · Paper page
  40. Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Aga, F., Huang, J., Bai, C., Gschwind, M., Gupta, A., Ott, M., Melnikov, A., Candido, S., Brooks, D., Chauhan, G., Lee, B., Lee, H.-H., Akyildiz, B., Balandat, M., Spisak, J., Jain, R., Rabbat, M., and Hazelwood, K. Sus- tainable ai: Environmental implications, challenges and opportunities. In Marculescu, D., Chi, Y., and Wu, C. (eds.), Proceedings of Machine Learning and Systems, volume 4, pp. 795-813, 2022. URL https://proceedings.mlsys.org/paper files/paper/2022/ file/462211f67c7d858f663355eff93b745e-Paper.pdf.
  41. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T.-Y. On layer normalization in the transformer architecture. In Proceed- ings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
  42. Xu, H., Song, Y., Liu, Q., van Genabith, J., and Xiong, D. Rewiring the transformer with depth-wise LSTMs. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 14122-14133, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024.lrec-main. 1231.
  43. Xu, J., Sun, X., Zhang, Z., Zhao, G., and Lin, J. Understanding and improving layer normalization. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32.
  44. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper files/paper/2019/ file/2f4fe03d77724a7217006e5d16728874-Paper.pdf.
  45. Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: tuning large neural networks via zero-shot hyperparameter transfer. In Proceedings of the 35th Inter- national Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024.
  46. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sen- tence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.

Cited by

Sorted by citations

Authors
Mak, B., & Flanigan, J.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.22696
arXiv
2506.22696
OpenAlex
W4416506257
Cited by
1
References
1
Citations by year
2025: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1730158, 1540112, 1826967, 2100237, 2112167, 1541349, 2120019)
  • University of California, San Diego
  • Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México
  • Thomas Jefferson National Accelerator Facility
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
residual stream, variance propagation, FLOPs reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review