AI and Memory Wall
Open accessAbstract
The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training LLMs. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0×/2yrs, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. Here, we analyze encoder and decoder Transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The focus group observations are consistent with Gholami et al.'s (2024) findings on AI's restricted memory capacity and its tendency to produce narrative incoherence in longer texts.
“These observations are consistent with Gholami et al.'s (2024) findings on AI's restricted memory capacity and its tendency to produce narrative incoherence in longer texts.”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Gholami et al.'s (2024)
The Introduction (page unavailable) describes limited memory capacity on a single accelerator when training or serving large language models. That is a hardware constraint, not a finding about how much earlier text an AI can retain; the paper does not report narrative incoherence in longer texts. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe relative weakness of zero-shot generation is consistent with research on AI's limitations in sustaining narrative coherence.
“the relative weakness of zero-shot generation is consistent with research on AI's limitations in sustaining narrative coherence (Gholami et al., 2024)”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Gholami et al., 2024
The Profiling and Conclusion sections discuss AI models’ training, deployment, and memory bottlenecks, not narrative coherence. Searches for zero-shot generation and coherent story generation found no passage in our copy addressing the claimed limitation. (LLM, conf=0.76)
Experimental — accuracy not yet measuredGholami et al. (2024) identify restricted memory capacity as one of the primary challenges of AI-assisted fiction writing: as a story progresses, the model tends to lose track of earlier plot points and character details, which result in narrative inconsistencies.
“Gholami et al. (2024) identify restricted memory capacity as one of the primary challenges of AI-assisted fiction writing: as a story progresses, the model tends to lose track of earlier plot points and character details, which result in narrative inconsistencies.”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Gholami et al. (2024)
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,484 · cited 1× · see citing works
“This is due to the higher memory operations and lower arithmetic intensity of matrix-vector operations inherent in the auto-regressive inference of GPT (see Figure 3c).”
Wulf, W. A., & McKee, S. A. (1995) · ACM SIGARCH Computer Architecture News · doi:10.1145/216585.216588 · Cited by 1,625 · see citing works
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · see citing works
Self-citationTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · see citing works
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423 · see citing works
Patterson, D. A. (2004) · Communications of the ACM · doi:10.1145/1022594.1022596 · Cited by 238 · see citing works
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
Yao, Z., Gholami, A., Shen, S., Mustafa, M., Keutzer, K., & Mahoney, M. W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17275 · Cited by 199 · see citing works
Self-citationCheckmate: Breaking the Memory Wall with Optimal Tensor RematerializationPreprint
Jain, P., Jain, A., Nrusimha, A., Gholami, A., Abbeel, P., Keutzer, K., Stoica, I. G., & Gonzalez, J. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02653 · Cited by 94 · see citing works
Self-citationFull Stack Optimization of Transformer Inference: a SurveyPreprint
Kim, S., Hooper, C. R. C., Wattanawong, T., Kang, M., Yan, R., Genc, H., Dinh, G., Huang, Q., Keutzer, K., Mahoney, M. W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.14017 · Cited by 38 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 29
- G. Ballard, J. Demmel, O. Holtz, and O. Schwartz, "Minimizing communication in numerical linear alge- bra," SIAM Journal on Matrix Analysis and Applications, vol. 32, no. 3, pp. 866-901, 2011.
- L. Bottou, F. E. Curtis, and J. Nocedal, "Optimization methods for large-scale machine learning," SIAM Review, vol. 60, no. 2, pp. 223-311, 2018.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learn- ers," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., "Palm: Scaling language modeling with pathways," Journal of Machine Learning Research, vol. 24, no. 240, pp. 1-113, 2023.
- H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma et al., "Scaling instruction-finetuned language models," arXiv preprint arXiv:2210.11416, 2022. · arXiv
- T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, "Spqr: A sparse-quantized representation for near-lossless llm weight compression," arXiv preprint arXiv:2306.03078, 2023. · arXiv
- E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, "OPTQ: Accurate quantization for generative pre-trained transformers," in The Eleventh International Conference on Learning Representations, 2023.
- J. L. Hennessy and D. A. Patterson, Computer architec- ture: a quantitative approach. Elsevier, 2011.
- A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al., "Mistral 7b," arXiv preprint arXiv:2310.06825, 2023. · arXiv
- S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, "Squeezellm: Dense-and-sparse quantization," arXiv preprint arXiv:2306.07629, 2023. · arXiv
- V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, "Reducing activation recomputation in large transformer models," Proceedings of Machine Learning and Systems, vol. 5, 2023.
- S. Krishna and R. Krishna, "Accelerating recom- mender systems via hardware scale-in," arXiv preprint arXiv:2009.05230, 2020. · arXiv
- W. Kwon, S. Kim, M. W. Mahoney, J. Hassoun, K. Keutzer, and A. Gholami, "A fast post-training prun- ing framework for transformers," in Advances in Neural Information Processing Systems, vol. 35. Curran Asso- ciates, Inc., 2022, pp. 24 101-24 116.
- Y. LeCun, L. Bottou, Y. Bengio, P. Haffner et al., "Gradient-based learning applied to document recogni- tion," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278- 2324, 1998.
- S. A. McKee, "Reflections on the memory wall," in Pro- ceedings of the 1st Conference on Computing Frontiers, ser. CF '04. New York, NY, USA: Association for Computing Machinery, 2004, p. 162.
- P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, "Mixed preci- sion training," in International Conference on Learning Representations, 2018.
- P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu et al., "Fp8 formats for deep learning," arXiv preprint arXiv:2209.05433, 2022. · arXiv
- G. Moore, "No exponential is forever: but "forever" can be delayed! [semiconductor industry]," in 2003 IEEE In- ternational Solid-State Circuits Conference, 2003. Digest of Technical Papers. ISSCC., 2003, pp. 20-23 vol.1.
- M. Naumov, D. Mudigere, H.-J. M. Shi, J. Huang, N. Sundaraman, J. Park, X. Wang, U. Gupta, C.-J. Wu, A. G. Azzolini et al., "Deep learning recommendation model for personalization and recommendation systems," arXiv preprint arXiv:1906.00091, 2019. · arXiv
- J. Ousterhout, "Why aren't operating systems getting faster as fast as hardware?" in USENIX Summer Con- ference, 1990, 1990.
- D. Patterson, T. Anderson, N. Cardwell, R. Fromm, K. Keeton, C. Kozyrakis, R. Thomas, and K. Yelick, "A case for intelligent ram," IEEE micro, vol. 17, no. 2, pp. 34-44, 1997.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
- S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, "Zero: Memory optimizations toward training trillion parameter models," in SC20: International Conference for High Performance Computing, Networking, Storage and Anal- ysis. IEEE, 2020, pp. 1-16.
- D. Sites, "It's the memory, stupid!" Microprocessor Re- port, pp. 18-24, 1996.
- H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhos- ale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in neural infor- mation processing systems, 2017, pp. 5998-6008.
- J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Met- zler et al., "Emergent abilities of large language models," arXiv preprint arXiv:2206.07682, 2022. · arXiv
- M. V. Wilkes, "The memory wall and the cmos end- point," ACM SIGARCH Computer Architecture News, vol. 23, no. 4, pp. 4-6, 1995.
- Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He, "Zeroquant: Efficient and affordable post-training quantization for large-scale transformers," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 27 168-27 183.
Cited by
Sorted by citations
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · see citing works
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · see citing works
(2021) · doi:10.1145/3458817.3476205 · Cited by 257 · see citing works
Towards sustainable AI: a comprehensive framework for Green AI
Tabbakh, A., Amin, A., Islam, M., Mahmud, G., Chowdhury, I. K., & Mukta, S. H. (2024) · Discover Sustainability · doi:10.1007/s43621-024-00641-4 · Cited by 142 · see citing works
ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization
Guo, C., Zhang, C., Leng, J., Liu, Z., Yang, F., Liu, Y., Guo, M., & Zhu, Y. (2022) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00095 · Cited by 116 · see citing works
Two-Dimensional Materials for Brain-Inspired Computing Hardware
Hadke, S. S., Kang, M., Sangwan, V. K., & Hersam, M. C. (2025) · Chemical Reviews · doi:10.1021/acs.chemrev.4c00631 · Cited by 104 · see citing works
Present and Future, Challenges of High Bandwith Memory (HBM)
Kim, K., & Park, M.-J. (2024) · IEEE International Memory Workshop (IMW) · doi:10.1109/imw59701.2024.10536972 · Cited by 70 · see citing works
PipeEdge: Pipeline Parallelism for Large-Scale Model Inference on Heterogeneous Edge Devices
Hu, Y., Imes, C., Zhao, X., Kundu, S., Beerel, P. A., Crago, S. P., & Walters, J. P. (2022) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd57027.2022.00048 · Cited by 46 · see citing works
Junior, G. F. D. O., Olgun, A., Yağlıkçı, A. G., Bostanci, F. N., Gómez-Luna, J., Ghose, S., & Mutlu, O. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00024 · Cited by 45 · see citing works
Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
Zhao, C., Deng, C., Ruan, C., Dai, D., Gao, H., Li, J., Zhang, L., Huang, P., Zhou, S., Ma, S., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731412 · Cited by 60 · see citing works
Physics and artificial intelligence: illuminating the future of optics and photonics
Rahman, M. S. S., & Özcan, A. (2024) · Advanced Photonics · doi:10.1117/1.ap.6.5.050500 · Cited by 38 · see citing works
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts
Prabhakar, R., Sivaramakrishnan, R., Gandhi, D., Du, Y., Wang, M., Song, X., Zhang, K., Gao, T., Wang, A., Li, X., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00100 · Cited by 36 · see citing works
Large-scale photonic processors and their applications
Pérez, D., & Torrijos‐Morán, L. (2025) · npj Nanophotonics · doi:10.1038/s44310-025-00075-4 · Cited by 36 · see citing works
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., et al. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2024.3476008 · Cited by 32 · see citing works
Flex-PE: Flexible and SIMD Multiprecision Processing Element for AI Workloads
Lokhande, M., Raut, G., & Vishvakarma, S. K. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3553069 · Cited by 29 · see citing works
Challenges and Opportunities to Enable Large-Scale Computing via Heterogeneous Chiplets
Yang, Z., Ji, S., Chen, X., Zhuang, J., Zhang, W., Jani, D., & Zhou, P. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473961 · Cited by 31 · see citing works
BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration
Chen, Y., AbouElhamayed, A. F., Dai, X., Wang, Y., Andronic, M., Constantinides, G. A., & Abdelfattah, M. S. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00084 · Cited by 26 · see citing works
TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting
Nie, X., Miao, X., Yang, Z. C., & Cui, B. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00241 · Cited by 26 · see citing works
BBS: Bi-Directional Bit-Level Sparsity for Deep Learning Acceleration
Chen, Y., Meng, J., Seo, J.-S., & Abdelfattah, M. S. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00048 · Cited by 25 · see citing works
Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
Yao, Z., Wu, X., Li, C., Youn, S., & He, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29908 · Cited by 64 · see citing works
Self-citationArtificial intelligence: Supply chain constraints and energy implications
de Vries, A. (2025) · Joule · doi:10.1016/j.joule.2025.101961 · Cited by 21 · see citing works
Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
An, W., Bi, X., Chen, G.-T., Chen, S., Deng, C., Ding, H., Dong, K., Du, Q., Gao, W., Guan, K., et al. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00089 · Cited by 26 · see citing works
Chiang, K. H., Ho, Y.-C., Chuang, M.-Y., Chang, C., Kao, Y.-F., Yang, H.-Y., Huang, C. ‐., Chien, Y.-J., Wu, Y.-H., Liu, Y.-C., et al. (2025) · IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits) · doi:10.23919/vlsitechnologyandcir65189.2025.11074854 · Cited by 20 · see citing works
Fischer, T., Rogenmoser, M., Benz, T., Gürkaynak, F. K., & Luca Benini, C. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3527225 · Cited by 19 · see citing works
Co-Designed Silicon Photonics Chip I/O for Energy-Efficient Petascale Connectivity
Wang, Y., Wang, S., Parsons, R., Sanyal, S., Gopal, V., Novick, A., Rizzo, A. J., Lipson, M. F., Gaeta, A. L., & Bergman, K. (2024) · IEEE Transactions on Components Packaging and Manufacturing Technology · doi:10.1109/tcpmt.2024.3492189 · Cited by 19 · see citing works
Mu, C., Huang, Z., Jiang, H., Liao, J., Zhou, Y. L., Chen, L., Zhang, Y., Zhu, H., Yang, J., Liu, Q., et al. (2025) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2025.3577335 · Cited by 17 · see citing works
Mosaic: Breaking the Optics versus Copper Trade-off with a Wide-and-Slow Architecture and MicroLEDs
Benyahya, K., Diaz, A. G., Liu, J., Lyutsarev, V., Pantouvaki, M., Shi, K., Siew, S. Y., Ballani, H., Burridge, T., Cletheroe, D., et al. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3750510 · Cited by 17 · see citing works
Green AI techniques for reducing energy consumption in AI systems
Khan, S., Naz, N. S., Mazhar, T., Tariq, M. U., Shahzad, T., Guizani, S., & Hamam, H. (2025) · Array · doi:10.1016/j.array.2025.100652 · Cited by 25 · see citing works
DASS: Differentiable Architecture Search for Sparse Neural Networks
Mousavi, H., Loni, M., Alibeigi, M., & Daneshtalab, M. (2023) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3609385 · Cited by 14 · see citing works
Ferroelectric Transistors: from Materials Innovation to Intelligent Electronic Systems
Li, E., Wang, W., Liu, Y., Wang, R., Luo, C., Zhou, H., Chen, S., Chen, S., Xie, Z., Zhu, K., et al. (2025) · Advanced Materials · doi:10.1002/adma.202515480 · Cited by 13 · see citing works
Efficient Inter-Datacenter AllReduce With Multiple Trees
Luo, S., Wang, R., & Xing, H. (2024) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2024.3419030 · Cited by 12 · see citing works
Ibrahim, M., Wan, Z., Li, H., Panda, P., Krishna, T., Kanerva, P., Chen, Y., & Raychowdhury, A. (2024) · International Conference on Hardware/Software Codesign and System Synthesis · doi:10.1109/codes-isss60120.2024.00012 · Cited by 13 · see citing works
Choi, S., Sim, J., Shin, J., Ryu, S., Hwang, T., Lim, S. Y., Oh, H., Kwag, J., Lee, J., Song, K., et al. (2024) · Small Structures · doi:10.1002/sstr.202400495 · Cited by 12 · see citing works
Emerging Nonvolatile Memory Technologies in the Future of Microelectronics
Katehi, L. P. B., Yi, S., Lin, Y. C., Banerjee, S., Xia, Q., & Yang, J. J. (2025) · ACS Omega · doi:10.1021/acsomega.5c01414 · Cited by 12 · see citing works
OPIMA: Optical Processing-in-Memory for Convolutional Neural Network Acceleration
Sunny, F., Shafiee, A., Balasubramaniam, A., Nikdast, M., & Pasricha, S. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3446870 · Cited by 11 · see citing works
PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU Clusters
Jain, R., Tran, B., Chen, K., Sinclair, M. D., & Venkataraman, S. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00032 · Cited by 11 · see citing works
HCiM: ADC-Less Hybrid Analog-Digital Compute in Memory Accelerator for Deep Learning Workloads
Negi, S., Saxena, U., Sharma, D., & Roy, K. (2025) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3658617.3697572 · Cited by 11 · see citing works
Inverse-designed nanophotonic neural network accelerators for ultra-compact optical computing
Sved, J., Song, S., Li, L., Li, G., Meng, D., & Yi, X. (2026) · Nature Communications · doi:10.1038/s41467-026-68648-1 · Cited by 10 · see citing works
Hwang, I., Lee, J., Kang, H., Lee, G., & Kim, H. (2024) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2024.103032 · Cited by 10 · see citing works
TB-STC: Transposable Block-wise N:M Structured Sparse Tensor Core
Liu, J., Zeng, S., Zhao, J., Ding, L., Wang, Z., Li, J., Zhu, Z., Ning, X., Zhang, C., Wang, Y., et al. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00075 · Cited by 10 · see citing works
Chrono: Meticulous Hotness Measurement and Flexible Page Migration for Memory Tiering
Qi, Z., Zheng, S., Huang, Y., Hui, Y., Zhang, B., Huang, L., & Mei, H. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3717462 · Cited by 9 · see citing works
UniNDP: A Unified Compilation and Simulation Tool for Near DRAM Processing Architectures
Xie, T., Zhu, Z., Li, B., He, Y., Li, C., Sun, G., Yang, H., Xie, Y., & Wang, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00054 · Cited by 9 · see citing works
Forest: Access-aware GPU UVM Management
Lin, M., Feng, Y., Cox, G., & Jeon, H. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731047 · Cited by 9 · see citing works
Emerging memory devices for neuromorphic computing in the Internet of Medical Things
Cao, T., Shen, C., Wang, D., Zhao, H.-N., Tan, R., Zhou, Z., Li, H., Li, B., Zhao, M., & Huang, H. (2025) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2025.102735 · Cited by 10 · see citing works
Vali, A., Comai, S., & Matteucci, M. (2024) · Remote Sensing · doi:10.3390/rs16142561 · Cited by 8 · see citing works
MCAIMem: A Mixed SRAM and eDRAM Cell for Area and Energy-Efficient On-Chip AI Memory
Nguyen, D.-T., Bhattacharjee, A., Moitra, A., & Panda, P. (2024) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2024.3439231 · Cited by 8 · see citing works
Bridging the Gap Between LLMs and LNS with Dynamic Data Format and Architecture Codesign
Haghi, P., Wu, C.-S., Azad, Z., Li, Y., Gui, A., Hao, Y., Li, A., & Geng, T. (2024) · Micro · doi:10.1109/micro61859.2024.00118 · Cited by 10 · see citing works
A Framework for Near Memory Processing With Computation Offloading and Load Balancing
Maity, S., Ghose, M., & Pasricha, S. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3542361 · Cited by 8 · see citing works
Yao, Y.-J., Lin, T.-J., Wei, C.-Y., Huang, K., Chen, B.-X., Fang, Y.-T., Chang, H.-J., Fu, Y., Luo, G.-L., Hou, F.-J., et al. (2025) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2025.3540768 · Cited by 8 · see citing works
Memory technology enabling future computing systems
Fantini, P. (2025) · APL Machine Learning · doi:10.1063/5.0253063 · Cited by 8 · see citing works
Ahmadpanah, S. H., Mirabi, M., Sobhanloo, S., Afsharfarnia, P., Fallah, D., & Bayati, M. (2025) · Ad Hoc Networks · doi:10.1016/j.adhoc.2025.104080 · Cited by 8 · see citing works
3D integrated system for advanced intelligent computing
Munde, R., Vaillancourt, N., Chuang, H.-R., Gu, C., Wang, Y., & Islam, R. (2025) · Advances in Physics X · doi:10.1080/23746149.2025.2599301 · Cited by 8 · see citing works
Kwag, J., Choi, S., Kim, D., Lee, J.-Y., Hwang, T., Oh, H.-J., Park, C., & Park, J. (2025) · International Journal of Extreme Manufacturing · doi:10.1088/2631-7990/add7a3 · Cited by 7 · see citing works
Schwarzenberger, A., Fang, D., Kotz, A., Kholeif, H., Wilhelm, C., Eschenbaum, C., Kelany, M., Grünewald, L., Singer, S., Feng, C., et al. (2025) · Optical Fiber Communication Conference · doi:10.1364/ofc.2025.m4k.1 · Cited by 7 · see citing works
OptiPIM: Optimizing Processing-in-Memory Acceleration Using Integer Linear Programming
Liu, J., Zhou, M., Pan, Y., Yang, C.-Y., Josipović, L., & Rosing, T. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731041 · Cited by 7 · see citing works
A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing
Meyer, S. M., Weidel, P., Plank, P., Campos‐Macías, L., Shreshta, S. B., Stratmann, P., Timcheck, J., & Richter, M. (2025) · Neuro Inspired Computational Elements Conference (NICE) · doi:10.1109/nice65350.2025.11065663 · Cited by 7 · see citing works
Zhang, H., Ran, S., Lu, L., Liu, Y., Wang, S., Cai, Y., Guo, Y., Li, Y., Chen, J., & Zhou, L. (2025) · Laser & Photonics Review · doi:10.1002/lpor.202500944 · Cited by 7 · see citing works
Samanta, A., Lee, S.-H., Cheng, C.-Y., Palermo, S. M., & Yoo, S. J. B. (2026) · IEEE Journal of Selected Topics in Quantum Electronics · doi:10.1109/jstqe.2026.3674145 · Cited by 7 · see citing works
Peykani, P., Emrouznejad, A., Ghanidel, S., Javadi-Sisi, I., & Mirjalili, S. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10546-2 · Cited by 8 · see citing works
Strategies of high-accuracy memristor-based analogue computing in memory for artificial intelligence
Jiang, Z., Zhao, H., Tang, J., Lu, Y., Qin, Q., Wang, Z., Hu, R., Yu, R., He, Y., Zhang, J., et al. (2026) · Nature Materials · doi:10.1038/s41563-026-02600-y · Cited by 8 · see citing works
An efficient optical-based binary neural network hardware accelerator for harsh environments
Jahannia, B., Ye, J.-C., Altaleb, S., Patil, C., Heidari, E., & Dalir, H. (2024) · OPTO · doi:10.1117/12.3003287 · Cited by 12 · see citing works
A Lightweight PUF-Based Weights Obfuscation Technique for Secure In-Memory AI Inference
Parrini, L., Kar, A., Hettwer, B., Soliman, T., Chauhan, Y. S., Amrouch, H., & Wehn, N. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3552902 · Cited by 6 · see citing works
Kim, D. E., Sharma, T., & Roy, K. (2025) · IEEE transactions on circuits and systems for artificial intelligence. · doi:10.1109/tcasai.2025.3601975 · Cited by 6 · see citing works
Metareasoning for Edge-Cloud Collaborative LLM Planning for Efficient Autonomous Navigation
Navardi, M., Gao, S., Walczak, M., Camacho, F., & Mohsenin, T. (2025) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3765750 · Cited by 6 · see citing works
A Review of Memory Wall for Neuromorphic Computing
Le, D., Arig, B., Isik, M., Dikmen, I. C., & Karadağ, T. (2025) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi65310.2025.11141074 · Cited by 6 · see citing works
Update Disturbance‐Resilient Analog ReRAM Crossbar Arrays for In‐Memory Deep Learning Accelerators
Choi, W., Stecconi, T., Falcone, D. F., Galetta, M., Clerico, V., Zaccaria, E., Ram, M. S., Porta, A. L., Horst, F., Jubin, D., et al. (2025) · Advanced Science · doi:10.1002/advs.202504578 · Cited by 6 · see citing works
Li, J., Zheng, L., Hong, P., Ye, C., Liu, H., Si, M., Miao, X., & Li, X. (2025) · Advanced Functional Materials · doi:10.1002/adfm.202514194 · Cited by 6 · see citing works
Sub-picojoule-per-bit volitional neuromorphic devices for precise targeting and tracking
Huang, Y., Sun, Q., Dai, F., Wang, F., Zhou, X., Huang, C., Wu, Y., Deng, Y., Luo, L., Li, X. F., et al. (2026) · Nature Communications · doi:10.1038/s41467-025-66295-6 · Cited by 6 · see citing works
Zhu, Y., Liu, Y., Yang, X., Liu, K., Hua, X., Luo, M., Liu, J., Chang, S., Yan, J., Zhang, S., et al. (2026) · Light Science & Applications · doi:10.1038/s41377-026-02209-5 · Cited by 7 · see citing works
Noise-Resilient Photonic Analog Neural Networks
Varri, A., Brückerhoff‐Plückelmann, F., Dijkstra, J., Wendland, D., Bankwitz, R., Agnihotri, A., & Pernice, W. H. P. (2024) · Journal of Lightwave Technology · doi:10.1109/jlt.2024.3433454 · Cited by 6 · see citing works
Bandwidth-Latency-Thermal Co-Optimization of Interconnect-Dominated Many-Core 3D-IC
Das, S., Riedel, S., Naeim, M., Brunion, M., Bertuletti, M., Luca Benini, C., Ryckaert, J., Myers, J., Biswas, D., & Milojevic, D. (2024) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2024.3467148 · Cited by 6 · see citing works
A qua : Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
Kumar, A. V., Antichi, G., & Singh, R. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3715983 · Cited by 5 · see citing works
Neuromorphic Device Based on Material and Device Innovation toward Multimode and Multifunction
Guo, F., Ren, H., Zhang, Y., & Hao, J. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202500477 · Cited by 5 · see citing works
GPU-accelerated simulations of turbulence: Review of current applications and future perspectives
Roccon, A., Amati, G., Brandt, L., Calhoun, D. A., Costa, P., Lu, W., Pirozzoli, S., Richter, D. H., Umair, M., You, D., et al. (2026) · Physical Review Fluids · doi:10.1103/vz9c-bbzm · Cited by 5 · see citing works
Tensor Movement Orchestration in Multi-GPU Training Systems
Lin, S., Chen, Y.-J., Cheng, H.-Y., & Yang, C.-L. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071043 · Cited by 5 · see citing works
Alhussein, O., Wei, M., & Akhavain, A. (2023) · Globecom · doi:10.1109/globecom54140.2023.10437933 · Cited by 5 · see citing works
Fixed-point Encoding and Architecture Exploration for Residue Number Systems
Deng, B., Nadendla, B. S., Suo, K., Xie, Y., & Lo, D. C.-T. (2024) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3664923 · Cited by 5 · see citing works
Design Considerations for Sub-1-V 1T1C FeRAM Memory Circuits
Adnaan, M., Chang, S.-C., Li, H., Liao, Y.-C., Young, I. A., & Naeemi, A. (2024) · IEEE Journal on Exploratory Solid-State Computational Devices and Circuits · doi:10.1109/jxcdc.2024.3488578 · Cited by 5 · see citing works
Ye, W., Gao, H., Zhou, Z., Wang, L., Li, W., Li, Z., Yue, J., Xu, X., Yang, J., Hu, H., et al. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3545866 · Cited by 4 · see citing works
Corsair: An In-Memory Computing Chiplet Architecture for Inference-Time Compute Acceleration
Srivastava, S., Arunkumar, A., Kurella, N., Panda, A., Jain, G., Kamath, P., Wutzke, M., Tiruvur, A., Gupta, M. M., Soloveychik, I., et al. (2025) · IEEE Micro · doi:10.1109/mm.2025.3593444 · Cited by 4 · see citing works
CXL-INTERPLAY: Unraveling and Characterizing CXL Interference in Modern Computer Systems
Mao, S., Luo, J., Li, Y., Zhou, J., Zhang, W., Liu, Z., Ma, T., & Deng, S. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132607 · Cited by 4 · see citing works
Sharma, D. D., Choudhary, S., Onufryk, P., & Pelt, R. (2025) · IEEE Symposium on High-Performance Interconnects (HOTI) · doi:10.1109/hoti66940.2025.00027 · Cited by 4 · see citing works
Wichrowski, M., Rezaee‐Hajidehi, M., Korelc, J., Kronbichler, M., & Stupkiewicz, S. (2025) · International Journal for Numerical Methods in Engineering · doi:10.1002/nme.70166 · Cited by 4 · see citing works
Alinsaif, S. (2026) · Electronics · doi:10.3390/electronics15061179 · Cited by 4 · see citing works
PIMnast: Balanced Data Placement for GEMV Acceleration with Processing-In-Memory
Ibrahim, M. A., Islam, M., & Aga, S. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00137 · Cited by 4 · see citing works
Mario: Near Zero-cost Activation Checkpointing in Pipeline Parallelism
Liu, W., Li, M., Tan, G., & Jia, W. (2025) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3710848.3710878 · Cited by 4 · see citing works
Li, T., Zhong, H., Xu, Y., Narayanan, V., Ni, K., Yang, H., Kämpfe, T., & Li, X. (2024) · doi:10.1145/3676536.3676715 · Cited by 4 · see citing works
A Compact One-Transistor-Multiple-RRAM Characterization Platform
Uhlmann, M., Krysik, M., Wen, J., Frohberg, M., Baroni, A., Reddy, K. D. S., Pérez, E., Ostrovskyy, P., Piotrowski, K., Carta, C., et al. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3555234 · Cited by 4 · see citing works
Overcoming Memory Capacity Wall of GPUs With Heterogeneous Memory Stack
Hong, J., Cho, S., & Kim, G. (2022) · IEEE Computer Architecture Letters · doi:10.1109/lca.2022.3196932 · Cited by 4 · see citing works
Unlocking High Performance with Low-Bit NPUs and CPUs for Highly Optimized HPL-MxP on Cloud Brain II
Xue, W., Yang, K., Liu, Y., Fan, D., Xu, P., & Tian, Y. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00088 · Cited by 3 · see citing works
SparSynergy: Unlocking Flexible and Efficient DNN Acceleration Through Multi-Level Sparsity
Yang, J., Wen, M., Shen, J., Yang, J., Cao, Y., He, J., Tang, M., Chen, Z., & Shi, Y. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992756 · Cited by 3 · see citing works
Cha, J., Lee, M., Kwon, J., Lee, J., & Kwon, Y. (2025) · ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (LCTES) · doi:10.1145/3735452.3735538 · Cited by 3 · see citing works
Li, T., Kämpfe, T., Wang, J., Ni, K., Narayanan, V., Yang, H., & Li, X. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132800 · Cited by 3 · see citing works
Xie, X., Wang, L., Xiao, L., Han, M., Liu, L., Xu, X., Wang, J., Song, Z. Q., & Liao, X. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756113 · Cited by 3 · see citing works
Silicon Photonic Accelerated Memory Pooling for Efficient Compute Resource Allocation
Wu, Z., & Bergman, K. (2025) · IEEE Symposium on High-Performance Interconnects (HOTI) · doi:10.1109/hoti66940.2025.00022 · Cited by 3 · see citing works
Zheng, T., Xu, Y., Mei, C., Chen, Y., Liu, C., Yang, G., Hu, Q., Shi, T., Yuan, Y., Yu, Z., et al. (2025) · Micromachines · doi:10.3390/mi16121304 · Cited by 3 · see citing works
Comparative Study on Device Type Configurations of 2T0C DRAM for Compute-in-Memory Applications
Kong, S., & Shim, W. (2025) · Electronics · doi:10.3390/electronics14234742 · Cited by 3 · see citing works
Panneerselvam, S., Khan, S., Francis, B., Rao, M. S. R., & Manivannan, A. (2025) · ACS Applied Electronic Materials · doi:10.1021/acsaelm.5c02081 · Cited by 3 · see citing works
A survey on memory-efficient transformer-based model training in AI for science
Tian, K., Qiao, L., Liu, B., Jiang, G., Li, S., & Li, D. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50302-6 · Cited by 10 · see citing works
Shen, J., Zhou, Z., Zha, W., Li, Z., Li, W., Wang, B., Zhu, J., Gao, H., Han, Z., Wang, Y. F., et al. (2026) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49663.2026.11409272 · Cited by 3 · see citing works
- Authors
- Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K.
- Affiliations
- International Computer Science Institute (US)
- Snowflake Inc. (United States) (US)
- University of California, Berkeley (US)
- Venue
- IEEE Micro
- Published
- 2024-05
- Volume
- 44
- Issue
- 3
- Pages
- 33-39
- ISSN
- 0272-1732 (ISSN-L)
- 1937-4143
- OpenAlex
- W4393145114
- Cited by
- 369
- References
- 13
- Citations by year
- 2026: 143 · 2025: 166 · 2024: 47 · 2023: 6 · 2022: 6 · 2021: 1
- FWCI
- 110.55
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Micro (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation
- U.S. Department of Energy
- Intel Corporation
- Samsung Advanced Institute of Technology
- Korea Foundation for Advanced Studies
- Samsung
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- memory wall, memory bandwidth, large language models, transformer models, DRAM scaling, interconnect bandwidth, model serving
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
