The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
AI and Memory Wall
Open accessAbstract
The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training LLMs. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0×/2yrs, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. This disparity has made memory, rather than compute, the primary bottleneck in AI applications, particularly in serving. Here, we analyze encoder and decoder Transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These observations are consistent with Gholami et al.'s (2024) findings on AI's restricted memory capacity and its tendency to produce narrative incoherence in longer texts.”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Gholami et al.'s (2024)
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“the relative weakness of zero-shot generation is consistent with research on AI's limitations in sustaining narrative coherence (Gholami et al., 2024)”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Gholami et al., 2024
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Gholami et al. (2024) identify restricted memory capacity as one of the primary challenges of AI-assisted fiction writing: as a story progresses, the model tends to lose track of earlier plot points and character details, which result in narrative inconsistencies.”
From How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction (2026) · cites this paper as Gholami et al. (2024)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“We examine two different variations of the Trans-former architecture: the encoder architecture (e.g., BERT [7]), which concurrently processes all tokens, and the decoder architecture (e.g., GPT [3, 33]), which runs auto-regressively to process and generate one token at each iteration.”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“We examine two different variations of the Trans-former architecture: the encoder architecture (e.g., BERT [7]), which concurrently processes all tokens, and the decoder architecture (e.g., GPT [3, 33]), which runs auto-regressively to process and generate one token at each iteration.”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Deploying recent SOTA models [4, 5, 16, 37] such as GPT-3 [3] or large recommendation systems [29] is quite challenging, as they require distributed-memory deployment for inference.”
InfluentialWilliams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798 · cited 1×
“This is due to the higher memory operations and lower arithmetic intensity of matrix-vector operations inherent in the auto-regressive inference of GPT (see Figure 3c).”
Wulf, W. A., & McKee, S. A. (1995) · ACM SIGARCH Computer Architecture News · doi:10.1145/216585.216588 · Cited by 1,625 · cited 1×
“They predicted that the diverging speed of improvement of how fast computations can be performed versus how fast data can be fetched is going to create a “memory wall” issue [25, 42].”
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219
Self-citationTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 1×
“In this work, we re-examined this trend by studying more recent data, with a particular focus on hardware used to train AI models, as well as the characteristics of the computations used for training/serving them.”
Method(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · cited 1×
“A promising line of work for that is the Zero framework from Microsoft, which showed how one can train 8 × bigger models with the same memory capacity by removing/sharding redundant optimization state variables [2, 34].”
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
Yao, Z., Gholami, A., Shen, S., Mustafa, M., Keutzer, K., & Mahoney, M. W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17275 · Cited by 257
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Items 201–300 of 371
Giardino, M. J., Gupta, S., Humbel, L., Mueller, R., & Nag, A. (2025) · doi:10.1145/3723851.3723856
DOME: Dynamic Optimization of GPU Memory for Training DNN Models with Limited Resources
Ma, Z., Jian, S., & Tan, Y. (2024) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn63567.2024.00102
Algorithm-Hardware Co-Design for Ultra-Low-Power Large Language Models
Abreu, S., & Eshraghian, J. K. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11043353
Analysis of Model Parallelism for AI Applications on a 64-core RV64 Server CPU
Malenza, G., Garcia, A. M., Birke, R., Benini, L., & Aldinucci, M. (2025) · International Journal of Parallel Programming · doi:10.1007/s10766-025-00802-6 · Cited by 1
“This contrasts with the more traditional data parallelism that leverages multiple inputs to distribute the workload across cores in parallel.”
Padiyal, A., Wang, T., Misawa, N., Matsui, C., & Takeuchi, K. (2025) · IEEE Electron Devices Technology & Manufacturing Conference (EDTM) · doi:10.1109/edtm61175.2025.11040348
SOT-MRAM Devices in Memory Hierarchy for Next-Generation Computing Systems
Kallinatha, H. D., & Talawar, B. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96473-2_25
Cordial: Cross-row Failure Prediction Method Based on Bank-level Error Locality for HBMs
Gu, W., Gu, J., Zhong, R., Zhang, W., Li, M., & Lyu, M. R.-T. (2025) · IEEE/IFIP International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S) · doi:10.1109/dsn-s65789.2025.00045
“However, the growing exponent for GPUs or NPUs is substantially larger than that for dynamic random-access memories (DRAMs) [3].”
Nakamura, F., Suzuki, K., Noriki, A., Suda, S., Kurosu, T., & Amano, T. (2025) · doi:10.1109/oip65843.2025.11081574
InterconnectLens: Enhancing Observability of Data Transfers in GPU Clusters
Eguchi, K., Nakamura, R., Kuga, Y., & Taura, K. (2025) · Practice and Experience in Advanced Research Computing · doi:10.1145/3708035.3736055
BiGRU with Nadam Optimization for Modeling Indonesia's Non-Oil and Gas Trade Values
Fadhilah, A. Z., Rochayani, M. Y., & Warsito, B. (2025) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera66156.2025.11087346
“The batch size values were selected based on findings from [18].”
Multilayer Light Processor for AI Compute and Networking
Yıldırım, M., Dinç, N. U., Oğuz, İ., Psaltis, D., & Moser, C. (2025) · OptoElectronics and Communications Conference and International Conference on Photonics in Switching and Computing (OECC/PSC) · doi:10.23919/oecc/psc62146.2025.11110024
“Current electronic systems suffer from low data transfer rates compared to processing speeds, leading to underutilization of computational resources and hindering further scaling[3, 4].”
Memory Access Priority Circuit Design and Veri cation in Heterogeneous Multi-core SystemsPreprint
Gao, Y. J., Wei, J., Zheng, Z. P., Liu, G., Zhu, J., & Wang, L. (2025) · Research Square · doi:10.21203/rs.3.rs-7271146/v1
SortingHat: System Topology-aware Scheduling of Deep Neural Network Models on Multi-GPU Systems
Namkoong, S., Park, T., Jung, K., Kim, J., & Park, Y. (2025) · International Conference on Supercomputing · doi:10.1145/3721145.3728490 · Cited by 1
SHARP: SHARing-Aware Cache Writeback byPass
Joshi, D., Bagchi, A., & Panda, P. R. (2025) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3760746
Simulating LLM training workloads for heterogeneous compute and network infrastructure
Kumar, S., Temura, A., Sharma, N., Singh, R., Dadhania, M., Tammana, P., Burla, S., Kamaluddin, A. M., & Shah, R. (2025) · doi:10.1145/3748273.3749212
Xiang, 张. Z., Zhang, H., Wenlin, 崔. C., Anle, 申. S., Zhijun, 梁. L., Chong, 李. L., Tao, 方. F., Jingwei, 李. L., Jiayi, 欧. O., Xinxiang, 牛. N., et al. (2025) · Acta Optica Sinica · doi:10.3788/aos251195
Liu, Z., Zhang, Y., Zhang, Z., Yang, Y. T., Zhang, Z., Wang, X., Chen, J., Li, X., Si, X., & Yang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4354-y
Caihua, 张. Z., Zheng, 黄. H., Conghe, 王. W., Shukai, 吴. W., Tuo, 李. L., Kejian, 朱. Z., & Hongwei, 陈. C. (2025) · Laser & Optoelectronics Progress · doi:10.3788/lop251303
Memory-efficient Model Growth for Finetuning Automatic Speech Recognition Models
Vanderreydt, G., Thienpondt, J., & Demuynck, K. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11166050
“The diverging speed of improvement of how fast computations can be performed versus how fast data can be fetched is known as the ”Memory Wall” [3]–[6].”
Physical data embedding for memory efficient AI
MacPhee, C., Zhou, Y., & Jalali, B. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae0f37
“In the case of AI data centers, as model parameter counts have increased from millions to hundreds of billions, model memory requirements have severely outpaced on-chip capacity [2].”
LG-Prefetcher: a prefetcher combining local spatial and global performance information for CXL-SSD
Li, S., Wang, C., Yang, Z., Zhang, Y., Li, L., Ma, C., Cui, X., Li, L., & Liu, J. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07910-7
Liu, T., Cheng, H., Erqi, E., Li, N., Liao, H., Huang, B., & Guo, J. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3772287
Xu, P., He, J., Kennes, K., Patil, A., Dvoretskii, A., Podpod, A., Lepage, G., Golshani, N., Magdziak, R., Bipul, S., et al. (2025) · IEEE Transactions on Components Packaging and Manufacturing Technology · doi:10.1109/tcpmt.2025.3628179
“TSV: through-silicon via. crucial for the interconnections between compute processor units (XPUs) in multinode AI / ML clusters, which contribute significantly to energy consumption in computing tasks [1], [2], [3], [4].”
Full field lithography as an answer to heterogeneous packaging challenges
Hennemeyer, M., & Erdt, L. (2025) · SPIE Photomask Technology Conference · doi:10.1117/12.3073645
“This is caused by the extremely high data transfer rate demand in AI computing [1], [2].”
Local feedback and dynamic adjustment best offset prefetcher in CXL-SSD
Li, S., Yang, Z., Wang, C., Zhang, Y., Dou, W., Cui, X., & Liu, J. (2025) · Cluster Computing · doi:10.1007/s10586-025-05782-3
“Based on Gholami’s [ 3 ] data, more recent model updates have been added [4–8 ].”
Vertical vs. Horizontal Integration in HBM and Market-Implied Valuation: A Text-Mining Study
Yang, H. J., & Kim, C. G. (2025) · Applied Sciences · doi:10.3390/app152212127
Memory access priority circuit design and verification in heterogeneous multi-core systems
Gao, Y. J., Wei, J., Zheng, Z. P., Liu, G., Zhu, J., & Wang, L. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-08074-0
DABench-LLM: Standardized and In-Depth Benchmarking of Post-Moore Dataflow AI Accelerators for LLMs
Hu, Z., Zhong, Z., Zheng, W., Ye, Z., Tan, X., Zhang, X., Xie, Z. Y., Kettimuthu, R., & Yu, X. (2025) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc66894.2025.00020
Improving the Performance of Out-of-Core LLM Inference Using Heterogeneous Host Memory
Gupta, S., & Dwarkadas, S. (2025) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc66894.2025.00035
“Given the exponential growth in LLM model sizes, storing all model weights on accelerator memory has been increasingly challenging [1].”
InfluentialKou, H., Xia, C., Yang, J., Li, L., Cai, H., Si, X., & Liu, B. (2025) · doi:10.1109/iccad66269.2025.11240702
“GPUs and CPUs face significant challenges due to the memory wall problem [4], particularly during the decoding phase, which accounts for over 80% of inference time.”
Baseband-Free End-to-End Communication System Based on Diffractive Deep Neural Network
Teng, X., Tang, W., Li, X., & Jin, S. (2025) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2025.3632886 · Cited by 1
“1) D 2 NN: As Moore’s law approaches its physical limits, the advancement of integrated circuit technology is expected to slow down in the foreseeable future, making it increasingly challenging to meet the rapidly expanding computational demands [17].”
Wang, C., Zhang, Y., Liu, W., & Gao, S. S. (2025) · SiPS ... design and implementation - IEEE Workshop on Signal Processing Systems · doi:10.1109/sips66314.2025.11261269
“Traditional DSPs adopt the Von Neumann architecture with a separation of storage and computing [9], which faces the so-called “memory wall” challenges [10], where limited data transfer bandwidth hampers overall computational throughput.”
Swap on Sky: A Skyrmion-Based In-Memory Swapping Architecture to Break Memory Wall
Chen, S. H., Huang, Y., Liao, Y.-Y., & Chen, T. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3779225
Kamal, M., & Pedram, M. (2025) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped65674.2025.11261758
“For example, studies reveal that the size of transformers, the backbone of large language models (LLMs), increases approximately 750X every two years [1].”
Dynamic Offloading Optimization for Multi-PIM LLM Inference
Kang, J. Y., Ko, J. H., Hwang, T., & Choi, K. H. (2025) · IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) · doi:10.1109/icce-asia67487.2025.11263662
“However, this ambition confronts a fundamental obstacle: achieving state-of-the-art performance with LLMs necessitates scaling model sizes to unprecedented levels, which introduces the memory wall problem [3].”
A Dimensional Order Search Based Mapping Algorithm for Deploying DNNs Onto Many-Core Neuromorphic IC
Wang, S., Lu, R., Xiao, B., Wang, Z., Guo, Y., Sun, X., & Yu, W. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3640508
“The growing scale and complexity of modern DNNs—exemplified by architectures such as transformers [2]—have led to an exponential rise in computational demands [3].”
Kim, D.-U., Moon, S., Song, J., Lee, J. C., Yoo, S.-H., Lee, K. K., & Lee, J. K. (2025) · European Conference on Optical Communication · doi:10.1109/ecoc66593.2025.11263044
“DTM supports inter-rack optical communication but requires repeated data copying and buffering to move data from local memory to remote memory, and requires the use of kernel software [3,4].”
Wrap-Around Word-Line DRAM Cell Transistor Enabling Enhanced Read/Write Speed
Sohn, D. K., Lee, M.-K., Bang, D.-E., Park, H., Yun, E.-C., Kang, S., Kim, M.-W., Jeon, H., & Park, J.-Y. (2025) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2025.3635561
“The rapid growth in model size has highlighted the limitations of main memory bandwidth, which has become a system-level bottleneck [14], [15].”
Jang, H., Ha, D., Lee, H., & Ro, W. W. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103650
Energy Consumption Evaluation for Large Language Models
Stoykova, S., Shakev, N. G., Popov, V., & Ahmed, S. (2025) · International Conference on Energy Efficiency and Agricultural Engineering (EE&AE) · doi:10.1109/eeae65901.2025.11273449
Хайдуков, Д. И., & Алексеев, А. А. (2025) · Nanoindustry Russia · doi:10.22184/1993-8578.2025.18.11s.1298.1304
Weighted Coding Scheme for Noise Reduction in Silicon Interposer of HBM
Jeon, S., Jang, S., Shin, K., Lee, D.-K., Chi, H., Shin, W., Pyo, C., Kim, J., & Jeon, D. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3629742
“H IGH-BANDWIDTH memory (HBM) has emerged as a critical enabler for memory-intensive computing work-loads, particularly in data processing for large-scale deep neural networks [1], [2].”
Learning dynamics in memristor-based equilibrium propagation
Döll, M., Müller, A., & Ulmann, B. (2026) · International Journal of Parallel Emergent and Distributed Systems · doi:10.1080/17445760.2026.2616770
“These challenges are particularly critical for neural networks due to the immense scale at which vector-matrix multiplications (VMMs), providing the networks’ computational backbone, are performed during both training and inference [5,6].”
Low-Power High-Density In-Memory Computing Circuits Based on MRAM
Zhu, S., Deng, E., Shen, Z., Zhang, Y., You, J., Hu, Q., & Liu, W. (2025) · IEEE Nanotechnology Magazine · doi:10.1109/mnano.2025.3607062
“However, the data transmission ability of current memories can not completely satisfy the requirements of artificial intelligence applications [2].”
Podasca, A., & Das, A. (2025) · doi:10.1109/iccad66269.2025.11240848
“Crossbars enable compute-in-memory operations that eliminate the costly data movement between separate memory and processing units, addressing the memory wall problem that constrains conventional computing systems [11]–[13].”
Slot Waveguide Assisted High Extinction Ratio Add-Drop Micro-Ring Modulator
Zhao, S., Zhang, Y., & Wang, J. (2025) · Journal of Lightwave Technology · doi:10.1109/jlt.2025.3642828
“Studies indicate that global computing power demand grows at a rate of approximately 750 times every two years, far outpacing the 3-fold improvement in single-chipperformanceover thesameperiod[1].”
Wafer-level packaging integration of LPDDR5X-signal integrity design and simulation exploration
Tang, W., Ji, Y., Zhou, L., Bai, Y., Zhu, Y., Yang, C., Wang, G., & Wang, C. (2025) · Microelectronics International · doi:10.1108/mi-09-2025-0204
TROOP: At-the-Roofline Performance for Vector Processors on Low Operational Intensity Workloads
Purayil, N. K., Shen, D., Perotti, M., & Benini, L. (2025) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd65941.2025.00091
“Recent advances in Artificial Intelligence (AI) have led to a significant increase in computation and memory requirements [1].”
Sparse weight mapping and computation reuse strategy for scalable photonic matrix multiplication
Liao, C., Zhang, H., Hao, X., Tian, Y., Xiong, G., Feixiang, L., Song, B., & Li, Q. (2026) · Optics Express · doi:10.1364/oe.584125
Performance and Cost Benefits of Workload-Aware Heterogeneous Deployment in Modern Applications
Krishnan, A. M., Inamdar, S., Nagda, Y., Pasumarti, V., Nambiar, M. K., & Singhal, R. (2025) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1109/aimlsystems67835.2025.11330387
Photonic Tensor Cores: A Comprehensive Review on System Architecture and Device Design
Dhrubo, M. N. S. (2026) · IEEE Access · doi:10.1109/access.2026.3655405
Yang, Z., Yang, Y., Wang, B., Tao, Y., Zelun, P., Cai, L., Zhang, T., Yan, L., Li, X., & Yang, Y. (2026) · Advanced Science · doi:10.1002/advs.202521815
“In conventional computing systems, frequent data transfers between memory and processing units further degrade energy efficiency, posing a critical bottleneck for scaling future models [ 16, 17 ].”
Low-Loss, Multi-Reticle-Stitched SiN Waveguides for 300 mm Wafer-Level Optical Interconnects
Xu, P., Marchese, C., Lepage, G., Golshani, N., Eenaeme, R. V., Mingardi, A., Ongeval, J. V., Magdziak, R., Halipre, L., Trivković, D., et al. (2026) · Photonics · doi:10.3390/photonics13010100
Silicon Photonics Accelerator and Processor for AI Using Integrated Coherent Technologies
Zhu, Y., Hua, X., Liu, Y., Yang, X., Jiang, X., Zhang, H., Chen, D., & Xiao, X. (2025) · Asia Communications and Photonics Conference · doi:10.1109/acp66871.2025.11349750
Electron Device Challenges to Enable Individual Centric Edge AI
Chidambaram, P. R. (2025) · Technical digest - International Electron Devices Meeting/Technical digest · doi:10.1109/iedm50572.2025.11353580
A fully hardware-managed scheduling architecture for AI accelerators
Cheng, L., Yang, L., Shao, J., Gu, X., Qian, R., Zhang, X., Li, D., Wang, H., & Xia, X. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00513-z
Zhao, Q., Li, Q., He, J., Yang, Y., Pei, Q., Feng, H., Wang, Q., & Huo, Z. (2026) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2026.3661894
“I N RECENT years, the rapid growth of artificial intelligence (AI) and edge systems has fueled demand for higher capacity and faster NAND flash [1], [2].”
Kim, J., Bang, S. J., Kim, M. S., Koo, M., & Kim, Y. C. (2026) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2026.3659834
“Specifically, the “memory wall” phenomenon, resulting from the speed disparity between the processing and memory units, and the considerable energy consumption due to data movement between these units, are major concerns [1].”
OpenCXD: An Open Real-Device-Guided Hybrid Evaluation Framework for CXL-SSDs
Chung, H., Park, J., Noh, T., Ahn, S., Kim, K., Zhao, M., & Kim, Y. (2025) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots67699.2025.11283378
“This widening gap between demand and capacity has resurrected the infamous memory wall [4], in which memory bandwidth and size become critical bottlenecks to system performance.”
Copper–Copper Thermal Compression Bonding Scheme for Heterogeneous Wafer-Scale Integration
Sahoo, K., Sabbir, G., Harish, V., Liu, J.-H., & Iyer, S. S. (2026) · IEEE Transactions on Components Packaging and Manufacturing Technology · doi:10.1109/tcpmt.2026.3665683
Design and Analysis of Reconfigurable FP16INT8 Dual-Mode Multiplier on FPGA
Oh, U., Kim, D. H., Jang, H., Hong, S., & Jung, S. (2026) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.1109/iceic69189.2026.11385903
“With the emergence of large language models (LLMs), the number of model parameters has increased dramatically, resulting in severe memory bottlenecks during both training and inference [1].”
Unified Approach to Scale AI-Mobile NAND Packages
Monroe, M., Mamun, M. A. Z., Yu, A., & Kumar, G. (2025) · IEEE Electronics Packaging Technology Conference (EPTC) · doi:10.1109/eptc67330.2025.11392635
“The UFS controller may be placed either side-by-side with the NAND stack in a SiP configuration or directly beneath the stack in a 2.5D-like arrangement [1].”
When Smaller Models Run Slower: Hardware Efficiency Bottlenecks in VLM Inference on Unified Memory
Harsh, F., Jarsania, J., Nagwani, A., Behl, S., Singh, M., & Gupta, A. K. (2026) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc67433.2026.11393753
“Gholami et al. [10] quantified the memory wall where bandwidth scaling lags compute growth substantially.”
InfluentialYuan, Y., Sun, Z., Yang, Y., Wang, X., Li, X., Ma, C., Chen, Q., Tang, M., Wei, X., Wu, H., et al. (2026) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2026.3668132
“…network models become larger and more complex, such as large language model (LLM), which require rapid and e ffi cient processing of large volumes of parallel data, exposing the inherent limitations of the traditional von Neumann architecture when tackling such challenges [1], [2], [3], [4], [5].”
Lin, X., Javadi, R., Bose, B., & Anand, T. (2026) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49663.2026.11409172
CoCoTree: A Computation-Capable Architecture for Collective Communication in Scalable PIM
Shi, S., Yang, Q., Yang, F., Huang, Y., Zhuo, Y., Li, Z., Sun, N., & Li, X. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408595
“This fundamental limitation stems from the massive data † Both authors contributed equally to this research ∗ Corresponding author movement bottlenecks between processors and memory, which constrains the overall system performance [22], [32].”
Han, R., Chai, J., Hu, T., Yang, J., Luo, T., Shao, X., Liao, M., Sun, X., Du, Y., Liu, W. L., et al. (2026) · IEEE Electron Device Letters · doi:10.1109/led.2026.3665632
“T HE rapid growth of semiconductor technology and the rise of data-intensive applications have driven strong demand for non-volatile memory (NVM) with high density, high speed, and robust reliability [1], [2].”
Secure AI Runtime System Based on the M3 Platform
Hasler, M., Raitza, M., Pauls, F., & Fettweis, G. P. (2025) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf67917.2025.11443879
“However, the batch and massively parallel processing nature of AI applica tions demand a constantly high bandwidth of data, mainly from the memory to the processing units [19].”
Seo, S., Kim, S., Heo, G., Kim, J., Chi, H., Park, H., Ryu, G., Yun, J., Choi, W., & Jeong, Y.-U. (2026) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2026.3678854
“However, the memory wall problem, referring to the bandwidth gap between processors and memory, has emerged as a serious bottleneck that hinders this development [1].”
Self-citationYuan, Y., Zhang, B., Yang, Y., Luo, Y., Chen, Q., Wang, H., Liu, Q., Chen, Z., Wu, H., Yue, J., et al. (2026) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2026.3679560
“To address this challenge, computing-in-memory (CIM) techniques have been proposed as a promising solution by tightly integrating computing units with memory cells, significantly improving the energy e ffi ciency of AI hardware [1], [2], [3], [4].”
Efficient neural network acceleration using redundant residue number systems
Mousavi, S., Rahmati, D., Sadr, A., Gorgin, S., & Kim, J. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08330-x
Technology solutions targeting the performance of gen-AI inference in resource constrained platforms
Kundu, J., Klein, J., Patel, A., & Biswas, D. (2026) · IEEE International Conference on IC Design and Technology (ICICDT) · doi:10.1109/icicdt69693.2026.11669195
“Now, due to the memory wall problem [8] data movement becomes the major bottleneck in the generation phase of inference – this nature of the workload is termed as memory-boundedness.”
HASE: Hardware-Aware Scheduling for Inference Tasks in Heterogeneous GPU Clusters
Chen, Y., Jiang, C., Wu, C., Wang, Y., Ye, Q., Yan, L., Niu, J., Liu, J. J., & Lao, L. (2026) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2026.3685862
“The performance bottleneck is the large-scale, continuous data flow be-tween the memory and computation units, and this method is a standard practice for measuring memory subsystem throughput [29].”
Reducing the GPU Memory Bottleneck with Lossless Compression for ML
Kamath, A. K., Krishnamurthy, A., Canini, M., & Peter, S. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3803595
“The threads then insert their chunk into the output array independently [line 32], maximizing parallelism.”
Balancing Compute in LLM Inference: Model Selection, Quantization, and Test-Time Scaling
Šubic, T., & Cozzini, S. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807631
Device-to-logic variability propagation in RRAM-based logic-in-memory architectures
Bende, A., Singh, S., Jha, C. K., Storelli, D., Nielinger, D., Drechsler, R., Dittmann, R., Menzel, S., Merchant, F., & Rana, V. (2026) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/ae65d5
A survey on edge multimodal large models: Compression, inference acceleration, and applications
Chen, S., Shu, Y., Wang, C., & Chen, J. (2026) · National Science Open · doi:10.1360/nso/20260016
Fisher-Based Layer-Wise Adaptive Sparsification for Efficient Pruning of Large Language Models
Sun, C., Yu, H., & Li, X. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3689307
“Thesemodels,typicallywithbillionsofparameters distributed across numerous Transformer layers, require signifi-cant computational resources for both training and inference [4].”
A Booth-Based Digital Compute-in-Memory Macro With Bitwise-Efficient Multiphase Accumulation
Jiang, Y., Wang, X., You, H., Li, Q., & Qiao, S. (2026) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2026.3687178
“The rise of data-centric applications has highlighted the “memory wall” problem, where high latency and energy consumption in data transfer become a critical bottleneck for von Neumann architectures [5].”
Innovations in Process Technology Enabling Next Generation Memory Solutions
Lee, R. T. P., Lynch, S., Grasso, J., Aoki, S., Basu, T., O’Meara, D., Maguire, J., Qin, Y., Sweet, E., Soedibyo, R., et al. (2026) · IEEE Electron Devices Technology & Manufacturing Conference (EDTM) · doi:10.1109/edtm65772.2026.11497901
“Figure 1 adapted from Gholami [2] and incorporating new estimates [3] – [5], clearly shows the widening gap in compute and memory performance over the last decade.”
Differential Resistance Lowering Effect in High-Speed Ge Photodetectors During Nanosecond ESD Events
Lin, P.-Y., Simicic, M., Hsieh, P.-Y., Chen, W.-C., Vermeersch, B., Coughlan, C., Berciano, M., Vaskasi, J. R., Bipul, S., Chakrabarti, M., et al. (2026) · IEEE International Reliability Physics Symposium proceedings · doi:10.1109/irps61424.2026.11499300
“Over the past decade, processor computational capability has scaled rapidly, accelerated by architectural advances, whereas electrical I/O bandwidth has increased at a significantly slower rate [2].”
Evolving Memory-Aware Schedules for Transformer Inference on Systolic Array Accelerators
Sedlak, D., Klhufek, J., & Mrázek, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-23604-3_20
Gate/Channel Interface Co-Design for Highly Reliable 3D FE NAND
Han, R., Yang, J., Hu, T., Bai, M., Ding, Y., Shao, X., Liu, W., Chai, J., Xu, H., Wang, X., et al. (2026) · IEEE International Reliability Physics Symposium proceedings · doi:10.1109/irps61424.2026.11499247
“The rapidly growing memory demand driven by artificial intelligence places substantial pressure on advanced storage technologies [1].”
CHARIOT: A Communication-Aware Exploration Framework for 2.5D/3D Silicon Interposer Design
Zeng, Y., Jiao, B., Jia, S., Xiong, X., Wang, C., Tian, H., Zhu, H., Liu, Q., & Chen, C. (2026) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3815192
On-chip large-scale all-optical interconnect for ultra-low-latency deep neural network inference
Tao, Z., Zhou, Y., Yu, W., Chang, H., Wu, H., Cheng, A., Hong, Q., Tang, Y., Bian, X., Meng, L., et al. (2026) · National Science Review · doi:10.1093/nsr/nwag282
“Despite parallel processors such as GPUs can prominently accelerate computation, the compute–memory separated Von Neumann architecture still leads to considerable latency due to the limited throughput between the processor and memory, known as the memory wall [21 ].”
Zhang, Y., Wang, J., & Bao, X. (2026) · Electronics · doi:10.3390/electronics15112255
Lee, K., Yu, S., & Datta, S. (2026) · IEEE International Memory Workshop (IMW) · doi:10.1109/imw68301.2026.11532884
Vishnumurthy, P., Ganser, R., Kersch, A., Mikolajick, T., Schroeder, U. P., Kim, D. H., Jung, C. H., & Lim, H. (2026) · IEEE International Memory Workshop (IMW) · doi:10.1109/imw68301.2026.11532794
“Dynamic Random-Access Memory (DRAM) remains central to this challenge, providing the high-speed data access necessary for real-time AI learning and decision-making [1], [2].”
Memory Technology: Enabling the AI Revolution
Ramaswamy, N., Calderoni, A., Mocuta, D., Daycock, D., Karda, K., Singh, A., & Sreeramaneni, R. (2026) · IEEE International Memory Workshop (IMW) · doi:10.1109/imw68301.2026.11532912
“While transistor scaling and architectural innovation have driven exponential growth in peak compute throughput, memory and interconnect bandwidth have improved at slower rates [1].”
3D Charge Coupled Device for High-Density and Cost-Efficient Buffer Memory
Kishore, R., Mahato, S. B., Izmailov, R., Jeong, Y., Zheng, W., Li, J., Subhechha, S., & Rosmeulen, M. (2026) · IEEE International Memory Workshop (IMW) · doi:10.1109/imw68301.2026.11532839
“The rapid expansion of data-centric computing, particularly in artificial intelligence (AI) and machine learning (ML), has intensified the performance disparity between processors and memory, commonly known as the memory wall [1].”
LPGSim: A Lightweight Parallel GPU Simulator Maximizing Speed with Trustworthy Simulation
Nam, H., Lee, J. H., Kim, Y., Zhang, M., Kim, J.-G., & Burgstaller, B. (2026) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3805642
Torrent : A Distributed DMA for Efficient and Flexible Point-to-Multipoint Data Movement
Deng, Y., Kong, F., Yi, X., Antonio, R., & Verhelst, M. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539609
Li, K.-S., Huang, M.-K., Chen, H.-M., Wu, W.-F., Wang, Y. S., Tseng, Y., & Su, C.-J. (2026) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2026.3698080
“Traditional von Neumann architectures su ff er from the memory bottleneck, limiting performance and energy e ffi ciency [1], [2], [3].”
IterQuant: Iterative Quantization Framework for Mixed-Precision LLM Compression
Jeong, H., Kim, J., Kwon, H., Lee, J., & Lee, Y. K. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539489
“The growth has far outpaced hardware improvements: while the parameter size of large transformer models has been increasing at a rate of about 410 × every two years, GPU memory capacity has only grown about 2 × over the same period [1].”
Geng, X., Liu, S., Wang, H., Han, J., & Jiang, H. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539270
Sotiropoulos, G., Frombach, F., Hoefer, J., Harbaum, T., Becker, J., Thorøe, H. I., Meyers, V., Tahoori, M. B., Demirdag, Z., Sikal, M. B., et al. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539555
“Computational acceleration alone is not sufficient since many of today’s AI workloads are memory bound due to their low operational intensity and the small increase in memory bandwidth compared to computing power over the years [8].”
Nakamura, F., Suda, S., Kurosu, T., Noriki, A., & Amano, T. (2026) · International Conference on Electronics Packaging (ICEP) · doi:10.23919/icep-hbs69241.2026.11550706
Zhao, H.-B., Huang, Z. J., Ying, X.-B., Xie, C., Huang, Z.-Y., Zhu, H.-Y., Jiang, R., Zhang, D. Y., & Lü, H.-L. (2026) · Applied Physics Letters · doi:10.1063/5.0333350
Huang, H.-C., Mitarai, Y., Kawano, M., Fukushima, T., Hamada, M., & Kosuge, A. (2026) · IEEE Electronic Components and Technology Conference (ECTC) · doi:10.1109/ectc51846.2026.00114
“As the parameter counts of transformer-based models surpass the trillion mark, the performance of AI accelerators is increasingly bottle-necked by the ”Memory Wall” rather than compute capability [1].”
Metadata
- Authors
- Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K.
- Affiliations
- International Computer Science Institute (US)
- Snowflake Inc. (United States) (US)
- University of California, Berkeley (US)
- Venue
- IEEE Micro
- Published
- 2024-05
- Volume
- 44
- Issue
- 3
- Pages
- 33-39
- ISSN
- 0272-1732 (ISSN-L)
- 1937-4143
- OpenAlex
- W4393145114
- Cited by
- 369
- References
- 13
- Citations by year
- 2026: 143 · 2025: 166 · 2024: 47
- FWCI
- 110.55
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Micro (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation
- U.S. Department of Energy
- Intel Corporation
- Samsung Advanced Institute of Technology
- Korea Foundation for Advanced Studies
- Samsung
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- memory wall, memory bandwidth, large language models, transformer models, DRAM scaling, interconnect bandwidth, model serving
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
