A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Open accessThe DOI matches its Crossref record; 215 of the 215 references it cites were found in a registry or an index.
Abstract
The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, the collaborative hardware and software design is also necessary to further break the bottleneck for training and inference acceleration [153].”
From A Survey of Large Language Models (2026) · cites this paper as [153]
In the citing paper
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [118], [119].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [119]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.”
Self-citationVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The backbone of modern LLMs is the transformer architecture, which has revolutionized NLP by addressing the limitations of previous recurrent and convolutional models.”
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“Encoder-decoder and Encoder-only models are considered BERT-style models [15], while Decoder-only models are termed GPT-style models [1].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Encoder-decoder and Encoder-only models are considered BERT-style models [15], while Decoder-only models are termed GPT-style models [1].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 6×
“The landscape changed significantly after 2021 with the introduction of transformative models like GPT-3 [4], which led to a surge in the adoption of decoder-only architectures.”
InfluentialLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“The construction of these multi-modal LLMs follows a well-established recipe.”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“LoRA, or Low-Rank Adaptation, is a technique designed to fine-tune large pre-trained models in a pa-rameter-efficient manner [36].”
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×
“In 2020, OpenAI’s introduction of the “scaling law” [17] highlighted that model performance is strongly correlated with size.”
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · cited 1×
“1) MoE: Mixture-of-experts (MoE) was first proposed in [96] and [97] by Jordan and Jacobs more than three decades ago.”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“However, these studies mainly targeted much smaller language models, like GNMT [69].”
Self-citationHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1×
“Weight pruning [166], [167], [168], [169], [170], [171], [172], [173], [174] reduces the number of parameters by removing less important weights.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency.”
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00286 · Cited by 4,114
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“QLo-RA [39], a highly memory-efficient technique that first quantizes a pre-trained model into a noval 4-bit Normal-Float data type with an innovative method called Double Quantization, and then back-propagate the gradients through the quantized weights with Paged Optimizers, can fine-tune a LLaMA…”
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“In scenarios where only the continuous prompts are tuned—keeping the PLM’s parameters unchanged—the training remains efficient while achieving comparable performance to full model tuning.”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 1×
“In scenarios where only the continuous prompts are tuned—keeping the PLM’s parameters unchanged—the training remains efficient while achieving comparable performance to full model tuning.”
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,673
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2×
“As LLMs become increasingly integrated into a wide range of applications—from chat-bots [2], [4], [5], [6] and virtual assistants [5], [7] to complex decision-making systems—their impact on research and industry becomes increasingly profound.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 2×
“Recent advancements have shown that prompts need not be confined to natural language forms but can also be optimized in a continuous space using gradient descent, enhancing their efficiency [45], [46], [47], [48], [49], [50].”
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · cited 1×
“In global attention, certain tokens attend to all other tokens, regardless of their position.”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · cited 1×
“Conversely, Zhao et al. [16] define BERT-style models as “small-scale language models.””
BackgroundSelf-citationA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090 · cited 1×
“Conversely, Zhao et al. [16] define BERT-style models as “small-scale language models.””
BackgroundSelf-citationMalkov, Y. A., & Yashunin, D. A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889473 · Cited by 1,904 · cited 1×
“Currently, approximate nearest neighbor search (ANNS), which retrieves the approximate nearest neighbors of a given query in the high-dimensional and large scale vector database, has been widely used as a RAG technique.”
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 4×
“Inspired by CPU virtual memory systems, vLLM [78] introduces PagedAttention to segment the KV cache into manageable blocks, enhancing memory management with 24 × higher throughput than HuggingFace Transformers [34].”
InfluentialSelf-citationEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802 · cited 4×
“Inspired by CPU virtual memory systems, vLLM [78] introduces PagedAttention to segment the KV cache into manageable blocks, enhancing memory management with 24 × higher throughput than HuggingFace Transformers [34].”
InfluentialSelf-citationMamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579 · cited 4×
“…architecture by combining transformers into SSMs, where [141] use the SSM layer to replace the FFN layer in transformers and Jamba [142] add four transformer layers to SSMs. Mamaba2 [140] proposes a new architecture based on State-Space Du-ality that achieves 2-8 × speed up compared to Mamba [139].”
InfluentialMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 1×
“Despite this, manually crafted prompts often exhibit a performance discrepancy compared to fine-tuned models, as noted by multiple studies [4], [42], [43], [44].”
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 1×
“On the other hand, PTQ [148], [151], [152] converts a floating-point model to a lower-bit model without requiring training data, making it particularly suitable for large-scale language models.”
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · cited 1×
“Longformer [200] utilizes a combination of local attention and global attention to specific tokens, while BigBird [201] further adds random attention on top of local and global attention, demonstrating its ability to encompass all sequence-to-sequence functions.”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 2×
“To optimize attention computation, FlashAttention [71], [72] integrates the entire process into a single operator, achieving 3 × training speed up on GPT-2 model.”
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180
Self-citationMicikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017) · arXiv · doi:10.48550/arxiv.1710.03740 · Cited by 1,179 · cited 1×
“For LLM, NVIDIA proposed a specific optimization for the training process, called Transformer Engine [61], based on the mix-precision training technology [62].”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space LayersPreprint
Gu, A., Johnson, I., Goel, K., Saab, K. K., Dao, T., Rudra, A., & Ré, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13985 · Cited by 1,038 · cited 1×
“Based upon the foundational architecture, the main-stream research focuses on better parametrization on transition matrices [136], [137], [138], [139] and better computational architecture based on SSMs [138], [139], [140], [141], [145].”
Scaling Vision with Sparse Mixture of ExpertsPreprint
Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Pinto, A. S., Keysers, D., & Houlsby, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05974 · Cited by 1,021 · cited 1×
“Xue et al. [105] and Riquelme et al. [106] develop vision transformer-based MoE models, with Riquelme achieving state-of-the-art performance with half the computation.”
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · cited 1×
“Switch Transformer [101] extended this to a transformer-based MoE model with 1.6 T parameters, confirming MoE expansion follows the scaling law.”
InfluentialGemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933 · cited 1×
“Shazeer et al. [100] developed an LSTM model with 137B parameters, significantly enhancing model capacity with minimal computational overhead.”
(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · cited 3×
“Index Terms —Large language model, hardware-software co-design.”
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908 · cited 2×
“Recent advancements have shown that prompts need not be confined to natural language forms but can also be optimized in a continuous space using gradient descent, enhancing their efficiency [45], [46], [47], [48], [49], [50].”
Self-citationMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 1×
“Model parallelism includes two categories: pipeline parallelism [18], [20], [21], which assigns individual layers to single GPUs, and tensor parallelism [22], [23], [24], which divides each tensor into chunks allocated to specific GPUs.”
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · cited 1×
“Linformer [124] is the first to explore the possibility of low-rank approximation of the attention matrix.”
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.8 · Cited by 810 · cited 1×
“Building on this, the p-tuning methodology was developed to further enhance performance by learning concrete prompts within the embedding space, a technique further refined in subsequent studies [48], [50], [53].”
Self-citationEfficiently Modeling Long Sequences with Structured State Spaces
Gu, A., Goel, K., & Ré, C. (2021) · arXiv · doi:10.48550/arxiv.2111.00396 · Cited by 805 · cited 4×
“Later on, S5 [138] proposes to share the transition matrices across all input dimensions to boost the computational efficiency, while Mamba [139] and S4 [137] propose input dependent transition matrices that improve the model capability.”
InfluentialRegister allocation via coloring
Chaitin, G. J., Auslander, M. A., Chandra, A. K., Cocke, J., Hopkins, M. E., & Markstein, P. W. (1981) · Computer Languages · doi:10.1016/0096-0551(81)90048-5 · Cited by 737 · cited 1×
“This approach is akin to register allocation via graph coloring [29], which seeks for scheduling strategies to maximize the reuse of limited registers.”
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMsPreprint
Ashkboos, S., Mohtashami, A., Croci, M. L., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2404.00456 · Cited by 734 · cited 1×
“AWQ (Activation-aware Weight Quantization) [154] optimizes low-bit weight-only quantization by protecting critical weights based on activation distributions, preserving model performance without backpropagation.”
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1×
“Performers [121] and RFA [122] employ the random feature projection as the feature map, while PolySketchFormer [123] exploits sketching techniques with polynomial functions.”
DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node
Subramanya, S. J., Devvrit, F., Simhadri, H. V., Krishnawamy, R., & Kadekodi, R. (2019) · neural information processing systems · Cited by 679
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models
Shang, Y., Cai, M., Xu, B., Lee, Y., & Yan, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02122 · Cited by 642 · cited 1×
“Another method is the algorithmic approach of PruMerge [147], which selectively retains important vision tokens while merging the less significant ones.”
HiPPO: Recurrent Memory with Optimal Polynomial ProjectionsPreprint
Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07669 · Cited by 628 · cited 2×
“The key idea of SSMs is to compress the input sequence of length L , { } , tt L = 1 L states 1 based on HiPPO theory [135].”
Cnvlutin: ineffectual-neuron-free deep neural network computing
Albericio, J., Judd, P., Hetherington, T., Aamodt, T., Jerger, N. E., & Moshovos, A. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001138 · Cited by 614
Cambricon-X: An accelerator for sparse neural networks
Zhang, S., Du, Z., Zhang, L., Lan, H., Liu, S., Li, L., Guo, Q., Chen, T., & Chen, Y. (2016) · 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783723 · Cited by 603 · cited 1×
“Weight pruning [166], [167], [168], [169], [170], [171], [172], [173], [174] reduces the number of parameters by removing less important weights.”
Self-citationLiu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2024) · AI Open · doi:10.1016/j.aiopen.2023.08.012 · Cited by 596
Self-citationTorres H., J. P. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.17754506 · Cited by 580
Training Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561 · cited 2×
“Rematerialization has been implemented in PyTorch for homogeneous sequential networks [28] and more advanced versions [30] are modified for heterogeneous networks.”
A Survey on Efficient Inference for Large Language ModelsPreprint
Zhou, Z., Ning, X., Hong, K., Fu, T., Xu, J., Li, S., Lou, Y., Wang, L., Yuan, Z., Li, X., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.14294 · Cited by 554 · cited 1×
“Profile results indicate that attention and linear operations dominate LLM runtime, accounting for over 75% of total inference duration [70].”
Self-citationBit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Network
Sharma, H., Park, J., Suda, N., Lai, L., Chau, B., Kim, J. K., Chandra, V., & Esmaeilzadeh, H. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00069 · Cited by 543 · cited 1×
“BitFusion [157] supports various bit-width quantizations by combining low-bit processing elements.”
Spikformer: When Spiking Neural Network Meets TransformerPreprint
Zhou, Z., Zhu, Y., He, C., Wang, Y., Yan, S., Tian, Y., & Li, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15425 · Cited by 533 · cited 1×
“Spikformer [214] was the first to implement spiking self-attention in Transformers.”
Self-citationJouppi, N., Kurian, G., Li, S., Ma, P., Nagarajan, R., Nai, L., Patil, N., Subramanian, S., Swing, A., Towles, B., et al. (2023) · Proceedings of the 50th Annual International Symposium on Computer Architecture · doi:10.1145/3579371.3589350 · Cited by 508 · cited 1×
“NorthPole targets large-scale work-load, comparing against Google TPUv4 [211], NVIDIA A100 & H100 AI processors.”
Self-citationSIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training
Qin, E., Samajdar, A., Kwon, H., Nadella, V., Srinivasan, S., Das, D., Kaul, B., & Krishna, T. (2020) · 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00015 · Cited by 487 · cited 1×
“Before this, many studies focused on efficient training based on sparsity and quantization, such as Sigma [65], Tensor-Dash [66], FAST [67], and Combricon-Q [68], including evaluations on Transformer-based models.”
PyTorch distributed: experiences on accelerating data parallel training
Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3415478.3415530 · Cited by 465
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., & Sanghai, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.298 · Cited by 454
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023) · arXiv · doi:10.48550/arxiv.2305.13245 · Cited by 447 · cited 1×
“Multi-Query Attention (MQA) [119] and Group-Query Attention (GQA) [120] improve the attention schemes by sharing the keys and values in multi-head attention.”
Ba, J. L., Kiros, J., & Hinton, G. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06450 · Cited by 440 · cited 1×
“The backbone of modern LLMs is the transformer architecture, which has revolutionized NLP by addressing the limitations of previous recurrent and convolutional models.”
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
Wang, H., Zhang, Z., & Han, S. (2021) · 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00018 · Cited by 427 · cited 1×
“On the algorithm side, optimizing the LLM computation paradigm with compression techniques and the removal of redundant computations enables the LLM processors to achieve both higher efficiency and better hardware utilization.”
What matters when building vision-language models?Preprint
Laurençon, H., Tronchon, L., Cord, M., & Sanh, V. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.02246 · Cited by 426 · cited 1×
“One such method involves the use of the Perceiver module [145], which employs a transformer with queries to perform learned pooling, effectively replacing the MLP as the projector [146].”
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · cited 2×
“One such method involves the use of the Perceiver module [145], which employs a transformer with queries to perform learned pooling, effectively replacing the MLP as the projector [146].”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2006.16668 · Cited by 412 · cited 3×
“Model parallelism includes two categories: pipeline parallelism [18], [20], [21], which assigns individual layers to single GPUs, and tensor parallelism [22], [23], [24], which divides each tensor into chunks allocated to specific GPUs.”
QuIP: 2-Bit Quantization of Large Language Models With Guarantees
Chee, J., Cai, Y., Kuleshov, V., & De, C. (2023) · PubMed · doi:10.48550/arxiv.2307.13304 · Cited by 372 · cited 1×
“QuIP [156] facilitates 2-bit quantization through incoherence processing, leveraging incoherent weight and Hessian matrices, and using adaptive rounding to minimize quantization error, supported by theoretical analysis.”
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K. (2024) · IEEE Micro · doi:10.1109/mm.2024.3373763 · Cited by 369 · cited 1×
“Index Terms —Large language model, hardware-software co-design.”
Fast Distributed Inference Serving for Large Language ModelsPreprint
Wu, B., Zhong, Y., Zhang, Z., Liu, S., Liu, F., Sun, Y., Huang, G., Liu, X., & Jin, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05920 · Cited by 357 · cited 2×
“To combat this, FastServe [86] leverages a preemptive scheduling strategy that priori-tizes requests based on their estimated completion time, thus improving throughput and reducing job completion times.”
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., & Tian, Q. C. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.14717 · Cited by 327 · cited 1×
“QA-LoRA [41] tries to get the best of both quantization and adaptation by balancing the unequal freedom between them inherent in LoRA through group-wise operators.”
BackgroundSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298 · cited 1×
“SpotServe [88] addresses the challenges of using cloud-based preemptible GPU resources by implementing strategies for dynamic adjustment and state recovery, ensuring robust service Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022) · SC22: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41404.2022.00051 · Cited by 297 · cited 5×
“When GPU memory is insufficient, techniques such as ZeRO-Inference by DeepSpeed [33] offload large model weights to CPU memory to improve performance by overlapping computation with weight fetching.”
InfluentialFactual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration
Lin, J., Tang, J., Tang, H., Yang, S., Xiao, G., & Han, S. (2025) · GetMobile: Mobile Computing and Communications · doi:10.1145/3714983.3714987 · Cited by 274
(2021) · doi:10.1145/3458817.3476205 · Cited by 257 · cited 2×
“Zero Redundancy Optimizer (ZeRO) [9] and its subsequent works [25], [26], [27] have been proposed to alleviate the high GPU memory requirement in large model training.”
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., & Tang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07602 · Cited by 254 · cited 2×
“Recent advancements have shown that prompts need not be confined to natural language forms but can also be optimized in a continuous space using gradient descent, enhancing their efficiency [45], [46], [47], [48], [49], [50].”
Self-citationSmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2022) · arXiv · doi:10.48550/arxiv.2211.10438 · Cited by 247 · cited 2×
“Innovative quantization methods have significantly enhanced the efficiency and performance of LLMs. SmoothQuant [153] enables 8-bit weight and activation quantization (W8A8) by migrating quantization difficulty from activations to weights through per-channel scaling, reducing activation outliers…”
WARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243 · cited 1×
“Recent advancements have shown that prompts need not be confined to natural language forms but can also be optimized in a continuous space using gradient descent, enhancing their efficiency [45], [46], [47], [48], [49], [50].”
NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
Heo, G., Lee, S., Cho, J., Choi, H., Lee, S., Ham, H., Kim, G., Mahajan, D., & Park, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651380 · Cited by 243 · cited 2×
“In the TranCIM macros, the SRAM arrays store the weight matrices and take in the input vectors along the bitlines in the same direction.”
ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads
Hu, C.-C., Huang, H., Xu, L., Chen, X., Wang, C., Xu, J., Chen, S., Feng, H., Wang, S., Bao, Y., et al. (2025) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3732941 · Cited by 222
AttAcc! Unleashing the Power of PIM for Batched Transformer-based Generative Model Inference
Park, J., Choi, J., Kyung, K., Kim, M. J., Kwon, Y., Kim, N. S., & Ahn, J. H. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640422 · Cited by 221 · cited 2×
“In the TranCIM macros, the SRAM arrays store the weight matrices and take in the input vectors along the bitlines in the same direction.”
Parameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A., & Kim, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.378 · Cited by 220 · cited 1×
“Other PEFT methods selectively update a subset of the pre-trained model weights during adaptation.”
Zhou, X., Du, Z., Guo, Q., Liu, S., Liu, C., Wang, C., Zhou, X., Li, L., Chen, T., & Chen, Y. (2018) · 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2018.00011 · Cited by 214 · cited 1×
“Weight pruning [166], [167], [168], [169], [170], [171], [172], [173], [174] reduces the number of parameters by removing less important weights.”
Self-citationSpeculative Decoding with Big Little Decoder
Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M. W., Gholami, A., & Keutzer, K. (2023) · neural information processing systems · doi:10.52202/075280-1705 · Cited by 211 · cited 1×
“Speculative decoding [127], [128] aims to speed up the decoding process for auto-regressive large language models (LLMs).”
SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks
Shi, X., Hao, Z., & Yu, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00536 · Cited by 199 · cited 1×
“Spikingformer demonstrated further innovation [216], incorporating spike-driven residual learning within Transformer-based SNNs.”
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training
Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., & Cheng, Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.890 · Cited by 192
StableMoE: Stable Routing Strategy for Mixture of Experts
Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.489 · Cited by 184
PowerInfer-2: Fast Large Language Model Inference on a SmartphonePreprint
Xue, Z., Song, Y., Mi, Z., Zheng, X., Xia, Y., & Chen, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.06282 · Cited by 181 · cited 3×
“However, driven by the growing need for latency-insensitive tasks, some studies [79], [80], [91], [92], [93] explore high-throughput LLM inference using limited resources, such as a single GPU, edge devices, and mobile devices.”
Liang, H.-X., Fan, Z.-W., Sarkar, R., Jiang, Z.-Y., Chen, T.-L., Zou, K., Cheng, Y., Hao, C., & Wang, Z.-Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.14793 · Cited by 178 · cited 1×
“SE-MoE [115], M3ViT [116], and SiDA-MoE [117] introduce strategies to optimize throughput and expert caching.”
SpecTr: Fast Speculative Decoding via Optimal Transport
Sun, Z., Suresh, A. T., Ro, J. H., Beirami, A., Jain, H., & Yu, F. (2023) · neural information processing systems · doi:10.52202/075280-1314 · Cited by 168
ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization
Guo, C., Zhang, C., Leng, J., Liu, Z., Yang, F., Liu, Y., Guo, M., & Zhu, Y. (2022) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00095 · Cited by 165 · cited 1×
“OLAccel [158] and GOBO [158] quantizes outliers with higher precision, but these approaches often suffer from unaligned memory accesses, leading to additional overhead and limited computing speed.”
Self-citationSimplified State Space Layers for Sequence Modeling
Smith, J. T. H., Warrington, A., & Linderman, S. W. (2022) · arXiv · doi:10.48550/arxiv.2208.04933 · Cited by 162 · cited 2×
“Later on, S5 [138] proposes to share the transition matrices across all input dimensions to boost the computational efficiency, while Mamba [139] and S4 [137] propose input dependent transition matrices that improve the model capability.”
FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction
Qin, Y., Wang, Y., Deng, D., Zhao, Z., Yang, X., Liu, L., Wei, S., Hu, Y., & Yin, S. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589057 · Cited by 162 · cited 1×
“On the hardware side, numerous research efforts have been focused on investigating how to take the advantages of novel architectures to minimize costly data movement and enhance computational parallelism.”
Self-citationZhu, M., Zhang, T., Gu, Z., & Xie, Y. (2019) · Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1145/3352460.3358269 · Cited by 153 · cited 2×
“Restrictions apply. into a structured N:M sparsity pattern, which can efficiently run on sparse tensor cores in GPUs [170].”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., & Gibbons, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03377 · Cited by 153 · cited 1×
“Model parallelism includes two categories: pipeline parallelism [18], [20], [21], which assigns individual layers to single GPUs, and tensor parallelism [22], [23], [24], which divides each tensor into chunks allocated to specific GPUs.”
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., & Zhu, Y. (2023) · Proceedings of the 50th Annual International Symposium on Computer Architecture · doi:10.1145/3579371.3589038 · Cited by 152 · cited 2×
“The digital SRAM-based Bitline-Transpose-CIM macro is introduced to process multi-ply-accumulate (MAC) operations.”
Self-citationZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151 · cited 2×
“ZeRO-Offload [25] enables the training of even larger models by offloading optimizer states and optimizer updates to the CPU in order to strike a balance between accessibility and efficiency.”
Training Neural Networks with Fixed Sparse MasksPreprint
Sung, Y.-L., Nair, V., & Raffel, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09839 · Cited by 151 · cited 1×
“FISH Mask [59] gauges the importance of each model parameter by estimating its’ Fisher information, resulting in a binary mask that indicates which parameters are crucial for the current task.”
Online Speculative DecodingPreprint
Liu, X., Hu, L., Bailis, P. D., Cheung, A., Deng, Z., Stoica, I. G., & Zhang, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07177 · Cited by 150
LLM-Pruner: On the Structural Pruning of Large Language Models
Ma, X., Fang, G., & Wang, X. (2023) · neural information processing systems · doi:10.52202/075280-0950 · Cited by 150
Wang, Y., Zhang, C., Xie, Z., Guo, C., Liu, Y., & Leng, J. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00088 · Cited by 144 · cited 2×
“For instance, the Dual-Side Sparse Tensor Core (DS-STC) [172] modifies tensor core architecture on GPUs to support dual-side sparse matrix multiplication with arbitrary sparsity, outperforming NVIDIA’s sparse tensor core on pruned BERT models.”
Self-citationFlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143 · cited 1×
“To optimize attention computation, FlashAttention [71], [72] integrates the entire process into a single operator, achieving 3 × training speed up on GPT-2 model.”
TransPIM: A Memory-based Acceleration via Software-Hardware Co-Design for Transformer
Zhou, M., Xu, W., Kang, J., & Rosing, T. (2022) · 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00082 · Cited by 143 · cited 2×
“The solutions to achieving this goal and boosting the efficiency of LLM inference involves advancements to both hardware and algorithms, and this research is summarized in Table 2.”
FlashDecoding++: Faster Large Language Model Inference on GPUsPreprint
Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, Y., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.01282 · Cited by 138 · cited 2×
“Flash-Decoding [73] and FlashDecoding ++ [74] further enhance this by optimizing parallelism and introducing efficiency in SoftMax computation.”
Self-citationPlug-and-Play: An Efficient Post-training Pruning Method for Large Language ModelsPreprint
Zhang, Y., Bai, H., Lin, H., Zhao, J., Hou, L., & Cannistraci, C. V. (2024) · Preprints.org · doi:10.20944/preprints202310.1487.v2 · Cited by 135
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
Song, Y., Mi, Z., Xie, H., & Chen, H. (2024) · Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles · doi:10.1145/3694715.3695964 · Cited by 133 · cited 3×
“However, driven by the growing need for latency-insensitive tasks, some studies [79], [80], [91], [92], [93] explore high-throughput LLM inference using limited resources, such as a single GPU, edge devices, and mobile devices.”
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning TasksPreprint
Park, J.-H., Park, J., Xiong, Z., Lee, N.-Y., Cho, J., Oymak, S., Lee, K., & Papailiopoulos, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.04248 · Cited by 133 · cited 3×
“Meanwhile, Mamba [139] and S4 [137] utilize a parallel scan technique that improves the computational efficiency of SSMs. MambaFormer [141] and Jamba [142] improve the SSM architecture by combining transformers into SSMs, where [141] use the SSM layer to replace the FFN layer in transformers and…”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
Frantar, E., & Alistarh, D. (2023) · arXiv · doi:10.48550/arxiv.2301.00774 · Cited by 131 · cited 1×
“SparseGPT [179] achieves up to 50% weight sparsity in GPT models through optimal partial updates and adaptive masked selection.”
DOTA: detect and omit weak attentions for scalable transformer acceleration
Qu, Z., Liu, L., Tu, F., Chen, Z., Ding, Y., & Xie, Y. (2022) · Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3503222.3507738 · Cited by 130 · cited 3×
“Algorithm and hardware co-design is often used to efficiently determine the sparsity pattern and compute the sparse attention [184], [187], [194], [197], [203].”
Learning Factored Representations in a Deep Mixture of Experts
Eigen, D., Ranzato, M., & Sutskever, I. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.4314 · Cited by 127 · cited 1×
“Later on, [98] proposes to stack several neural-network-based MoE layers to make the model deeper with the rise of deep learning.”
Wang, Z., Fang, Y., Cao, J., Zhang, Q., Wang, Z., & Xu, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00169 · Cited by 124 · cited 1×
“Following this, Masked Spiking Transformers were realized, utilizing Random Spike Masking (RSM) to reduce redundant spikes effectively [215].”
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124 · cited 1×
“Despite this, manually crafted prompts often exhibit a performance discrepancy compared to fine-tuned models, as noted by multiple studies [4], [42], [43], [44].”
RaPiD: AI Accelerator for Ultra-low Precision Training and Inference
Venkataramani, S., Srinivasan, V., Wang, W., Sen, S., Zhang, J., Agrawal, A., Kar, M., Jain, S., Mannari, A., Tran, H., et al. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00021 · Cited by 120 · cited 1×
“IBM presented RaPiD [207], [208] While numerous hardware and algorithmic accelerations have been proposed in academia, they are limited to certain applications and uses. from their previous works, which were categorized as ASIC accelerators.”
Self-citationParameter-Efficient Fine-Tuning without Introducing New Latency
Liao, B., Meng, Y., & Monz, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.233 · Cited by 117
Neural inference at the frontier of energy, space, and time
Modha, D. S., Akopyan, F., Andreopoulos, A., Appuswamy, R., Arthur, J. V., Cassidy, A. S., Datta, P., DeBole, M. V., Esser, S. K., Otero, C. O., et al. (2023) · Science · doi:10.1126/science.adh1174 · Cited by 115 · cited 1×
“The performance achieved is a joint effort of both architecture and their SDK tool-chain.”
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. (2024) · Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 · doi:10.1145/3620666.3651335 · Cited by 114
LLM in a flash: Efficient Large Language Model Inference with Limited Memory
Alizadeh, K., Mirzadeh, S. I., Belenko, D., Khatamifard, S., Cho, M., Del Mundo, C. C., Rastegari, M., & Farajtabar, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.678 · Cited by 113
FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding
Zhang, S. Q., McDanel, B., & Kung, H. T. (2022) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00067 · Cited by 113
Multi-Source and Multi-modal Deep Network Embedding for Cross-Network Node Classification
Yang, H., He, H., Zhang, W., Wang, Y., & Jing, L. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3653304 · Cited by 111 · cited 3×
“This focus is based on three primary reasons: ■■ Shift in Popularity: BERT models, especially En-coder-only variants, have gradually begun to fade away within the community [14].”
InfluentialSelf-citationPeng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., & Kong, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02143 · Cited by 110 · cited 1×
“Performers [121] and RFA [122] employ the random feature projection as the feature map, while PolySketchFormer [123] exploits sketching techniques with polynomial functions.”
LLMCad: Fast and Scalable On-device Large Language Model InferencePreprint
Xu, D., Yin, W., Jin, X., Zhang, Y., Wei, S., Xu, M., & Liu, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.04255 · Cited by 110 · cited 1×
“One line of research focuses on exploiting the computing units that ask small models to generate several candidates to be evaluated by LLMs in parallel [129], [130], [131].”
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Wang, G., Qin, H., Jacobs, S. A., Holmes, C., Rajbhandari, S., Ruwase, O., Yan, F., Yang, L., & He, Y. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2306.10209 · Cited by 109 · cited 2×
“Zero Redundancy Optimizer (ZeRO) [9] and its subsequent works [25], [26], [27] have been proposed to alleviate the high GPU memory requirement in large model training.”
Learning Channel-Wise Interactions for Binary Convolutional Neural Networks
Wang, Z., Lu, J., Tao, C., Zhou, J., & Tian, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00066 · Cited by 107
Xue, F., Shi, Z., Wei, F., Lou, Y., Liu, Y., & You, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20858 · Cited by 107
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
Guo, H., Greengard, P., Xing, E. P., & Kim, Y. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.12023 · Cited by 104 · cited 1×
“LQ-LoRA [40] further extends the quantization limit to sub-3 bits by decomposing each pre-trained matrix into a frozen quantized matrix and an adaptable low-rank matrix.”
BackgroundFast Transformer Decoding: One Write-Head is All You Need
Shazeer, N. (2019) · arXiv · doi:10.48550/arxiv.1911.02150 · Cited by 101 · cited 1×
“Multi-Query Attention (MQA) [119] and Group-Query Attention (GQA) [120] improve the attention schemes by sharing the keys and values in multi-head attention.”
LLM as a System Service on Mobile DevicesPreprint
Yin, W., Xu, M., Li, Y., & Liu, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.11805 · Cited by 100 · cited 2×
“However, driven by the growing need for latency-insensitive tasks, some studies [79], [80], [91], [92], [93] explore high-throughput LLM inference using limited resources, such as a single GPU, edge devices, and mobile devices.”
ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs
Zhai, Y., Jiang, C., Wang, L., Jia, X., Zhang, S., Chen, Z., Liu, X., & Zhu, Y. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00042 · Cited by 99 · cited 2×
“Additional optimizations include the fusion of lightweight operations such as LayerNorm, SwiGLU, activation functions, and residual additions by frameworks like Deep-Speed [33], Byte-Transformer [76], xFormers [77], and TensorRT-LLM [75], which also uses a pattern-matching algorithm to identify potential fusions acros…”
SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian ApproximationPreprint
Guo, C., Qiu, Y., Leng, J., Gao, X., Zhang, C., Liu, Y., Yang, F., Zhu, Y., & Guo, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.07471 · Cited by 96 · cited 1×
“On the other hand, PTQ [148], [151], [152] converts a floating-point model to a lower-bit model without requiring training data, making it particularly suitable for large-scale language models.”
Self-citationYu, D., Shen, L., Hao, H., Gong, W., Wu, H., Bian, J., Dai, L.-R., & Xiong, H. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3399654 · Cited by 95
TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training
Mahmoud, M., Edo, I., Zadeh, A. H., Awad, O. M., Pekhimenko, G. G., Albericio, J., & Moshovos, A. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00069 · Cited by 91 · cited 1×
“Before this, many studies focused on efficient training based on sparsity and quantization, such as Sigma [65], Tensor-Dash [66], FAST [67], and Combricon-Q [68], including evaluations on Transformer-based models.”
BASE Layers: Simplifying Training of Large, Sparse Models
Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2103.16716 · Cited by 88 · cited 1×
“Base Layer [110] uses a linear assignment problem for balanced loading, while Hash Layer [111] uses predefined hash functions.”
Paradigm Shift in Natural Language Processing
Sun, T.-X., Liu, X.-Y., Qiu, X.-P., & Huang, X.-J. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1331-6 · Cited by 86 · cited 1×
“Despite this, manually crafted prompts often exhibit a performance discrepancy compared to fine-tuned models, as noted by multiple studies [4], [42], [43], [44].”
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.06865 · Cited by 86 · cited 3×
“…due to limited DRAM capacity, resulting in data transfer between DRAM and storage; the second scenario occurs when CPU and GPU cannot share memory [79], [93], requiring model parameters and intermediate results to be stored in host memory due to limited GPU memory, thus leading to data transfer…”
Self-citationLi, C., Zhou, Z., Zheng, S., Zhang, J., Liang, Y. E., & Sun, G. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651352 · Cited by 78 · cited 2×
“By performing MAC operations locally within the SRAM array, CIM macros eliminate excessive and costly data transference for intermediate data.”
Self-citationPark, S., Kim, K.-S., So, J., Jung, J., Lee, J.-G., Woo, K., Kim, N., Lee, Y., Kim, H., Kwon, Y., et al. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00078 · Cited by 77 · cited 1×
“In another study, the authors of CXL-PNM [206] introduce a processing near memory (PNM) platform using Compute Express Link (CXL), leveraging both model parallelism and data parallelism for workload partitioning.”
Dao, T., & Gu, A. (2024) · arXiv · doi:10.48550/arxiv.2405.21060 · Cited by 76 · cited 2×
“…architecture by combining transformers into SSMs, where [141] use the SSM layer to replace the FFN layer in transformers and Jamba [142] add four transformer layers to SSMs. Mamaba2 [140] proposes a new architecture based on State-Space Du-ality that achieves 2-8 × speed up compared to Mamba [139].”
LRT: An Efficient Low-Light Restoration Transformer for Dark Light Field Images
Zhang, S., Meng, N., & Lam, E. Y. M. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3297412 · Cited by 74 · cited 1×
“LRT [125] then proposes to apply low-rank approximation on both the attention matrix and feed-forward layers.”
SpotServe: Serving Generative Large Language Models on Preemptible Instances
Miao, X., Shi, C., Duan, J., Xi, X., Lin, D., Cui, B., & Jia, Z. (2024) · Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3620665.3640411 · Cited by 74 · cited 1×
“SpotServe [88] addresses the challenges of using cloud-based preemptible GPU resources by implementing strategies for dynamic adjustment and state recovery, ensuring robust service Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Fast Inference from Transformers via Speculative Decoding
Leviathan, Y., Kalman, M., & Matias, Y. (2022) · arXiv · doi:10.48550/arxiv.2211.17192 · Cited by 71 · cited 1×
“Speculative decoding [127], [128] aims to speed up the decoding process for auto-regressive large language models (LLMs).”
Jamba: A Hybrid Transformer-Mamba Language Model
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.19887 · Cited by 70
(2024) · doi:10.52202/079017-3180 · Cited by 70
Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity
Guo, C., Hsueh, B. Y., Leng, J., Qiu, Y., Guan, Y., Wang, Z., Jia, X., Li, X., Guo, M., & Zhu, Y. (2020) · SC20: International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc41405.2020.00020 · Cited by 69 · cited 1×
“Weight pruning [166], [167], [168], [169], [170], [171], [172], [173], [174] reduces the number of parameters by removing less important weights.”
Self-citationThe memory fragmentation problem
Johnstone, M. S., & Wilson, P. R. (1998) · ACM SIGPLAN Notices · doi:10.1145/301589.286864 · Cited by 68
X-Former: In-Memory Acceleration of Transformers
Sridharan, S., Stevens, J. R., Roy, K., & Raghunathan, A. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3282046 · Cited by 68 · cited 1×
“As a result, there is a strong need for energy-efficient, dedicated processors to minimize these costs, especially on power-constrained edge devices.”
Mixtures of Experts Unlock Parameter Scaling for Deep RLPreprint
Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J., Dziugaite, G. K., Precup, D., & Castro, P. S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.08609 · Cited by 67 · cited 1×
“Obando-Ceron et al. [107] constructs an MoE model for reinforcement learning, providing empirical evidence for scaling laws in this domain.”
Cerebras Architecture Deep Dive: First Look Inside the Hardware/Software Co-Design for Deep Learning
Lie, S. (2023) · IEEE Micro · doi:10.1109/mm.2023.3256384 · Cited by 66 · cited 1×
“To keep up with the growth of extreme-scale ML models, they proposed a new chip architecture that is wafer-sized.”
(2020) · neural information processing systems · Cited by 66
Hash Layers For Large Sparse Models
Roller, S., Sukhbaatar, S., Szlam, A., & Weston, J. (2021) · arXiv · doi:10.48550/arxiv.2106.04426 · Cited by 66 · cited 1×
“Base Layer [110] uses a linear assignment problem for balanced loading, while Hash Layer [111] uses predefined hash functions.”
What matters when building vision-language models?
Laurençon, H., Tronchon, L., Cord, M., & Sanh, V. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2789 · Cited by 66
Guo, C., Zhang, R., Xu, J., Leng, J., Liu, Z., Huang, Z., Guo, M., Wu, H., Zhao, S., Zhao, J., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640423 · Cited by 64 · cited 1×
“To address this, GMLake [31] employs a virtual memory stitching (VMS) mechanism to merge non-contiguous memory blocks, significantly reducing GPU memory usage for LLM fine-tuning.”
MethodSelf-citationSparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation
Yazdanbakhsh, A., Moradifirouzabadi, A., Li, Z., & Kang, M. (2022) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00059 · Cited by 63 · cited 1×
“Additionally, SPRINT [185] computes attention scores approximately and prunes low attention scores using lightweight analog thresholding circuitry within the processing element (PE) arrays.”
Self-citationKim, S., Kim, S., Jo, W., Kim, S., Hong, S., & Yoo, H. (2024) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49657.2024.10454330 · Cited by 61 · cited 3×
“These advancements have been further augmented by the realization of C-Transformer [195], a processor designed to accelerate Spiking Transformer operations and LLMs.”
LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
Zhang, M., Chen, H., Shen, C., Yang, Z., Ou, L., Yu, X., & Zhuang, B. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.178 · Cited by 61 · cited 1×
“SPLoRA [37] and LoRAPrune [38] both leverage structured channel-pruning to remove groups of weights in order to increase the computational efficiency.”
Accelerating attention through gradient-based learned runtime pruning
Li, Z., Ghodrati, S., Yazdanbakhsh, A., Esmaeilzadeh, H., & Kang, M. (2022) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3470496.3527423 · Cited by 59 · cited 1×
“Algorithm and hardware co-design is often used to efficiently determine the sparsity pattern and compute the sparse attention [184], [187], [194], [197], [203].”
Self-citationTurbo Sparse: Achieving LLM SOTA Performance with Minimal Activated ParametersPreprint
Song, Y., Xie, H., Zhang, Z., Wen, B., Ma, L., Mi, Z., & Chen, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.05955 · Cited by 57 · cited 1×
“Existing work has observed that retaining only a subset of effective activation values does not degrade model performance, and the sparsity pattern of activation values is predictable [80], [91], [93], [94], [95].”
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language ModelsPreprint
Sun, J., Xu, Z., Yin, H., Yang, D., Xu, D., Chen, Y., & Roth, H. R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01467 · Cited by 53 · cited 1×
“The concept of prompt tuning [45], [51], [52] was introduced to fine-tune a continuous vector that is concatenated to the input embeddings, optimizing the prompt directly within the embedding space.”
Self-citationLory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingPreprint
Zhong, Z., Xia, M., Chen, D., & Lewis, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.03133 · Cited by 52 · cited 1×
“Differentiable MoE architectures like Soft MoE [113] and Lory [114] enhance stability by making operations differentiable.”
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
Jang, H., Song, J., Jung, J., Park, J., Kim, Y., & Lee, J. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00034 · Cited by 51 · cited 1×
“Smart-Infinity [64] is the first study to leverage host memory and storage as an extended memory hierarchy for LLM training, enhancing efficiency by addressing storage bandwidth bottle-necks through near-storage processing.”
LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Liu, S.-Y., Liu, Z., Huang, X., Dong, P., & Cheng, K.-T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.39 · Cited by 50 · cited 1×
“LLM-FP4 [164] utilizes 4-bit floating-point quantization, optimizing exponent bits and clipping ranges, achieving minimal accuracy loss while enabling efficient deployment in resource-constrained environments.”
Effective Training of Convolutional Neural Networks With Low-Bitwidth Weights and Activations
Zhuang, B., Tan, M., Liu, J., Liu, L., Reid, I. D., & Shen, C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3088904 · Cited by 49 · cited 1×
“QAT, as discussed in [148], [149], and [150], involves retraining the model to adapt to quantization noise.”
Fusing Models with Complementary Expertise
Wang, H., Polo, F. M., Sun, Y., Kundu, S., Xing, E. P., & Yurochkin, M. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.01542 · Cited by 49 · cited 1×
“Analogy to MoE, FoE (Fusion of Experts) [118] also aggregates knowledge from different experts which can be pre-trained in their respective domains.”
ST-MoE: Designing Stable and Transferable Sparse Expert Models
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., & Fedus, W. (2022) · arXiv · doi:10.48550/arxiv.2202.08906 · Cited by 48 · cited 1×
“Addressing instability in training large MoE models, ST-MoE [103] improved transfer learning performance.”
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
Zhang, Z., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.71 · Cited by 48 · cited 1×
“MoEfication [108], [109] converts dense models to MoE models by grouping weights in FFNs. Routing strategies have been explored to balance load and address training instability.”
Unified Scaling Laws for Routed Language Models
Clark, A., Casas, D. D. L., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.01169 · Cited by 40 · cited 1×
“Clark et al. [102] investigated the scaling law for routing-based language models, deriving an Effective Parameter Count for scalable models.”
Cambricon-Q: A Hybrid Architecture for Efficient Training
Zhao, Y., Liu, C., Du, Z., Guo, Q., Hu, X., Zhuang, Y., Zhang, Z., Song, X., Li, W., Zhang, X., et al. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00061 · Cited by 39 · cited 1×
“Before this, many studies focused on efficient training based on sparsity and quantization, such as Sigma [65], Tensor-Dash [66], FAST [67], and Combricon-Q [68], including evaluations on Transformer-based models.”
Self-citationLee, S. K., Agrawal, A., Silberman, J. A., Ziegler, M. M., Kang, M., Venkataramani, S., Cao, N., Fleischer, B., Guillorn, M., Cohen, M., et al. (2021) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2021.3120113 · Cited by 39 · cited 1×
“IBM presented RaPiD [207], [208] While numerous hardware and algorithmic accelerations have been proposed in academia, they are limited to certain applications and uses. from their previous works, which were categorized as ASIC accelerators.”
Liang, H., Fan, Z., Sarkar, R., Jiang, Z., Chen, T., Zou, K., Cheng, Y., Hao, C., & Wang, Z. (2022) · neural information processing systems · doi:10.52202/068431-2062 · Cited by 39
Herrmann, J., Beaumont, O., Eyraud‐Dubois, L., Hermann, J., Joly, A., & Shilova, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.13214 · Cited by 38 · cited 1×
“Rematerialization has been implemented in PyTorch for homogeneous sequential networks [28] and more advanced versions [30] are modified for heterogeneous networks.”
Mokey: enabling narrow fixed-point inference for out-of-the-box floating-point transformer models
Zadeh, A. H., Mahmoud, M., Abdelhadi, A., & Moshovos, A. (2022) · Proceedings of the 49th Annual International Symposium on Computer Architecture · doi:10.1145/3470496.3527438 · Cited by 37 · cited 2×
“Some of these accelerators, which were introduced in the previous subsection, focus on compression techniques such as sparsity and quantization.”
Lu, Z., Wang, X., Arafin, M. T., Yang, H., Liu, Z., Zhang, J., & Qu, G. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3345651 · Cited by 37 · cited 2×
“RIME employs single-cycle NOR, NAND, and innovative minority (Min3) logic functions within RRAM cells to perform high-precision floating-point operations directly in memory.”
Self-citationFrom Sparse to Soft Mixtures of Experts
Puigcerver, J., Riquelme, C., Mustafa, B., & Houlsby, N. (2023) · arXiv · doi:10.48550/arxiv.2308.00951 · Cited by 36 · cited 1×
“Differentiable MoE architectures like Soft MoE [113] and Lory [114] enhance stability by making operations differentiable.”
HAIMA: A Hybrid SRAM and DRAM Accelerator-in-Memory Architecture for Transformer
Ding, Y., Liu, C., Duan, M., Chang, W., Li, K., & Li, K. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247913 · Cited by 34 · cited 5×
“Algorithm Acceleration.”
Influential11.4 IBM NorthPole: An Architecture for Neural Network Inference with a 12nm Chip
Cassidy, A. S., Arthur, J. V., Akopyan, F., Andreopoulos, A., Appuswamy, R., Datta, P., DeBole, M., Esser, S. K., Otero, C. O., Sawada, J., et al. (2024) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49657.2024.10454451 · Cited by 34 · cited 1×
“The performance achieved is a joint effort of both architecture and their SDK tool-chain.”
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-DesignPreprint
Xia, H., Zhen, Z., Wu, X., Chen, S., Yao, Z., Youn, S., Bakhtiari, A., Wyatt, M. R., Zhuang, D., Zhou, Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.14112 · Cited by 32 · cited 1×
“FP6-LLM [163] introduces a 6-bit floating-point format that leverages TC-FPx, a GPU kernel design, to reduce inference costs and improve performance for large language models (LLMs).”
Self-citationAn Efficient 2D Method for Training Super-Large Deep Learning Models
Xu, Q., & You, Y. (2023) · 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00031 · Cited by 31 · cited 1×
“Model parallelism includes two categories: pipeline parallelism [18], [20], [21], which assigns individual layers to single GPUs, and tensor parallelism [22], [23], [24], which divides each tensor into chunks allocated to specific GPUs.”
Self-citationDFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation
Hong, S., Moon, S., Kim, J., Lee, S., Kim, M., Lee, D., & Kim, J.-Y. (2022) · IEEE Hot Chips Symposium · doi:10.1109/hcs55958.2022.9895626 · Cited by 29
ASADI: Accelerating Sparse Attention Using Diagonal-based In-Situ Computing
Li, H., Li, Z., Bai, Z., & Mitra, T. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00065 · Cited by 28 · cited 3×
“On the algorithm side, optimizing the LLM computation paradigm with compression techniques and the removal of redundant computations enables the LLM processors to achieve both higher efficiency and better hardware utilization.”
Self-citationRM-STC: Row-Merge Dataflow Inspired GPU Sparse Tensor Core for Energy-Efficient Sparse Acceleration
Huang, G., Wang, Z. Y., Tsai, P.-A., Zhang, C., Ding, Y., & Xie, Y. (2023) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3613424.3623775 · Cited by 28 · cited 1×
“DS-STC changes the dataflow of the tensor core from inner-product to outer-product, making it more suitable for arbitrary sparse computation.”
Hedegaard, L., Alok, A., Jose, J., & Iosifidis, A. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110724 · Cited by 27
Progressive Gradient Flow for Robust N:M Sparsity Training in TransformersPreprint
Bambhaniya, A. R., Yazdanbakhsh, A., Subramanian, S., Kao, S.-C., Agrawal, S., Evci, U., & Krishna, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.04744 · Cited by 25 · cited 1×
“…applies N:M structured sparsity to both forward and backward passes, achieves significant speedups and memory savings compared to dense LLMs. PGF [178] further explored sparse LLM training recipes at high sparsity level, by introducing progressive gradient flow techniques for N : M structured…”
Effective Interplay between Sparsity and Quantization: From Theory to Practice
Harma, S. B., Chakraborty, A., Kostenok, E., Mishin, D., Ha, D., Falsafi, B., Jaggi, M., Liu, M., Oh, Y., Subramanian, S., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2405.20935 · Cited by 25 · cited 1×
“Reference [204] provides both theoretical and empirical evidence on the optimal way to combine sparsity and quantization in deep neural networks, offering valuable insights for model compression and efficient deployment of large language models.”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., & Ramjee, R. (2024) · arXiv · doi:10.48550/arxiv.2403.02310 · Cited by 24 · cited 1×
“This method is extended in Sarathi [82], Sarathi-Serve [83] and LightLLM [84], which employ a split-and-fuse technique to balance load across different processing stages, thereby minimizing response times and enhancing throughput.”
Accelerating Sparse DNNs Based on Tiled GEMM
Guo, C., Xue, F., Leng, J., Qiu, Y., Guan, Y., Cui, W., Chen, Q., & Guo, M. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3365942 · Cited by 21 · cited 1×
“Weight pruning [166], [167], [168], [169], [170], [171], [172], [173], [174] reduces the number of parameters by removing less important weights.”
Self-citationFan, Z., Zhang, Q., Abillama, P., Shoouri, S., Lee, C. S. G., Blaauw, D. T., Kim, H.-S., & Sylvester, D. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589040 · Cited by 20 · cited 2×
“On the algorithm side, optimizing the LLM computation paradigm with compression techniques and the removal of redundant computations enables the LLM processors to achieve both higher efficiency and better hardware utilization.”
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2023) · arXiv · doi:10.48550/arxiv.2308.16369 · Cited by 20 · cited 1×
“This method is extended in Sarathi [82], Sarathi-Serve [83] and LightLLM [84], which employ a split-and-fuse technique to balance load across different processing stages, thereby minimizing response times and enhancing throughput.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Ré, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17157 · Cited by 20 · cited 1×
“Existing work has observed that retaining only a subset of effective activation values does not degrade model performance, and the sparsity pattern of activation values is predictable [80], [91], [93], [94], [95].”
Parameter-Efficient Transfer Learning with Diff Pruning
Guo, D., Rush, A. M., & Kim, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.07463 · Cited by 20 · cited 1×
“Other PEFT methods selectively update a subset of the pre-trained model weights during adaptation.”
Transformer-OPU: An FPGA-based Overlay Processor for Transformer Networks
Bai, Y., Zhou, H., Zhao, K., Chen, J., Yu, J., & Wang, K. (2023) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm57271.2023.00049 · Cited by 19 · cited 1×
“On the hardware side, numerous research efforts have been focused on investigating how to take the advantages of novel architectures to minimize costly data movement and enhance computational parallelism.”
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs
Mozaffari, M., Yazdanbakhsh, A., Zhang, Z., & Dehnavi, M. M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2405.16325 · Cited by 17 · cited 1×
“The Plug-and-Play [176] prunes weights Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
TF-MVP: Novel Sparsity-Aware Transformer Accelerator with Mixed-Length Vector Pruning
Yoo, E., Park, G., Min, J. G., Kwon, S. J., Park, B., Lee, D., & Lee, Y. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247799 · Cited by 16 · cited 1×
“On the hardware side, numerous research efforts have been focused on investigating how to take the advantages of novel architectures to minimize costly data movement and enhance computational parallelism.”
Accelerating Sparse Attention with a Reconfigurable Non-volatile Processing-In-Memory Architecture
Zheng, Q., Li, S., Wang, Y., Li, Z., Chen, Y., & Li, H. H. (2023) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac56929.2023.10247908 · Cited by 15 · cited 1×
“This avoids the need for transpose buffers on the output side to transpose the generated self-attention matrices.”
Self-citationYu, D., Shen, L., Hao, H., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.10034 · Cited by 12
Research works printed but not linked: 23
- Radford, 2018, Improving language understanding by generative pre-training
- 2023, GPT-4 technical report
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Ren, Zero-offload: Democratizing billionscale model training, 551
- 2020, NVIDIA A100 Tensor Core GPU Architecture
- 2022, NVIDIA H100 Tensor Core GPU Architecture
- 1982, Why systolic architectures?, 15, 3746
- Dao, 2023, Flashdecoding for Long-Context Inference
- Vaidya, 2023, Optimizing Inference on Large Language Models With NVIDIA TensorRT-LLM, Now Publicly Available
- Lefaudeux, 2022, xformers: A Modular and Hackable Transformer Modelling Library
- Yu, Orca: A distributed serving system for transformer-based generative models, 521
- 2024, Lightllm
- Jin, s3: Increasing GPU utilization during generative inference for higher throughput, 36, 18015
- Jiang, 2024, Mixtral of experts, arXiv:2401.04088 · arXiv
- Du, SiDA-MoE: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models, 6, 224
- Gupta, 2023, FLuRKA: Fast and accurate unified low-rank & kernel attention, arXiv:2306.15799 · arXiv
- Zhang, 2023, Draft & verify: Lossless large language model acceleration via self-speculative decoding, arXiv:2309.08168 · arXiv
- Liu, 2024, LLaVA-NeXT: Improved Reasoning, OCR, and World Knowledge
- Dettmers, GPT3.int8(): 8-bit matrix multiplication for transformers at scale, 35, 30318
- Tu, A 28 nm $15.59 \mu\text{J}$ /token full-digital bitline-transpose CIM-based sparse transformer accelerator with pipeline/parallel reconfigurable modes, 466
- Tambe, A 12 nm 18.1TFLOPs/W sparse transformer processor with entropy-based early exit, mixed-precision predication and fine-grained power management, 342
- Amirshahi, TiC-SAT: tightly-coupled systolic accelerator for transformers, 657
- Imani, FloatPIM: in-memory acceleration of deep neural network training with high precision, 802
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Self-citationAI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [116,117].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [116,117].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
A Comprehensive Survey on Tiny Machine Learning for Human Behavior Analysis
Lamaakal, I., Essahraui, S., Maleh, Y., Makkaoui, K. E., Ouahbi, I., Bouami, M. F., El‐Latif, A. A. A., Almousa, M., Peng, J., & Niyato, D. T. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3565688 · Cited by 76
“Unlike TinyML, which prioritizes efficiency and low power usage, LLMs require extensive computational resources, high-end GPUs or TPUs, and vast amounts of training data [199].”
Metadata
- Authors
- Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., Shan, H., Sun, J., Wang, Y., Wei, C., Wu, X., Wu, Y., Yang, H. F., Zhang, J., Zhang, J., Zheng, Q., Zhou, G., Li, H., & Chen, Y.
- Affiliations
- Duke University (US)
- ORCID
- Venue
- IEEE Circuits and Systems Magazine
- Published
- 2025
- Volume
- 25
- Issue
- 1
- Pages
- 35-57
- ISSN
- 0163-6812 (ISSN-L)
- 1531-636X
- 1558-0830
- OpenAlex
- W4407212618
- Cited by
- 32
- References
- 212
- Citations by year
- 2026: 12 · 2025: 20
- FWCI
- 21.86
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Circuits and Systems Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Keywords
- large language models, hardware-software co-design, energy efficiency, algorithm optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.