A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Open accessThe DOI matches its Crossref record; 215 of the 215 references it cites were found in a registry or an index.
Abstract
The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, the collaborative hardware and software design is also necessary to further break the bottleneck for training and inference acceleration [153].”
From A Survey of Large Language Models (2026) · cites this paper as [153]
In the citing paper
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [118], [119].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [119]
Citations
Cites
Items 201–215 of 215
Fractal: Joint Multi-Level Sparse Pattern Tuning of Accuracy and Performance for DNN Pruning
Guan, Y., Yu, C., Zhou, Y., Leng, J., Li, C., & Guo, M. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651351 · Cited by 11
ClipFormer : Key–Value Clipping of Transformers on Memristive Crossbars for Write Noise Mitigation
Bhattacharjee, A., Moitra, A., & Panda, P. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3435762 · Cited by 11 · cited 3×
“The core innovation is the Compute-ACAM unit, which performs various non-matrix-vector-multiplication operations within the analog domain using analog content-address-able memories (CAMs), significantly improving computation efficiency.”
InfluentialGOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
Zadeh, A. H., Edo, I., Awad, O. M., & Moshovos, A. (2020) · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00071 · Cited by 10 · cited 1×
“OLAccel [158] and GOBO [158] quantizes outliers with higher precision, but these approaches often suffer from unaligned memory accesses, leading to additional overhead and limited computing speed.”
DistillSpec: Improving Speculative Decoding via Knowledge Distillation
Zhou, Y., Lyu, K., Rawat, A. S., Menon, A. K., Rostamizadeh, A., Kumar, S., Kagy, J.-F., & Agarwal, R. (2023) · arXiv · doi:10.48550/arxiv.2310.08461 · Cited by 9 · cited 1×
“The other line of research aims to tackle the problem through the lens of algorithms, improving the alignment between LLMs and small models [132], [133], [134].”
PIVOT- Input-aware Path Selection for Energy-efficient ViT Inference
Moitra, A., Bhattacharjee, A., & Panda, P. (2024) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3649329.3655679 · Cited by 9 · cited 1×
“PIVOT [205] improves transformer efficiency by dynamically adjusting attention mechanisms based on input complexity, achieving significant reductions in energy consumption.”
RACE-IT: A Reconfigurable Analog Computing Engine for In-Memory Transformer Acceleration
Zhao, L., Natarajan, A., Buonanno, L., Gajjar, A., Roth, R. M., Serebryakov, S., Moon, J., Eldash, O., Ignowski, J., & Pedretti, G. (2025) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd65941.2025.00021 · Cited by 9
Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads
Hu, C., Huang, H., Xu, L., Chen, X., Xu, J., Chen, S., Feng, H., Wang, C., Wang, S., Bao, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.11181 · Cited by 8 · cited 1×
“SpotServe [88] addresses the challenges of using cloud-based preemptible GPU resources by implementing strategies for dynamic adjustment and state recovery, ensuring robust service Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Kar, M., Silberman, J. A., Venkataramani, S., Srinivasan, V., Fleischer, B., Rubin, J., Lancaster, J., Lee, S., Cohen, M., Ziegler, M. M., et al. (2024) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49657.2024.10454301 · Cited by 8 · cited 1×
“It is designed for scalability and sustainability through end-to-end system optimization.”
Self-citationMixture-of-Depths: Dynamically allocating compute in transformer-based language models
Raposo, D., Ritter, S., Richards, B., Lillicrap, T., Humphreys, P. C., & Santoro, A. (2024) · arXiv · doi:10.48550/arxiv.2404.02258 · Cited by 8 · cited 1×
“Mixture-of-Depth (MoD) [104] explores skipping layers in transformer models.”
Splitwise: Efficient generative LLM inference using phase splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2023) · arXiv · doi:10.48550/arxiv.2311.18677 · Cited by 6 · cited 1×
“SpotServe [88] addresses the challenges of using cloud-based preemptible GPU resources by implementing strategies for dynamic adjustment and state recovery, ensuring robust service Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
TReX- Reusing Vision Transformer’s Attention for Efficient Xbar-Based Computing
Moitra, A., Bhattacharjee, A., Kim, Y., & Panda, P. (2024) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2024.3480524 · Cited by 4 · cited 2×
“TReX [220] propose a novel approach to optimize Transform-ers for In-Memory Computing architectures by reusing attention blocks, leading to significant improvements in energy efficiency and area utilization while maintaining The core innovation is the Compute-ACAM unit, which performs various…”
RACE-IT: A Reconfigurable Analog Computing Engine for In-Memory Transformer AccelerationPreprint
Zhao, L., Natarajan, A., Buonanno, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.06532 · Cited by 3
PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels
Kacham, P., Mirrokni, V., & Zhong, P. (2023) · arXiv · doi:10.48550/arxiv.2310.01655 · Cited by 2
Structured Pruning AdaptersPreprint
Hedegaard, L., Alok, A., Jose, J., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.10155 · Cited by 1 · cited 1×
“SPLoRA [37] and LoRAPrune [38] both leverage structured channel-pruning to remove groups of weights in order to increase the computational efficiency.”
BackgroundFairness in Serving Large Language Models
Sheng, Y., Cao, S., Li, D., Zhu, B., Li, Z., Zhuo, D., Gonzalez, J. E., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2401.00588 · cited 1×
“Additionally, VTC [87] introduces a fairness-ori-ented scheduling model that adjusts resource allocation based on the workload of incoming requests, ensuring a balanced service across different users.”
Self-citation
Research works printed but not linked: 23
- Radford, 2018, Improving language understanding by generative pre-training
- 2023, GPT-4 technical report
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Ren, Zero-offload: Democratizing billionscale model training, 551
- 2020, NVIDIA A100 Tensor Core GPU Architecture
- 2022, NVIDIA H100 Tensor Core GPU Architecture
- 1982, Why systolic architectures?, 15, 3746
- Dao, 2023, Flashdecoding for Long-Context Inference
- Vaidya, 2023, Optimizing Inference on Large Language Models With NVIDIA TensorRT-LLM, Now Publicly Available
- Lefaudeux, 2022, xformers: A Modular and Hackable Transformer Modelling Library
- Yu, Orca: A distributed serving system for transformer-based generative models, 521
- 2024, Lightllm
- Jin, s3: Increasing GPU utilization during generative inference for higher throughput, 36, 18015
- Jiang, 2024, Mixtral of experts, arXiv:2401.04088 · arXiv
- Du, SiDA-MoE: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models, 6, 224
- Gupta, 2023, FLuRKA: Fast and accurate unified low-rank & kernel attention, arXiv:2306.15799 · arXiv
- Zhang, 2023, Draft & verify: Lossless large language model acceleration via self-speculative decoding, arXiv:2309.08168 · arXiv
- Liu, 2024, LLaVA-NeXT: Improved Reasoning, OCR, and World Knowledge
- Dettmers, GPT3.int8(): 8-bit matrix multiplication for transformers at scale, 35, 30318
- Tu, A 28 nm $15.59 \mu\text{J}$ /token full-digital bitline-transpose CIM-based sparse transformer accelerator with pipeline/parallel reconfigurable modes, 466
- Tambe, A 12 nm 18.1TFLOPs/W sparse transformer processor with entropy-based early exit, mixed-precision predication and fine-grained power management, 342
- Amirshahi, TiC-SAT: tightly-coupled systolic accelerator for transformers, 657
- Imani, FloatPIM: in-memory acceleration of deep neural network training with high precision, 802
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Self-citationAI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [116,117].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [116,117].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
A Comprehensive Survey on Tiny Machine Learning for Human Behavior Analysis
Lamaakal, I., Essahraui, S., Maleh, Y., Makkaoui, K. E., Ouahbi, I., Bouami, M. F., El‐Latif, A. A. A., Almousa, M., Peng, J., & Niyato, D. T. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3565688 · Cited by 76
“Unlike TinyML, which prioritizes efficiency and low power usage, LLMs require extensive computational resources, high-end GPUs or TPUs, and vast amounts of training data [199].”
Metadata
- Authors
- Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., Shan, H., Sun, J., Wang, Y., Wei, C., Wu, X., Wu, Y., Yang, H. F., Zhang, J., Zhang, J., Zheng, Q., Zhou, G., Li, H., & Chen, Y.
- Affiliations
- Duke University (US)
- ORCID
- Venue
- IEEE Circuits and Systems Magazine
- Published
- 2025
- Volume
- 25
- Issue
- 1
- Pages
- 35-57
- ISSN
- 0163-6812 (ISSN-L)
- 1531-636X
- 1558-0830
- OpenAlex
- W4407212618
- Cited by
- 32
- References
- 212
- Citations by year
- 2026: 12 · 2025: 20
- FWCI
- 21.86
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Circuits and Systems Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Keywords
- large language models, hardware-software co-design, energy efficiency, algorithm optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.