A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Open accessThe DOI matches its Crossref record; 215 of the 215 references it cites were found in a registry or an index.
Abstract
The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, the collaborative hardware and software design is also necessary to further break the bottleneck for training and inference acceleration [153].”
From A Survey of Large Language Models (2026) · cites this paper as [153]
In the citing paper
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [118], [119].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [119]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.”
Self-citationVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“The backbone of modern LLMs is the transformer architecture, which has revolutionized NLP by addressing the limitations of previous recurrent and convolutional models.”
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“Encoder-decoder and Encoder-only models are considered BERT-style models [15], while Decoder-only models are termed GPT-style models [1].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Encoder-decoder and Encoder-only models are considered BERT-style models [15], while Decoder-only models are termed GPT-style models [1].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 6×
“The landscape changed significantly after 2021 with the introduction of transformative models like GPT-3 [4], which led to a surge in the adoption of decoder-only architectures.”
InfluentialLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“The construction of these multi-modal LLMs follows a well-established recipe.”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“LoRA, or Low-Rank Adaptation, is a technique designed to fine-tune large pre-trained models in a pa-rameter-efficient manner [36].”
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×
“In 2020, OpenAI’s introduction of the “scaling law” [17] highlighted that model performance is strongly correlated with size.”
Research works printed but not linked: 23
- Radford, 2018, Improving language understanding by generative pre-training
- 2023, GPT-4 technical report
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Ren, Zero-offload: Democratizing billionscale model training, 551
- 2020, NVIDIA A100 Tensor Core GPU Architecture
- 2022, NVIDIA H100 Tensor Core GPU Architecture
- 1982, Why systolic architectures?, 15, 3746
- Dao, 2023, Flashdecoding for Long-Context Inference
- Vaidya, 2023, Optimizing Inference on Large Language Models With NVIDIA TensorRT-LLM, Now Publicly Available
- Lefaudeux, 2022, xformers: A Modular and Hackable Transformer Modelling Library
- Yu, Orca: A distributed serving system for transformer-based generative models, 521
- 2024, Lightllm
- Jin, s3: Increasing GPU utilization during generative inference for higher throughput, 36, 18015
- Jiang, 2024, Mixtral of experts, arXiv:2401.04088 · arXiv
- Du, SiDA-MoE: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models, 6, 224
- Gupta, 2023, FLuRKA: Fast and accurate unified low-rank & kernel attention, arXiv:2306.15799 · arXiv
- Zhang, 2023, Draft & verify: Lossless large language model acceleration via self-speculative decoding, arXiv:2309.08168 · arXiv
- Liu, 2024, LLaVA-NeXT: Improved Reasoning, OCR, and World Knowledge
- Dettmers, GPT3.int8(): 8-bit matrix multiplication for transformers at scale, 35, 30318
- Tu, A 28 nm $15.59 \mu\text{J}$ /token full-digital bitline-transpose CIM-based sparse transformer accelerator with pipeline/parallel reconfigurable modes, 466
- Tambe, A 12 nm 18.1TFLOPs/W sparse transformer processor with entropy-based early exit, mixed-precision predication and fine-grained power management, 342
- Amirshahi, TiC-SAT: tightly-coupled systolic accelerator for transformers, 657
- Imani, FloatPIM: in-memory acceleration of deep neural network training with high precision, 802
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Self-citationAI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [116,117].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
“This architectural evolution marks a critical shift in AI system design: from content creation to autonomous task execution [116,117].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
A Comprehensive Survey on Tiny Machine Learning for Human Behavior Analysis
Lamaakal, I., Essahraui, S., Maleh, Y., Makkaoui, K. E., Ouahbi, I., Bouami, M. F., El‐Latif, A. A. A., Almousa, M., Peng, J., & Niyato, D. T. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3565688 · Cited by 76
“Unlike TinyML, which prioritizes efficiency and low power usage, LLMs require extensive computational resources, high-end GPUs or TPUs, and vast amounts of training data [199].”
LAERACE: Taking the policy fast-track towards low-altitude economy
Sun, X., Wang, S., Zhang, X., & Wandelt, S. (2025) · Journal of the Air Transport Research Society · doi:10.1016/j.jatrs.2025.100058 · Cited by 77
M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type
Hu, W., Zhang, H., Guo, C., Yu, F., Guan, R., Hua, Z., Liu, Z., Guan, Y., Guo, M., & Leng, J. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00086 · Cited by 46
“The memory footprint of generative LLM inference comprises two major components: the weight and the KV cache, either of which can be the memory bottleneck under various scenarios [26], [35], [38], [50], [69], [88].”
Self-citationGenAI at the Edge: Comprehensive Survey on Empowering Edge DevicesPreprint
Navardi, M., Aalishah, R., Fu, Y., Lin, Y., Li, H., Chen, Y., & Mohsenin, T. (2025) · Qeios · doi:10.1609/aaaiss.v5i1.35586 · Cited by 40
Self-citationNeuromorphic Computing in the Era of Large Models
Shan, H., Wei, C., Ramos, N., Yang, X., Guo, C., Li, H., & Chen, Y. (2025) · Artificial Intelligence Science and Engineering · doi:10.23919/aise.2025.000002 · Cited by 28
Self-citationA Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency
Park, S., Jeon, S., Lee, C., Jeon, S., Kim, B.-S., & Lee, J. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3803798 · Cited by 47
Technology Roadmap of Bioinspired Computing Hardware
Wang, S., Li, Z., Pei, M., Ren, Q., Deng, M., Kah-Wee, A., Ascoli, A., Banerjee, S., Bonnin, M., van de Burgt, Y., et al. (2026) · ACS Nano · doi:10.1021/acsnano.5c17087 · Cited by 20
Self-citationProsperity: Accelerating Spiking Neural Networks via Product Sparsity
Wei, C., Guo, C., Cheng, F., Li, S., Yang, H. F., Li, H. H., & Chen, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00066 · Cited by 20
Self-citationEcco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-Aware Cache Compression
Cheng, F., Guo, C., Wei, C., Zhang, J., Zhou, C., Hanson, E., Zhang, J., Liu, X., Li, H. H., & Chen, Y. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731024 · Cited by 29
Self-citationA survey on memory-efficient transformer-based model training in AI for science
Tian, K., Qiao, L., Liu, B., Jiang, G., Li, S., & Li, D. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50302-6 · Cited by 16
“Meanwhile, Guo et al. [129] present a comprehensive survey of LLM-centric co-design, outlining the challenges posed by large model size, energy consumption, and memory bottlenecks.”
Self-citationPhi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networks
Wei, C., Duan, B., Guo, C., Zhang, J., Song, Q., Li, H. H., & Chen, Y. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731035 · Cited by 16
Self-citationVeriOpt: PPA-Aware High-Quality Verilog Generation via Multi-Role LLMs
Tasnia, K., García, A., Farheen, T., & Rahman, M. S. (2025) · doi:10.1109/iccad66269.2025.11240771 · Cited by 33
“Despite their great promises, these methods produce partially functional codes [22] which are difficult to debug and often overlooks scalability, interpretability, synthesizability, and most importantly, power, performance, and area (PPA) optimization, critical metrics for industrial-grade designs…”
InfluentialTransitive Array: An Efficient GEMM Accelerator with Result Reuse
Guo, C., Wei, C., Tang, J., Duan, B., Han, S., Li, H. H., & Chen, Y. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731043 · Cited by 12
Self-citationAlqahtani, H., & Kumar, G. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.067738 · Cited by 8
Sivasubramani, S., Kishore, C., Acharyya, A., Georgiev, V., Shafik, R., & Prodromakis, T. (2025) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2025.3587613 · Cited by 3
Joint Hardware-Workload Co-Optimization for In-Memory Computing Accelerators
Krestinskaya, O., Fouda, M. E., Eltawil, A. M., & Salama, K. N. (2026) · IEEE Access · doi:10.1109/access.2026.3671360 · Cited by 2
“For IMC specifically, achieving high efficiency requires adapting neural network models to the constraints and non-idealities of IMC hardware, while simultaneously optimizing the hardware to efficiently support target workloads [14], [41].”
Platinum: Path-Adaptable LUT-Based Accelerator Tailored for Low-Bit Weight Matrix Multiplication
Shan, H., Guo, C., Wei, C., Cheng, F., Zhang, J., Li, H. H., & Chen, Y. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420289 · Cited by 4
Self-citationMSLoRA: Meta-learned scaling for adaptive fine-tuning of LoRA
Dan, L., Zheng, K., Wu, C., & Wang, X. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130374 · Cited by 5
lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models
Wang, H., Tu, X., Ke, H., Chai, H., Chen, D., & Han, K. (2025) · ACM/IEEE Symposium on Edge Computing (SEC) · doi:10.1145/3769102.3770614 · Cited by 6
Evaluating Large Language Models: Challenges, Limitations, and Future Directions
Li, L.-H., & Nugroho, A. C. (2025) · Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON) · doi:10.1109/ectidamtncon64748.2025.10961969 · Cited by 3
“Larger LLMs have been easier to train with to hardware developments like GPUs and TPUs [24][25], and their performance and generalization abilities have been greatly improved by the availability of big and varied datasets [26].”
Al Compute Architecture and Evolution Trends [Feature]
Liang, B.-S. (2026) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2025.3650725 · Cited by 4
Large language models in NLP: evolution, architectural trends, and open challenges
Javed, H., Shah, B., Ali, F., & Kwak, D. (2026) · Journal Of Big Data · doi:10.1186/s40537-026-01429-1 · Cited by 1
“By 2024, models like LLaMA 2 and Falcon were predicted to provide advanced benchmarks in affordability and accessibility in AI, and in turn, predicted to revolutionize interactive AI, content automation, and even assist in medical diagnosis [47].”
Multimodal and Distributed LLMs: Bridging Scalability and Cross-Modal ReasoningPreprint
Kumar, R., Laurent, I., Müller, D., & Elli, K. (2025) · Preprints.org · doi:10.20944/preprints202505.1156.v1 · Cited by 3
From Centralized to Composable: Advances in Distributed and Multimodal Language ModelingPreprint
Mirta, I., & Elli, K. (2025) · Engineering Archive (engrXiv) · doi:10.31224/4624
Supporting Stakeholder Management with Natural Language Processing and Large Language Models
Alves, J. R., Chaves, M. S., & Simas, M. A. F. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-97799-2_18
Latent Diffusion Model Assisted Image Deblurring
Wu, C.-C., & Tang, C.-W. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3644426
“If an image deblurring model uses the LDM during both training and inference, specialized accelerators have to be designed for the LDM, as in the case of large language models [36].”
A Review of FPGA-Driven LLM Acceleration
Fu, Y., Li, J., Cheng, C., Ma, S. L., Sham, C., & Zou, N. (2025) · IEEE International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC) · doi:10.1109/mcsoc67473.2025.00018
“In recent studies, Guo et al. [7] explored the field from the broad perspective of hardware-software co-design, while Li et al. [8] covered a wide spectrum of platforms ranging from CPUs to ASICs.”
Pongiannan, R. K., Rajasekar, L., S., S. G., Ruchi, Chauhan, S., & Maddileti, T. (2025) · International Conference on Ubiquitous Computing and Intelligent Information Systems (ICUIS) · doi:10.1109/icuis67429.2025.11380536
“Domain-specific architectures (DSAs) [5], along with application-specific integrated circuits (ASICs), would certainly provide some degree of performance-per-watt enhancement, at least for some of the smaller class of workloads [6].”
Secure eFPGA-Enabled Edge LLM Inference: Architectural and Hardware Countermeasures
Das, V., Khan, M. Z. S., Vafaei, J., Azar, K. Z., & Kamali, H. M. (2026) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts69484.2026.11563371
“Recent work shows that advances in LLM capabilities are closely tied to specialized hardware [9].”
Generative AI at the Edge: A Comprehensive Survey of Architectures, Hardware and Applications
Navardi, M., Fu, Y., Lin, Y., Li, H. H., Chen, Y., & Mohsenin, T. (2026) · ACM Computing Surveys · doi:10.1145/3829079 · Cited by 3
Self-citationDeep learning for anomaly detection in railway systems: A structured survey
Bouketta, A., Niar, S., & Ouarnoughi, H. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115776
AI-Powered Automated Verilog Generation and Simulation for Semiconductor Design
Syed Ali Ansari, A., Jones, A., Monish Kumar, S., Maheshwar Siva, M., & Angel Felciya, I. (2026) · International Conference on Data Science and Business Systems (ICDSBS) · doi:10.1109/icdsbs69077.2026.11635016
Amiri, M., Mokari, A., & Bocklitz, T. W. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02369-8
“LLM training and inference and neuro-symbolic workloads are therefore often described as being dominated by data movement and input/output operations, although some purely neural components remain compute-bound [54, 60, 61].”
Metadata
- Authors
- Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., Shan, H., Sun, J., Wang, Y., Wei, C., Wu, X., Wu, Y., Yang, H. F., Zhang, J., Zhang, J., Zheng, Q., Zhou, G., Li, H., & Chen, Y.
- Affiliations
- Duke University (US)
- ORCID
- Venue
- IEEE Circuits and Systems Magazine
- Published
- 2025
- Volume
- 25
- Issue
- 1
- Pages
- 35-57
- ISSN
- 0163-6812 (ISSN-L)
- 1531-636X
- 1558-0830
- OpenAlex
- W4407212618
- Cited by
- 32
- References
- 212
- Citations by year
- 2026: 12 · 2025: 20
- FWCI
- 21.86
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Circuits and Systems Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
- Keywords
- large language models, hardware-software co-design, energy efficiency, algorithm optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.