CoChat

A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models

Open access

The DOI matches its Crossref record; 215 of the 215 references it cites were found in a registry or an index.

Abstract

The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.

Overview

Integrity signals

Registry

Crossref · 10.1109/mcas.2024.3476008

Cites

215 of 215 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 23

  1. Radford, 2018, Improving language understanding by generative pre-training
  2. 2023, GPT-4 technical report
  3. Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
  4. Ren, Zero-offload: Democratizing billionscale model training, 551
  5. 2020, NVIDIA A100 Tensor Core GPU Architecture
  6. 2022, NVIDIA H100 Tensor Core GPU Architecture
  7. 1982, Why systolic architectures?, 15, 3746
  8. Dao, 2023, Flashdecoding for Long-Context Inference
  9. Vaidya, 2023, Optimizing Inference on Large Language Models With NVIDIA TensorRT-LLM, Now Publicly Available
  10. Lefaudeux, 2022, xformers: A Modular and Hackable Transformer Modelling Library
  11. Yu, Orca: A distributed serving system for transformer-based generative models, 521
  12. 2024, Lightllm
  13. Jin, s3: Increasing GPU utilization during generative inference for higher throughput, 36, 18015
  14. Jiang, 2024, Mixtral of experts, arXiv:2401.04088 · arXiv
  15. Du, SiDA-MoE: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models, 6, 224
  16. Gupta, 2023, FLuRKA: Fast and accurate unified low-rank & kernel attention, arXiv:2306.15799 · arXiv
  17. Zhang, 2023, Draft & verify: Lossless large language model acceleration via self-speculative decoding, arXiv:2309.08168 · arXiv
  18. Liu, 2024, LLaVA-NeXT: Improved Reasoning, OCR, and World Knowledge
  19. Dettmers, GPT3.int8(): 8-bit matrix multiplication for transformers at scale, 35, 30318
  20. Tu, A 28 nm $15.59 \mu\text{J}$ /token full-digital bitline-transpose CIM-based sparse transformer accelerator with pipeline/parallel reconfigurable modes, 466
  21. Tambe, A 12 nm 18.1TFLOPs/W sparse transformer processor with entropy-based early exit, mixed-precision predication and fine-grained power management, 342
  22. Amirshahi, TiC-SAT: tightly-coupled systolic accelerator for transformers, 657
  23. Imani, FloatPIM: in-memory acceleration of deep neural network training with high precision, 802

Cited by

See all 37

Metadata

Authors
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., Shan, H., Sun, J., Wang, Y., Wei, C., Wu, X., Wu, Y., Yang, H. F., Zhang, J., Zhang, J., Zheng, Q., Zhou, G., Li, H., & Chen, Y.
Affiliations
Duke University (US)
Venue
IEEE Circuits and Systems Magazine
Published
2025
Volume
25
Issue
1
Pages
35-57
ISSN
  • 0163-6812 (ISSN-L)
  • 1531-636X
  • 1558-0830
OpenAlex
W4407212618
Cited by
32
References
212
Citations by year
2026: 12 · 2025: 20
FWCI
21.86
Type
journal article
Language
en
Open access
Locations
  • IEEE Circuits and Systems Magazine (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
Keywords
large language models, hardware-software co-design, energy efficiency, algorithm optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.