Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 215 of the 215 references it cites were found in a registry or an index.

A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models

Open access

Abstract

The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.

Overview

Integrity signals

Registry

Crossref · 10.1109/mcas.2024.3476008

Cites

215 of 215 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

  • Deep Residual Learning for Image Recognition

    He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×

    “Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.”
    Self-citation
  • Attention Is All You Need

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×

    “The backbone of modern LLMs is the transformer architecture, which has revolutionized NLP by addressing the limitations of previous recurrent and convolutional models.”
  • Exploiting Generative AI to Scale up Intelligent Tutoring Systems

    Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×

    “Encoder-decoder and Encoder-only models are considered BERT-style models [15], while Decoder-only models are termed GPT-style models [1].”
    Method
  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×

    “Encoder-decoder and Encoder-only models are considered BERT-style models [15], while Decoder-only models are termed GPT-style models [1].”
    Method
  • Language Models are Few-Shot Learners

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 6×

    “The landscape changed significantly after 2021 with the introduction of transformative models like GPT-3 [4], which led to a surge in the adoption of decoder-only architectures.”
    Influential
  • Learning Transferable Visual Models From Natural Language Supervision

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×

    “The construction of these multi-modal LLMs follows a well-established recipe.”
  • LoRA: Low-Rank Adaptation of Large Language Models

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×

    “LoRA, or Low-Rank Adaptation, is a technique designed to fine-tune large pre-trained models in a pa-rameter-efficient manner [36].”
  • TensorFlow: A system for large-scale machine learning

    Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×

    “In 2020, OpenAI’s introduction of the “scaling law” [17] highlighted that model performance is strongly correlated with size.”

Show all 212 references

Research works printed but not linked: 23

  1. Radford, 2018, Improving language understanding by generative pre-training
  2. 2023, GPT-4 technical report
  3. Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
  4. Ren, Zero-offload: Democratizing billionscale model training, 551
  5. 2020, NVIDIA A100 Tensor Core GPU Architecture
  6. 2022, NVIDIA H100 Tensor Core GPU Architecture
  7. 1982, Why systolic architectures?, 15, 3746
  8. Dao, 2023, Flashdecoding for Long-Context Inference
  9. Vaidya, 2023, Optimizing Inference on Large Language Models With NVIDIA TensorRT-LLM, Now Publicly Available
  10. Lefaudeux, 2022, xformers: A Modular and Hackable Transformer Modelling Library
  11. Yu, Orca: A distributed serving system for transformer-based generative models, 521
  12. 2024, Lightllm
  13. Jin, s3: Increasing GPU utilization during generative inference for higher throughput, 36, 18015
  14. Jiang, 2024, Mixtral of experts, arXiv:2401.04088 · arXiv
  15. Du, SiDA-MoE: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models, 6, 224
  16. Gupta, 2023, FLuRKA: Fast and accurate unified low-rank & kernel attention, arXiv:2306.15799 · arXiv
  17. Zhang, 2023, Draft & verify: Lossless large language model acceleration via self-speculative decoding, arXiv:2309.08168 · arXiv
  18. Liu, 2024, LLaVA-NeXT: Improved Reasoning, OCR, and World Knowledge
  19. Dettmers, GPT3.int8(): 8-bit matrix multiplication for transformers at scale, 35, 30318
  20. Tu, A 28 nm $15.59 \mu\text{J}$ /token full-digital bitline-transpose CIM-based sparse transformer accelerator with pipeline/parallel reconfigurable modes, 466
  21. Tambe, A 12 nm 18.1TFLOPs/W sparse transformer processor with entropy-based early exit, mixed-precision predication and fine-grained power management, 342
  22. Amirshahi, TiC-SAT: tightly-coupled systolic accelerator for transformers, 657
  23. Imani, FloatPIM: in-memory acceleration of deep neural network training with high precision, 802

Cited by

See all 37

Metadata

Authors
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., Shan, H., Sun, J., Wang, Y., Wei, C., Wu, X., Wu, Y., Yang, H. F., Zhang, J., Zhang, J., Zheng, Q., Zhou, G., Li, H., & Chen, Y.
Affiliations
Duke University (US)
Venue
IEEE Circuits and Systems Magazine
Published
2025
Volume
25
Issue
1
Pages
35-57
ISSN
  • 0163-6812 (ISSN-L)
  • 1531-636X
  • 1558-0830
OpenAlex
W4407212618
Cited by
32
References
212
Citations by year
2026: 12 · 2025: 20
FWCI
21.86
Type
journal article
Language
en
Open access
Locations
  • IEEE Circuits and Systems Magazine (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
Keywords
large language models, hardware-software co-design, energy efficiency, algorithm optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models. https://app.cochat.ai/papers/10.1109/mcas.2024.3476008
The paper
Guo, C., Cheng, F., Du, Z., Kiessling, J., Ku, J., Li, S., Li, Z., Ma, M., Molom-Ochir, T., Morris, B., Shan, H., Sun, J., Wang, Y., Wei, C., Wu, X., Wu, Y., Yang, H. F., Zhang, J., Zhang, J., . . . Chen, Y. (2025). A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models. IEEE Circuits and Systems Magazine. https://doi.org/10.1109/mcas.2024.3476008

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.