A Survey on Efficient Training of Transformers

May combine two claims.

Lion generally converges faster and is more memory-efficient and accurate than AdamW for training Transformers on various benchmarks.

“In practice, Lion in general converges faster, and is more memory-efficient and accurate than AdamW for training Transformers on various benchmarks.”

• cites Chen et al., 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Training a Transformer base model with neural architecture search produces roughly 626,155 pounds of carbon dioxide, comparable to the lifetime emissions of five cars.

“A widely discussed energy study of deep learning models [Strubell et al., 2019] estimates that training a Transformer base model with neural architecture search (NAS) [So et al., 2019] produces about 626,155 pounds of planet-warming carbon dioxide, equal to the lifetime emissions of five cars”

• cites Strubell et al., 2019Not checked yet

  Not checked in the free run.

• cites So et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

AdamW is the most widely used optimizer for Transformers by default.

“By default, AdamW [Loshchilov and Hutter, 2019], a variant of Adam which decouples the L 2 regularization and the weight decay, is the most widely used optimizer for Transformers.”

• cites Loshchilov and Hutter, 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-3 was trained on half a trillion words and has 175 billion parameters.

“GPT-3 [Brown et al., 2020] (the precursor to ChatGPT) was trained on half a trillion words and equips with 175 billion parameters.”

• cites Brown et al., 2020Not checked yet

  Not checked in the free run.

May combine two claims.

LTH requires identifying winning tickets through alternating train-prune-retrain, which makes training very costly for large models and datasets.

“However, LTH requires identifying the winning tickets in an alternating train-prune-retrain manner, which makes the training extremely costly for large models and datasets, limiting the practical benefits.”

• cites Frankle and Carbin, 2019Not checked yet

  Not checked in the free run.

May combine two claims.

ZeRO combines uniform partitioning with data parallelism, where each data parallel process handles one partition of model states in a mixed precision regime.

“More recently, ZeRO [Rajbhandari et al., 2020] incorporates uniform partitioning strategy with DP, where each data parallel process merely deals with one partition of model states, working in a mixed precision regime.”

• cites Rajbhandari et al., 2020Not checked yet

  Not checked in the free run.

May combine two claims.

Fixup rescales standard initialization to maintain proper gradient norm and can train very deep networks with over 10,000 layers without adding normalization layers.

“Fixup [Zhang et al., 2019a] proposes to properly rescale a standard initialization to ensure proper gradient norm to avoid exploding or vanishing gradients, which can train very deep networks with over 10,000 layers without adding normalization layers.”

• cites Zhang et al., 2019aNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ConViT controls locality by adding a soft gating parameter that balances content and positional terms, incorporating soft convolutional inductive biases into global self-attention.

“ConViT [d' Ascoli et al., 2021] learns to control the locality by adding a soft gating parameter to balance the two terms, which has the effect of incorporating soft convolutional inductive biases into global self-attention.”

• cites d' Ascoli et al., 2021Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Averaging the weights of a certain number of latest checkpoints can facilitate faster training.

“[Kaddour, 2022] shows that averaging the weights of a certain number of latest checkpoints can facilitate faster training.”

• cites Kaddour, 2022Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Sparse training directly trains sparse subnetworks from scratch instead of full networks without sacrificing accuracy.

“The key idea of sparse training is to directly train sparse subnetworks instead of the full networks from scratch without sacrificing accuracy.”

May combine two claims.

Overparameterization empirically improves both convergence and generalization, although theoretical guarantees are not sufficient.

“It is observed that overparameterization empirically improves both convergence and generalization, with theoretical guarantee though not sufficient.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Sharpness-aware minimization aims to minimize both loss value and loss sharpness, based on the idea that a flatter minimum tends to improve generalization.

“Sharpnessaware minimization (SAM) [Foret et al., 2021] seeks to simultaneously minimize loss value and loss sharpness, based on the connection between the geometry of the loss landscape and generalization, i.e., a flatter minimum tends to improve generalization.”

• cites Foret et al., 2021Not checked yet

  Not checked in the free run.

May combine two claims.

Progressively increasing input image resolution, by reducing patch size, can significantly accelerate ViT training, consistent with training dynamics that focus on low-frequency structure early and high-frequency semantics later.

“A key observation is that progressively increasing the resolution of the input images (reducing the patch size) can significantly accelerate ViT training, aligning with the widely known training dynamics that focus on low-frequency structure in the early stage and high-frequency semantics in the latter stage.”

• cites Li et al., 2022aNot checked yet

  Not checked in the free run.

May combine two claims.

Parameter sharing, known as Albert, improves the scalability of data parallelism for Transformers but limits representational power.

“To improve the scalability of DP, one solution for Transformer is parameter sharing [Lan et al., 2020], known as Albert, but it limits the representational power.”

• cites Lan et al., 2020Not checked yet

  Not checked in the free run.

May combine two claims.

AMP keeps a full-precision master copy of weights for updates while storing activations, gradients and weights in FP16 for arithmetic.

“Specifically, AMP stores a master copy of weights in full-precision for updates while the activations, gradients and weights are stored in FP16 for arithmetic.”

• cites Micikevicius et al., 2018Not checked yet

  Not checked in the free run.

May combine two claims.

Skipping processing of masked tokens gives considerable training efficiency gains for MLM and MIM because reducing sequence length lowers computational and memory complexity quadratically.

“Since squeezing the sequence length reduces both the computational and memory complexity quadratically, skipping processing the masked tokens brings considerable training efficiency gain for MLM and MIM.”

May combine two claims.

The initialization methods discussed remove batch or layer normalization from all blocks and train without learning rate warmup.

“All the above-mentioned methods remove batch/layer normalization from all blocks and train without learning rate warmup.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LARS for SGD and LAMB for Adam propose layerwise adaptive learning rates for ResNet and Transformers respectively.

“The widely used methods are LARS [You et al., 2017] for SGD and LAMB [You et al., 2019] for Adam, which propose to use layerwise adaptive learning rates for ResNet and Transformers respectively.”

• cites You et al., 2017Not checked yet

  Not checked in the free run.

• cites You et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Training over 400 Transformer language models with varying model sizes and token counts led to the practical hypothesis that model size and number of training tokens should be scaled equally for compute-optimal LLM training.

“DeepMind in [Hoffmann et al., 2022] trains over 400 Transformer language models with varying scales of model size and # of training tokens, reaching to a practical hypothesis that the model size and the number of training tokens should be scaled equally for compute-optimal LLM training.”

• cites Hoffmann et al., 2022Not checked yet

  Not checked in the free run.

May combine two claims.

A sufficiently overparameterized three-layer neural network generalizes to the population risk, and an accurate network exists near any point on the SGD training trajectory with high probability over random initialization.

“[Allen-Zhu et al., 2019a] theoretically proves that a sufficiently overparameterized (three-layer) neural network generalizes to the population risk and an intriguing property is that there exists an accurate network in the close neighborhood of any point on the SGD training trajectory with high probability over random initialization.”

• cites Allen-Zhu et al., 2019aNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Goyal et al. used linear scaling of the learning rate to train ResNet-50 on ImageNet with batch size 8,192 in one hour.

“[Goyal et al., 2017] employs the linear scaling of the learning rate to train ResNet-50 on ImageNet with a batch size of 8,192 in 1 hour.”

• cites Goyal et al., 2017Not checked yet

  Not checked in the free run.

May combine two claims.

torchgpipe performs micro-batch pipeline parallelism with checkpointing, allowing scaling to a large number of micro-batches to minimize bubble overhead.

“In particular, PyTorch implements the torchgpipe [Kim et al., 2020], which performs micro-batch PP with checkpointing, allowing scaling to a large number of micro-batches to minimize the bubble overhead.”

• cites Kim et al., 2020Not checked yet

  Not checked in the free run.

May combine two claims.

Megatron-LM slices both multi-head self-attention and feed-forward networks across GPUs and needs only a few extra All-Reduce operations in the forward and backward pass, allowing training of models up to 8.3 billion parameters using 512 GPUs.

“Customized to Transformerbased models, Megatron-LM [Shoeybi et al., 2019] slices both MSA and FFN across GPUs and requires only a few extra All-Reduce operations in the forward and backward pass, allowing them to train models up to 8.3 billion parameters using 512 GPUs.”

• cites Shoeybi et al., 2019Not checked yet

  Not checked in the free run.

May combine two claims.

Data parallelism has high communication and computation efficiency but poor memory efficiency; when the model becomes large, a single device cannot store the model replica and synchronized gradient communications can hinder scalability.

“However, it is obvious that DP has high communication/computation efficiency but poor memory efficiency -when model becomes large, the single device cannot store the model replica and the synchronized communications for gradients can hinder the scalability of DP.”

May combine two claims.

For DNNs, a principal way to estimate per-sample importance is gradient norm, and approximations make calculating these norms tractable.

“For DNNs, a principal way of estimating per-sample importance is to use gradient norm, and [Katharopoulos and Fleuret, 2018;Johnson and Guestrin, 2018] use different approximations to make calculating these norms tractable.”

• cites Katharopoulos and Fleuret, 2018Not checked yet

  Not checked in the free run.

• cites Johnson and Guestrin, 2018Not checked yet

  Not checked in the free run.

May combine two claims.

Large batch training reduces the variance of stochastic gradient estimates, so a reliable step size must be tuned for better convergence.

“From the statistical view, large batch training reduces the variance of the stochastic gradient estimates, so a reliable step size needs to be tuned for better convergence [Bottou et al., 2018].”

• cites Bottou et al., 2018Not checked yet

  Not checked in the free run.

May combine two claims.

Model parallelism has good memory efficiency but low communication and computation efficiency due to high-volume data transfer across devices and poor processing element utilization.

“Though MP has good memory efficiency, its communication and computation efficiency is low due to high volume data transfer cross devices and poor PE utilization.”

May combine two claims.

Reduced precision training can accelerate neural network training with favorable performance.

“It has been shown in previous works that reduced precision training [Zhou et al., 2016;Hubara et al., 2017] can accelerate neural network training with favorable performance.”

• cites Zhou et al., 2016Not checked yet

  Not checked in the free run.

• cites Hubara et al., 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The scaling of test error with dataset size can break beyond power scaling laws and be reduced to at least exponential scaling if a superior data pruning metric is used.

“[Sorscher et al., 2022] shows an exciting analytic theory that the scaling of test error with dataset size can break beyond power scaling laws and be reduced to at least exponential scaling if equipped with a superior data pruning metric”

• cites Sorscher et al., 2022Not checked yet

  Not checked in the free run.

May combine two claims.

For deep vision Transformers, channel-wise learnable scaling factors and re-introducing warmup and layer normalization can make training more stable.

“On training deep vision Transformers (ViT), [Touvron et al., 2021a] proposes channel-wise learnable scaling factors and empirically ob-serve that re-introducing the warmup and layer normalization techniques can make training more stable.”

• cites Touvron et al., 2021aNot checked yet

  Not checked in the free run.

May combine two claims.

Data parallelism and model parallelism are orthogonal and can be used together to train larger models with higher computation and memory capacity.

“Note that DP and MP are orthogonal and so one can use both simultaneously to train larger models with higher computation and memory capacity.”

May combine two claims.

In vision, removing masked image patches before the encoder gives stronger performance and at least 3 times lower overall pre-training time and memory than keeping masked tokens.

“For MIM, representative work [He et al., 2022] shows that in vision, removing the masked image patches before the encoder demonstrates stronger performance and 3× or more lower overall pre-training time and memory consumption than keeping the masked tokens.”

• cites He et al., 2022Not checked yet

  Not checked in the free run.

May combine two claims.

Activation compressed training stores low-precision approximate copies of activations while computing the forward pass exactly, reducing overall memory consumption during training.

“Apart from reduced precision training which simul-taneously quantizes activations/weights/gradients, activation compressed training (ACT) [Chen et al., 2021a] stores lowprecision approximate copies of activations while computing the forward pass exactly, which helps to reduce the overall memory consumption during training.”

• cites Chen et al., 2021aNot checked yet

  Not checked in the free run.

May combine two claims.

With batch size 64 and image resolution 224 by 224, training DeiT-B on RTX3090 under AMP is 2 times faster than full-precision training, 305 versus 124 images per second, and uses 22% less peak GPU memory, 7.9GB versus 10.2GB.

“For example, based on a batch size of 64 and image resolution of 224 × 224, training a DeiT-B [Touvron et al., 2021b] on RTX3090 under AMP is 2× faster than full-precision training (305 vs. 124 images/s), as well as consuming 22% less peak GPU memory (7.9GB vs. 10.2GB).”

• cites Touvron et al., 2021bNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A tensor processing unit designed by Google delivered 30 to 80 times higher performance-per-watt than contemporary CPUs and GPUs.

“a tensor processing unit (TPU) designed by Google delivered 30∼80× higher performance-per-watt than contemporary CPUs and GPUs [Jouppi et al., 2017].”

• cites Jouppi et al., 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Higher data efficiency achieved by sampling informative training samples leads towards more efficient neural scaling laws of test error with respect to dataset size.

“2) higher data efficiency by sampling informative training samples towards more efficient neural scaling laws of test error with respect to dataset size;”

Probably not a substantial claim: may not pass S3 (load-bearing).

Hardware-algorithm co-design takes the constraints and capabilities of the hardware into account when designing the algorithm, in order to optimize the training efficiency of DNNs, especially Transformers.

“To further optimize the training efficiency of DNNs, especially Transformers, hardware-algorithm co-design takes the constraints and capabilities of the hardware into account when designing the algorithm”

Probably not a substantial claim: may not pass S4 (based).

Appropriate initialization and optimization paradigms can accelerate the convergence rate with fewer training iterations, resulting in lower computational costs.

“1) appropriate initialization and optimization paradigms that can accelerate the convergence rate with fewer training iterations, resulting in lower computational costs;”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sanger rearranges the sparse attention masks into structured blocks that are handled by reconfigurable hardware.

“Then, the sparse attention masks are rearranged into structured blocks and handled by reconfigurable hardware.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Beaumont et al. proposes jointly optimizing activation offloading and rematerialization.

“For the best of both worlds, [Beaumont et al., 2021] proposes to jointly optimize activation offloading and rematerialization.”

• cites Beaumont et al., 2021Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Rematerialization, also called checkpointing, is a widely used space-time tradeoff that stores only a portion of activations or weights during the forward pass and recomputes the rest during the backward pass.

“Rematerialization, also known as checkpointing [Chen et al., 2016], is a widely used technique for space-time tradeoff that only stores a portion of activations/weights during the forward pass and recomputes the rest during the backward pass.”

• cites Chen et al., 2016Not checked yet

  Not checked in the free run.

May combine two claims.

For applications focused on specific computation tasks, application-specific integrated circuits have the advantage of low power consumption and high training and inference speed.

“For applications that focus on specific computation tasks, application-specific integrated circuits (ASICs) have the advantage of low power consumption, and high training/inference speed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Efficiently tuning readily available pretrained models is becoming a prevailing way to drastically cut training costs.

“Efficient tuning these readily available models is becoming a prevailing way to drastically cut training costs.”

May combine two claims.

ViTCoD uses a learnable auto-encoder to compress sparse attentions into a much more compact representation and designs encoder and decoder engines to boost hardware utilization.

“ViTCoD [You et al., 2023] uses a learnable auto-encoder to compress the sparse attentions to a much more compact representation and designs encoder and decoder engines to boost the hardware utilization.”

• cites You et al., 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

ASICs are not easily reprogrammable or adaptable to a new task.

“However, ASICs are not easily reprogrammable or adaptable to a new task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Herrmann et al. implements optimal checkpointing for heterogeneous networks.

“More advanced methods such as [Herrmann et al., 2019] implements optimal checkpointing for heterogeneous networksfoot_2 .”

• cites Herrmann et al., 2019Not checked yet

  Not checked in the free run.

May combine two claims.

Lowering the precision of computations reduces the amount of memory and computation and can be implemented in hardware-friendly fixed-point or integer representations instead of floating-point ones.

“Lowering the precision of the computations reduces the amount of memory and computation, which can be implemented in hardware-friendly fixed-point or integer representations instead of floating-point ones.”

Probably not a substantial claim: may not pass S3 (load-bearing).

FlashAttention is becoming a default fast and memory-efficient attention module for speedup.

“which is becoming a default fast and memory-efficient attention module for speedup.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Compared with CPUs, GPUs are more powerful at performing matrix multiplication because of their high degree of parallelism.

“compared with central processing units (CPUs), graphics processing units (GPUs) are more powerful to perform matrix multiplication due to the high degree of parallelism.”

May combine two claims.

Reparameterization-based parameter-efficient tuning tunes parameters inherently in the model or new parameters that can be reparameterized into the model, yielding no sacrifice on inference efficiency.

“To tackle this challenge, the latter proposes to tune parameters that are inherently in the model [Zaken et al., 2022] or new parameters that can be reparameterized into the model [Hu et al., 2022], thereby yielding no sacrifice on the inference efficiency.”

• cites Zaken et al., 2022Not checked yet

  Not checked in the free run.

• cites Hu et al., 2022Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ZeRO-Offload offloads all fp32 model states and fp16 gradients to CPU memory, computes fp32 parameter updates on CPU, keeps fp16 parameters on GPU, and performs forward and backward computations on GPU.

“It offloads all fp32 model states and the fp16 gradients on the CPU memory, and computes the fp32 parameter updates on CPU.The fp16 parameters are kept on GPU and the forward and backward computations are on GPU.”

• cites Ren et al., 2021Not checked yet

  Not checked in the free run.

May combine two claims.

OuterSPACE transforms matrix multiplication into an outer product procedure and eliminates redundant memory accesses by decoupling multiplication from accumulation.

“OuterSPACE [Pal et al., 2018] transforms matrix multiplication into an outer product procedure and eliminates redundant memory accesses by decoupling multiplication from accumulation.”

• cites Pal et al., 2018Not checked yet

  Not checked in the free run.

May combine two claims.

OuterSPACE builds a custom accelerator with a reconfigurable memory hierarchy and achieves a mean speedup of 7.9 times over a CPU running Intel Math Kernel Library and 14.0 times against a GPU running CUSP.

“OuterSPACE builds a custom accelerator with reconfigurable memory hierarchy and achieves a mean speedup of 7.9× over the CPU running Intel Math Kernel Library and 14.0× against the GPU running CUSP.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Stanford Alpaca is finetuned from the open-sourced LLaMA models using 52K instruction-following data generated from ChatGPT.

“which is finetuned from the open-sourced LLaMA models [Touvron et al., 2023] using the 52K instruction-following data generated from ChatGPT.”

• cites Touvron et al., 2023Not checked yet

  Not checked in the free run.

May combine two claims.

These large-scale training frameworks support data and model parallelism in mixed precision, along with offloading and rematerialization.

“All of these support data and model parallelism in mixed precision, along with other general practices such as offloading and rematerialization.”

May combine two claims.

Microsoft DeepSpeed, HPC-AI Tech Colossal-AI and Nvidia Megatron-LM are pioneering, widely adopted prototypes for training large Transformer models at scale.

“There are several widely adopted prototypes for training large Transformer models at scale, in which Microsoft DeepSpeedfoot_5 , HPC-AI Tech Colossal-AIfoot_6 and Nvidia Megatron-LMfoot_7 are the pioneering ones.”

May combine two claims.

Addition-based parameter-efficient tuning methods add trainable parameters to the input space or add adapter modules to each Transformer block after the multi-head self-attention and feed-forward network.

“For example, [Lester et al., 2021;Jia et al., 2022] add trainable parameters to the input space, and [Houlsby et al., 2019] adds the adapter module twice to each Transformer block after the MSA and FFN.”

• cites Lester et al., 2021Not checked yet

  Not checked in the free run.

• cites Jia et al., 2022Not checked yet

  Not checked in the free run.

• cites Houlsby et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Popular sparse matrix computation libraries include Intel Math Kernel Library on CPU and cuSPARSE, CUSP and 2:4 structured sparsity on GPU.

“There are several popular sparse matrix computation libraries, such as Intel Math Kernel Libraryfoot_13 on CPU and cuSPARSEfoot_14 , CUSP 16 and 2:4 structured sparsityfoot_16 [Mishra et al., 2021] on GPU.”

• cites Mishra et al., 2021Not checked yet

  Not checked in the free run.

May combine two claims.

Parameter-efficient tuning updates only a small number of additional parameters while freezing the pretrained model, which significantly reduces the storage burden and scales with dynamic deployment scenarios without storing a separate model instance per case.

“As a powerful alternative for the vanilla full fine-tuning, parameterefficient tuning (PET) only updates a small number of additional parameters while freezing the pretrained model to significantly reduce the storage burden, which scales with dynamic deployment scenarios without the need to store a separate instance of model for each case.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The public model zoo represented by HuggingFace, which contains pretrained models ready to be downloaded and executed, contributes significantly to reductions in training costs.

“The public model zoo represented by HuggingFace, which contains rich pretrained models that are ready to be downloaded and executed, is contributing significantly to reductions in training costs.”

May combine two claims.

Alpaca-LoRA adopts low-rank LoRA to enable instruct-tuning LLaMA on consumer hardware, with training done within hours on a single RTX 4090.

“its variant Alpaca-LoRAfoot_4 further adopts the low-rank LoRA to enable instruct-tuning LLaMA on customer hardware, showing training can be done within hours on a single RTX 4090.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Field Programmable Gate Arrays are designed to be reprogrammed to perform different functions as needed and can serve as a prototype for ASICs before the design is finalized.

“In contrast, Field Programmable Gate Arrays (FPGAs) are designed to be reprogrammed to perform different functions as needed, and can also be used as a prototype for ASICs before finalizing the design.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Offloading uses external memory such as CPU memory as an extension of GPU memory to increase training memory capacity through GPU-CPU communications.

“In terms of offloading, it is a technique to use external memory such as CPU memory as an extension of GPU memory to increase memory capacity during training, through communications between GPU and CPU.”

Probably not a substantial claim: may not pass S4 (based).

Memory-efficient techniques meet the memory requirements for training large Transformers, which requires jointly optimizing processing-element utilization, memory and communication footprints across accelerators using parallelism, low-precision arithmetic, checkpointing and offloading.

“3) memory-efficient techniques to meet the memory requirements for training large Transformers, which requires jointly optimizing PE utilization, memory and communication footprints across accelerators, using parallelism, low-precision arithmetic, checkpointing and offloading strategies, etc;”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

It is commonly believed that at least 16 bits are necessary to train networks without impacting model accuracy.

“While it is commonly believed that at least 16-bits is necessary to train networks without impacting model accuracy [Micikevicius et al., 2018;Das et al., 2018]”

• cites Micikevicius et al., 2018Not checked yet

  Not checked in the free run.

• cites Das et al., 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

DeepSpeed is implemented mainly based on Rasley et al. 2020 and the ZeRO series works, Colossal-AI is built upon Bian et al. 2021, and Megatron-LM implements Narayanan et al. 2021.

“Specifically, DeepSpeed is implemented mainly based on [Rasley et al., 2020] and ZeRO series works [Rajbhandari et al., 2020;Ren et al., 2021], Colossal-AI is built upon [Bian et al., 2021], and Megatron-LM implements [Narayanan et al., 2021].”

• cites Rasley et al., 2020Not checked yet

  Not checked in the free run.

• cites Bian et al., 2021Not checked yet

  Not checked in the free run.

• cites Narayanan et al., 2021Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Sanger uses 4-bit queries and keys to compute the quantized prediction of the sparse attention matrix.

“Sanger [Lu et al., 2021] uses 4-bit queries and keys to compute the quantized prediction of sparse attention matrix.”

• cites Lu et al., 2021Not checked yet

  Not checked in the free run.

May combine two claims.

Chen et al. provides a simple periodic schedule implemented in PyTorch, but it is only optimal for homogeneous sequential networks.

“[Chen et al., 2016] provides a simple periodic schedule which was implemented in PyTorchfoot_1 , but it is only optimal for homogeneous sequential networks.”

• cites Chen et al., 2016Not checked yet

  Not checked in the free run.

May combine two claims.

With an efficient hardware accelerator, ELSA achieves a speedup of 58.1 times and three orders of magnitude improvement in energy efficiency compared to an Nvidia V100 GPU equipped with 16GB memory.

“With an efficient hardware accelerator, ELSA achieves a speedup of 58.1× as well as three orders of magnitude improvements in energy efficiency compared to an Nvidia V100 GPU equipped with 16GB memory.”

May combine two claims.

Compared with full-precision training, AMP achieves faster training and inference speed and reduces memory consumption during network training.

“Compared to full-precision training, AMP is able to achieve faster training/inference speed and reduce memory consumption during network training.”

• cites Micikevicius et al., 2018Not checked yet

  Not checked in the free run.

May combine two claims.

LoRA approximates the update of self-attention weights into two low-rank matrices that can be merged into the pretrained weights during inference.

“the representative work LoRA [Hu et al., 2022] approximates the update of self-attention weights into two low-rank matrices, which can be merged into the pretrained weights during inference.”

• cites Hu et al., 2022Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Hardware and algorithm co-design maximizes the training scalability on hardware platforms.

“4) hardware and algorithm co-design to maximize the training scalability on hardware platforms.”

May combine two claims.

Using lower precision multipliers, adders and memory blocks results in a significant improvement in power consumption and speedup.

“As a result, we can use lower precision multipliers, adders, and memory blocks, resulting in a significant improvement in power consumption and speedup.”