Convolutional Sequence to Sequence Learning

“Our model is equipped with gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2015a).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Models with many layers often rely on shortcut or residual connections (He et al., 2015a; Zhou et al., 2016; Wu et al., 2016).”

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“…(BPE) segmentation (Sennrich et al., 2016) to split rare words into sub-units, which has very few applications in summarization so far (Fan et al., 2018; Kiyono et al., 2018), though being a common technique in machine translation (Wu et al., 2016; Vaswani et al., 2017; Gehring et al., 2017).”

“The ConvS2S model (Facebook’s AI Lab) [30], which encoders and decoders are based on convolutional neural networks, and the model is used for automatic text summary tasks to achieve good results.”

“Sophisticated sequence-to-sequence architectures (Gehring et al., 2017; Vaswani et al., 2017) On the other hand, there are a number of studies recently on integrating cross-sentence context for neural machine translation (NMT).”

“The Transformer generates an ordered sequence of positional embeddings by a positional encoding mechanism (Gehring et al., 2017a) to explicitly encode the order of dependencies between words in a sentence.”

“Hence it can perform any sequence-to-sequence mapping with high accuracy (Gehring et al. 2017).”

“Neural machine translation (NMT) generally adopts an attentive encoder-decoder framework (Sutskever et al., 2014; Gehring et al., 2017; Vaswani et al., 2017), where the encoder maps the source sentence into a sequence of contextual representations (source contents), and the decoder generates the…”

“Neural machine translation (NMT) has significantly improved the quality of machine translation in recent years, which has shown promising results on multiple language pairs [1,4,7,18,21].”

“paper proposes to model the hierarchical syntax structure of code using triplet position, inspired by the positional encoding used in sequence modeling (Gehring et al., 2017; Vaswani et al., 2017), and incorporates the triplet position into current GNNs for better code summarization.”

“The original Transformer architecture used an absolute sinusoidal position encoding, which was later improved to a learnable absolute position encoding [15].”

“For initial translation, we rely on the convolutional translation system from (Gehring et al., 2017) learned from the training portion of the dataset.”

“, 2019], convolutional neural networks (CNN) [Gehring et al., 2017], graph neural networks (GNN) [Li et al.”

“To validate the contribution of each component, we exclude one of key components from our model as follows: (i) -NoDilated with conventional convolutions instead, (ii) -NoMulti with no multi-level memory (iii) -NoNGTU with existing gated linear units (Gehring et al., 2017).”

“(2018), who use a convolutional architecture (Gehring et al., 2017), and Wu et al.”

“A similar position embedding was used earlier in a convolution-based architecture [Collobert and Weston, 2008], and later in an architecture that combines convolution with attention [Gehring et al., 2017].”

“The first approaches simply added a transformation of the tokens’ positions, e.g., scaled si-nusoids (Vaswani et al., 2017) or learned embed-dings (Gehring et al., 2017), to the embeddings of the input sequence.”

“…state-space models with more general structured masks (Dao and Gu, 2024), convolutional architectures (Shi et al., 2023; Poli et al., 2023; Gehring et al., 2017; Gu et al., 2021; Bradbury et al., 2017; Smith et al., 2023; Hochreiter and Schmidhuber, 1997), and more novel computational…”