Convolutional Sequence to Sequence Learning

“Our model is equipped with gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2015a).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Models with many layers often rely on shortcut or residual connections (He et al., 2015a; Zhou et al., 2016; Wu et al., 2016).”

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“We also experimented with using learned positional embeddings [8] instead, and found that the two versions produced nearly identical results (see Table 3 row (E)).”

“, non-recurrent) networks for modeling sequences in speech and language [36, 54, 15].”

“, non-recurrent) networks for modeling sequences in speech and language [36, 54, 15].”

“For neural networks, input encodings have proven useful in the attention components of recurrent architectures [Gehring et al. 2017] and, subsequently, transformers [Vaswani et al.”

“Inspired by [Gehring et al. , 2017], we employ entire convolutional structures on time axis to capture temporal dynamic behaviors of traffic flows.”

“These methods include LSTMs for sequence modeling (Hochreiter and Schmidhuber 1997), Convolutional Sequence to Sequence for machine translation (Gehring et al. 2017), and even the ImageNet 2017 winning model, Squeeze and Excitation Networks (Hu, Shen, and Sun 2017).”

“vanilla transformer, learned positional encoding [49] and relative positional encoding [50] are also utilized in various models [10], [15].”

“Particularly inspiring for our work are the recent applications of convolutional architectures to machine translation (Kalchbrenner et al., 2016; Gehring et al., 2017a;b), audio synthesis (van den Oord et al., 2016), and language modeling (Dauphin et al., 2017).”

“, 2015), convolutional models (Gehring et al., 2017; Wu et al., 2019) and Transformer (Vaswani et al.”

“This encoder enables the graph network to later reason about both appearance and position jointly, especially when combined with attention, and is an instance of the “positional encoder” popular in language processing [20, 55].”

“Relation Networks and Attention Modules are first proposed in natural language processing [13, 14, 3, 35] and physical system modeling [2, 37, 22, 34, 9, 31].”

“WMT14 En-De ConvS2S (Gehring et al., 2017) 0.”

“Normalization with √ 0.5 helps to stabilize learning by ensuring that the variance throughout the network does not change dramatically (Gehring et al. 2017).”

“They have been shown to still benefit from position encodings (Gehring et al., 2017), however.”

“Self-attention mechanisms have recently been successfully applied in various tasks, such as machine translation [8, 9, 28], graph embedding [29], generative modeling [39], and visual recognition [30, 13, 31, 36].”

“[2017] was added to the contextual representations, while a trainable absolute position encoding Gehring et al. [2017], Devlin et al. [2019], Lan et al.”

“Inspired by the ConvS2S(Gehring et al. 2017), we equip position embedding to the spatial-temporal network series so that the model can take the spatial and temporal information into account, which can enhance the ability to model the spatial-temporal correlations.”

“State-of-the-art results have also been achieved by using numerous convolutional layers in both the encoder and decoder, allowing information to be viewed in several hierarchical layers rather than a multitude of recurrent steps [234].”

“ViT [11] showed that the deterministic and learnable positional embeddings [13] works equally well.”

“Another way is to learn a set of positional embeddings for each position which is more flexible [Kenton and Toutanova, 2019; Gehring et al., 2017].”

“For convolutional neural networks and recurrent neural networks, a relatively large learning rate is usually set in the beginning, and then decreased along with the optimization process (He et al., 2016; 2017; Sutskever et al., 2014; Gehring et al., 2017; He et al., 2019).”

“With the development of deep learning, various neural networks have been widely used to solve Natural Language Processing (NLP) tasks, such as convolutional neural networks (CNNs) [80, 86, 49], recurrent neural networks (RNNs) [173, 109], graph-based neural networks (GNNs) [159, 174, 124] and attention mechanisms [7,…”

“Another way of representing absolute positions is to learn a set of positional embeddings for each position [28, 37].”

“Following common practices [14], we add learnable positional embedding to BEV queries Q before inputting them to BEVFormer.”

“In NLP, there is a recent trend of replacing recurrent neural networks by attention models, enabling parallelized implementations and more efficient learning [21, 56, 13].”

“On the other hand, convolutional networks have been very successful in modeling temporal information in tasks that were traditionally tackled with RNNs, such as neural machine translation [11], language modeling [7], speech generation [57], and speech recognition [6].”

“[29] J.”

“Similar to Gehring et al. (2017), we train using the Nesterov accelerated gradient method (Sutskever et al., 2013) using gradient clipping (Pascanu et al., 2013).”

“Recently, a convolutional alternative to sequence modeling has been proposed showing promise for machine translation (Gehring et al., 2017a,b) and story generation (Fan et al., 2018).”

“The Topic-aware Convolutional Sequence to Sequence model (TCONVS2S) introduced by Narayan et al. (2018a) is an abstractive system which is conditioned on the article’s topics and based entirely on Convolutional Neural Networks (Gehring et al., 2017).”

“Recently, attempts have been made to replace the recurrent networks by full convolution or full attention architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b; Shen et al., 2017a).”

“Complementing recent evidence that convolutional architectures are on a par with recurrent ones in seq2seq tasks (Gehring et al., 2017), we find that Gated ConvNet’s overall probing task performance is comparable to that of the best LSTM architecture (although, as shown in Appendix, the LSTM has a…”

“Other works have investigated attention with multiple ”hops”, such as (Sukhbaatar et al., 2015) and (Gehring et al., 2017).”

“…2018; He et al., 2018; Hassan et al., 2018; Song et al., 2018; Shen et al., 2018), text summarization (Ayana et al., 2016; Suzuki & Nagata, 2017; Gehring et al., 2017), question answering (Yuan et al., 2017; Fedus et al., 2018) and conversational response generation (Shang et al., 2015; Vinyals…”

“An emerging theme of augmenting convolution models with self-attention has yielded gains in several vision tasks.”

“While several recently proposed models avoid recurrence at train time by leveraging convolutions (Kalchbrenner et al., 2016; Gehring et al., 2017; Kaiser et al., 2017) or self-attention (Vaswani et al.”

“Since then, various attention based architectures have been proposed for NMT, such as GNMT [60] that combines bi-directional and uni-directional encoders in a deep architecture and [22] which introduced a convolution based seq2seq learning approach.”

“Other works have replaced them with parametric encoding [33] or adopt Fourier-based kernelized versions [22].”

“In transformer, the features are usually added with position embedding, which provides the position information of the sequence [15,47,11,10,54], image [13,53,27] and video [1,24,52].”

“Therefore, recently, convolutional architectures are used in other sequence to sequence tasks, e.g., conditional image generation [137] and machine translation [42, 43, 138].”

“Sequence-to-sequence models come in various forms, such as [3], [15], [58].”

“Unlike other sequence-to-sequence models [61, 27], transformer networks do not employ recurrence or convolutions, thus lacking the positional information within sequences.”

“First, we capture the ordering information in the temporal attention module by using position embeddings [5], {p1, .”

“…is important, because that is Although not at the level of professional human performance yet, Facebook’s AI Research team recently improved upon the best machine language translation algorithms available using convolutional neural net sequence prediction techniques (Gehring et al., 2017).”

“ral networks [47, 8] and CNNs [110, 36], to more general architectures constructed upon self-attention.”

“In a similar vein, research on convolutional sequence to sequence learning [7] has used fixed and learned position embeddings at the input layer; in that work, positions were represented via an overcomplete basis that is added to the incoming data rather than being compactly represented and input as separate channels.”

“To this end, process H can be built around attention mechanisms (see Section 2) that first learn to compute a soft alignment between two given sequences of words and then perform a word by word comparison [23].”

“Previous NMT models are dominated by RNN-based (Bahdanau, Cho, and Bengio 2014) or CNN-based (e.g. ConvS2S (Gehring et al. 2017), ByteNet (Kalchbrenner et al. 2016)) neural networks.”

“While effective, these RNN-based models, such as [3, 15], depend on a hidden state of the entire past that cannot fully utilize parallel computation within a sequence [8].”

“2) Convolutional Neural Network: Recently, CNN has achieved success in many tasks, such as computer vision [88], [89] and machine translation [90].”

“eration. Moreover, because our model is one type of convolutional neural networks (CNNs), it is supposed to run faster than RNN-based models and is more easily parallelized (van den Oord et al. 2016; Gehring et al. 2017). We dub our model as the multi-track sequential generative adversarial network, or MuseGAN for sho…”

“In addition to image restoration, feature correlations are widely exploited along with neural networks in many other areas, including graphical models [51, 4, 17], relational reasoning [32], machine translation [13, 40] and so on.”

“We further compare activation quantization performance with 3 previous schemes, DoReFa, LPBN, and HWGQ.”

“Attention mechanisms have been successfully applied in various tasks such as machine translation [11], object recognition [25], image captioning [3, 36], visual question answering [34, 41] and pose estimation [6].”

“Even though gradient clipping is a standard practice in tasks such as language models (e.g. Merity et al., 2018; Gehring et al., 2017; Peters et al., 2018), it lacks a firm theoretical grounding.”

“…in sequence modeling through recurrent neural networks (RNN; Sutskever et al. 2014; Bahdanau et al. 2015; Wu et al. 2016), convolutional networks (CNN; Kalchbrenner et al. 2016; Gehring et al. 2016; 2017; Kaiser et al. 2017) and self-attention models (Paulus et al., 2017; Vaswani et al., 2017).”

“A refinement of this was proposed by Gehring et al. (2017) who use multiple convolutional layers in the encoder and the decoder that do not reduce the length of the encoded sequence but incorporate wider context with each layer.”

“For En–Fr, we train on WMT’14 and borrow the setup of Gehring et al. (2017) with 36M training sentence pairs.”

“In [14], the weight assignment additionally accounts for the inner product of spatial positions encoded in high-dimensional vectors.”

“Inspired by [14], [73], the position embedding is adopted because the transformation operators in SAM are not positionsensitive.”

“f enc is a sentence-level feature extractor and can be implemented as multi-layer bidirectional RNNs [3, 45], a convolutional model (ConvS2S), [16] or a Transformer [38].”

“…et al., 2017; Hamilton et al., 2017a; Veliˇckovi´c et al., 2018; Li et al., 2018), which generalizes the stan-∗ These two authors contributed equally. dard definition of convolution over a regular grid topology (Gehring et al., 2017; Krizhevsky et al., 2012) to ‘convolution’ over graph structures.”

“In language modeling (Dauphin et al., 2017; Kalchbrenner et al., 2016; van den Oord et al., 2016; Gehring et al., 2017), Gated Tanh Units (GTU) and Gated Linear Units (GLU) have shown effectiveness of gating mechanisms.”

“Gehring et al. (2017) proposed convolutional seq2seq models and applied it to perform abstractive summarization.”

“local patches in CNNs for images [29] or positional encodings in language models for sequential data [30], [31], [32].”

“Recently, there are several CNNs-based language models having excellent performance on various language modeling tasks [6, 7, 9, 24, 25, 31].”

“, non-recurrent) convolutional neural networks for modeling sequences in speech, language [34, 10, 53, 48], and skeleton [18, 58, 30, 51] due to their superior performance.”

“With the development of convolutional neural networks (CNNs), they have shown powerful feature extraction abilities in many applications[9] [13] [14], thus attracting much interests to apply them into traffic forecasting area.”

“The same technique was improved in a number of neural translation models recently, including Gehring et al. (2017) and Kaiser et al. (2017). continues in applying the model in more domains and exploring the space of self-attention mechanisms.”

“An example of model fitting when training the Seq2Seq model [33] in Table 1 is given in Fig.”

“decoder generates output sequence until it hits a stop symbol. We also refer readers to the following papers for recent advances in machine translation: Transformer [151], Convolutional Seq2Seq [31], [32], and the architecture evaluation survey [11]. These sequence modeling modules allow end-to-end gradient-based lear…”

“It includes models such as ConvS2S [60], transformers [61], and wav2vec [62].”

“Following [26], a learned positional embedding p ∈ R(N+1)×D is added (element-wise).”

“Our SLT architecture is a variant of the system proposed by B´erard et al. (2018), which we re-implemented in the fairseq toolkit (Gehring et al., 2017).”

“, 2017), ConvS2S (Gehring et al., 2017), MMN (Kim et al.”

“We further introduce new architectures built with different components borrowed from RNMT+, ConvS2S and Transformer.”

“We add learned absolute position embedding, pu ∈ Rdh×k (Gehring et al., 2017), resulting in a new vector, Hi = Hi +pu.”

“There are several choices of absolute positional encodings, such as the fixed encodings by sine and cosine functions with different frequencies and the learnable encodings through training parameters [7, 23].”

“Convolutional and fully-attentional feed-forward architectures like the Transformer model have recently emerged as viable alternatives to recurrent neural networks (RNNs) for a range of sequence modeling tasks, notably machine translation [9, 31].”

“The implementation of both the baseline transformer and our CMLM is based on fairseq (Gehring et al., 2017), which efficiently decodes left-to-right transformers by caching the state.”

“One common feature among recent architectures such as the Transformer and convolutional sequence-to-sequence models is an increased capacity for parallel computation, making them a better fit for today’s massively parallel hardware accelerators (Vaswani et al., 2017; Gehring et al., 2017).”

“Figure 6 shows the same bug in both the Java (Figure 6a) and Python (Figure 6b) We compare CoCoNuT with DLFix [42] and SequenceR [13], two state-of-the-art NMT-based APR techniques and three other state-of-the-art NMT architectures (i.e., LSTM [56], Transformer [78], and FConv [20]).”

“An attention mechanism [4, 47, 11, 49, 48] uses inputdependent weights to linearly combine the inputs.”

“6.6 for detailed descriptions of GNMT, ConvS2S, and the Transformer).”