Convolutional Sequence to Sequence Learning

“Our model is equipped with gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2015a).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Models with many layers often rely on shortcut or residual connections (He et al., 2015a; Zhou et al., 2016; Wu et al., 2016).”

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“memory networks and CNN [10] for sequential recommendation.”

“To tackle these barriers, there is an emerging trend of leveraging Convolutional Neural Network (CNN) for sequence learning in language modeling for NLP tasks (Meng et al. 2015; Bradbury et al. 2016; Kalchbrenner et al. 2016; Gehring et al. 2017).”

“, 2016), Convolutional Sequence-to-sequence networks (Gehring et al., 2017) and Squeeze-and-excitation Networks (Hu et al.”

“We follow the preprocessing and data splits by previous work (EN-DE: Vaswani et al., 2017; EN-FR: Gehring et al., 2017; EN-ZH: Hassan et al., 2018; Wu et al., 2019).”

“It is different from the popular position embedding methods [11, 40], which add the absolute position embedding to the corresponding input features in each dimension.”

“Regarding the imitation model, since the architecture of the victim model is unknown to the adversary, we simulate this black-box setting by using three different architectures as the imitator, namely (m)BART, Transformer-base, and ConvS2S [9].”

“Positional encoding is critical to exploit the position order of the sequence in transformer networks [27, 28, 29].”

“An important question in designing coordinate-based methods is how to embed positional information into a model [50, 13].”

“…(BT; Bojar and Tamchyna 2011; Sennrich et al. 2016a; Poncelas et al. 2018a) is a data augmentation method that is a key ingredient for improving translation quality of neural machine translation systems (NMT; Sutskever et al. 2014; Bahdanau et al. 2015; Gehring et al. 2017; Vaswani et al. 2017).”

“Many other positional encoding choices are available, such as learned positional encoding [88] and relative positional encoding [89].”

“Different from the recurrent decoder, our attention-based decoder is a multi-block convolutional neural network (Gehring et al. 2017).”

“Successful non-recurrent alternatives have also been proposed, such as convolutional encoders and decoders with attention (Gehring et al, 2017), and the fully-connected deep transformers which employ the idea of self-attention in addition to the default cross-attention mechanism (Vaswani et al,…”

“Inspired by the method for modeling positional information from the NLP field [39], [40], [41], the temporal distance between the i th predicted point and k th observed point is defined as We have presented an illustrative example in Fig.”

“As a result, a number of more parallelizable sequence models were proposed such as ConvS2S (Gehring et al., 2017) and the Transformer (Vaswani et al., 2017).”

“Neural language modeling-based story generation approaches in particular [Roemmele and Gordon, 2015; Khalifa et al., 2017; Gehring et al., 2017; Martin et al., 2018] are prone to generating stories with little aim since each sentence, event, word,”

“Unlike phrase-based MT systems which enjoy the benefits of very large beam sizes (in the order of 100–500) (Koehn et al., 2007) , most NMT systems choose tiny beam sizes up to 5; for example, Google’s GNMT (Wu et al., 2016) and Face-book’s ConvS2S (Gehring et al., 2017) use beam sizes 3 and 5, respectively.”

“There are two versions of the positional encoding, one is the learned positional embeddings [26] and the other one is sinusoidal version.”

“Typically, trigonometric functions are adopted as the non-learnable PE [16] to encode both absolute and relative positions, each dimension corresponding to a sinusoid.”

“We base our layout embeddings on the method originally used in [7], and then in [28] to define the sequential embeddings.”

“These embeddings can be implemented using fixed sine and cosine functions across various frequencies [24] or through learnable embed-dings that adapt during training [30].”

“Notably, positional embeddings contain information about the relative position (such as sinusoidal positional encodings (Gehring et al., 2017)) or absolute position (Shaw et al., 2018) of the tokens in the sequence, which are essential since Transformer completely avoids recurrence and convolutions.”

“Their use has proved particularly beneficial in initial transformer models whose design rejects recurrence and convolution [64].”

“In order to validate the advantage of modeling position information using the Gaussian kernel over other manners, we replace the position-aware vectors with position embeddings [39], [44] which are learned as model parameters during training.”

“Transformer (Vaswani et al. 2017) 28.40 ConvS2S (Gehring et al. 2017) 25.16 Weighted Transformer (Ahmed, Keskar, and Socher 2017) 28.90 Layer-wise Transformer (He et al. 2018) 29.01 RNMT+ (Chen et 1 and Table 2 show that our ALM has significant improvements over baselines without pre-training or…”

“We design our CNN encoderMS CNN as a Temporal Convolutional Networks which has provided state-of-art sequential modeling performance [3, 15, 53].”

“This adaptability stands in contrast to other deep learning architectures such as CNN [46], LSTM [64], GAN [49] and ResNet [62], which often assume fixed inputs like matrices or sequences. ii.”

“Recent work on convolutional neural network architectures showed that they are competitive with recurrent architectures even on tasks where modeling long-range dependencies is critical, such as language modeling [1], machine translation [2, 3] and speech synthesis [4].”

“Future directions include validating our approach on other architectures such as RNN (Bah-danau et al., 2015) or CNN (Gehring et al., 2017) based NMT models, as well as combining with other advanced techniques (Shaw et al., 2018; Shen et al., 2018; Yang et al., 2018; Li et al., 2018) to further improve the performance…”

“There are some techniques doing this, such as position embedding [13] and position encoding [47], and we follow the position embedding for simplicity.”

“Gehring et al. (2017) proposed a convolutional sequence-to-sequence model and achieved significant improvement in both performance and training speed.”

“where σ is the sigmoid function, and ⊗ is the operation of elementwise multiplication [12].”

“Some recent studies have shown that CNN is able to outperform RNN in capturing long-term dependencies [8, 11, 23].”

“Traditionally, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been used for this task [9, 10].”

“They were first studied in natural language processing (NLP), where encoder-decoder attention modules were developed to facilitate neural machine translation [24, 25, 26, 27].”

“Another benefit is that the hierarchical structure of CNN enables a short path to incorporate distant information than the chain structure of RNN [9].”

“Deep neural networks with autoregressive encoder-decoder framework have achieved great success on machine translation, with different choices of architectures such as RNN, CNN and Transformer.”

“Nevertheless, full parallelization of the encoder suffices to make this model faster than most of its RNN rivals [17].”

“CNN have been widely applied to the field of NLP, leading to improved performances in many tasks, such as language modeling, machine translation and text classification [5, 10, 11, 13, 4].”

“Thus we update the input representation of the token q for the tunable SemEncoder by summing the corresponding token embeddings in Xcr and its absolute position embeddings, where the absolute position embeddings are trained with learnable parameters (Gehring et al. 2017).”

“, 2014), convolutional (Gehring et al., 2017), or self-attentive layers (Vaswani et al.”

“The model choice could be, for example, RNN-based (Wu et al., 2016), CNN-based (Gehring et al., 2017), Transformer model (Vaswani et al., 2017) or RNMT+ (Chen et al., 2018).”

“Recently, CNNs have also been employed to model sequential data [58], [59], [60], [61], [62].”

“As a result, these methods have been applied to a variety of other domains including NLP and speech (Gehring et al., 2017; Wu et al., 2018; Subramani and Rao, 2020).”

“The dominant approaches for NMT are based on autoregressive translation (AT), where the generation of the current token in the target sentence depends on the previously generated tokens as well as the source sentence.”

“, 2014), which showed immense success in neural machine translation (NMT) (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017).”

“researchers have proposed frameworks based on CNN [8] and attention mechanism [37].”

“The benefits of such mechanism have been shown across a range of tasks, from understanding and generation in media [53], [54], to detection and localisation of media [55], [56].”

“Note that, there are several approaches for positional encoding, including the sinusoidal functions in [30], the diagonal-disabled mask in [12], the bi-directional masks for obtaining asymmetric attention matrix in [27], and other forms [8, 28].”

“With proper padding, we ensure that the output of the convolution matches the length of the input for different ngram-window sizes (Gehring et al., 2017).”

“, 2017) and convolutional sequence to sequence models (Gehring et al., 2017), are promising directions which may lead to better context-dependent pore models.”

“In order to improve the inference speed, FastSpeech [58] used a feed-forward network based on 1D convolution [117], [118] and the self-attention mechanism in transformers to generate”

“Convolutional sequence-to-sequence learning [49] is another NMT architecture.”

“Neural network approaches to machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Lu-ong et al., 2015a; Gehring et al., 2017) are appealing for their single-model, end-to-end training process, and have demonstrated competitive performance compared to earlier statistical approaches…”

“Gehring et al. (2017) developed convolutional machine translation models augmented with attention over the input sentence.”

“Recently, CNN-based temporal frameworks [8, 11, 13, 37] have shown advantages over RNNs in modeling temporal information.”

“We do not show the results of some meth-ods, e.g., CCPM [14], GBDT [7], FFM [24], HoFM [5], AFM [59], NFM [19], as many works [10, 56, 57] have surpassed them.”

“Conv-seq2seq has many advantages than RNN-seq2seq, including better parallelism, more stable gradient, more flexible reception field and lower memory requirement for training [19].”

“DV3 simplified the previous systems by using a convolutional sequence to sequence architecture (Gehring et al., 2017) and incorporating the key-value attention mechanism of (Vaswani et al., 2017).”

“Deep learning [1] has recently enabled remarkable progress in diverse fields including as speech [2, 3], computer vision [4, 5], machine translation [6], and natural language processing [7].”

“The position embedding module takes each index t as the input and returns its 512-d embedding from a learnable lookup table [17].”

“In contrast to RNNs, CNNS do not depend on the computations of the previous time-step and therefore allow parallelization over every element in the sequence [52].”

“Subsequent to these, attention mechanisms have penetrated into various tasks in the field of information recognition, such as machine translation [43], object recognition [30], pose estimation [44], saliency detection [45].”

“ByteNet (Kalchbrenner et al., 2016) 23.75 GNMT+RL (Wu et al., 2016) 24.6 ConvS2S (Gehring et al., 2017) 25.16 Transformer (Vaswani et al., 2017) 27 The best setting of BPT on WMT14 is k = 4 , the same as the best setting of BPT on Document-Level Machine Translation(IWSLT) and Text…”

“2, we present an attention memory network which is inspired by recent convolutional sequenceto-sequence learning task in machine translation (Gehring et al. 2017).”

“With more gains arising from continued research on new neural network architectures and accompanying training techniques (Vaswani et al., 2017; Gehring et al., 2017; Chen et al., 2018), NMT researchers, both in industry and academia, have doubled down on their ability to train high capacity models…”

“We use 7-layer convolutional seq2seq models 6 (Gehring et al., 2017) as our error correction and error generation model, which have proven to be effective for GEC (Chollampatt & Ng, 2018).”

“There are many choices of positional encodings, learned and fixed (Gehring et al., 2017).”

“Temporal Convolution: Convolutions over the sequence length has been used in numerous prior works [60, 40, 17, 28, 5] for sequence modeling.”

“We do not adopt the conventional learning approach [Gehring et al. 2017] through learning a fixed-size lookup table of positional encodings.”

“Sequence to sequence models directly estimate the posterior probability of a target sequence y given a source sequence x (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) and can be trained with pairs of source and target sequences.”

“Very recently, convolutional networks have reached comparable or superior performance on the same task (Gehring et al., 2017).”

“Traditional NMT models are based on the sequenceto-sequence framework (Bahdanau et al., 2014; Sutskever et al., 2014), taking the source sentence as input and generating the target sentence in an autoregressive manner.”

“Due to the space limitation, we refer readers to Bahdanau et al. (2014), Gehring et al. (2017) and Vaswani et al. (2017) for the details on the definitions of f and g .”

“Convolu-tional sequence-to-sequence models outperform mature recurrent models on very large benchmark datasets by an order of magnitude in terms of speed and have achieved the state-of-the-art results on several public benchmark datasets [29].”

“…31.02 1.79 6M 25.92 7M Varational attention (Deng et al., 2018) 33.10 Transformer** (Vaswani et al., 2017) 32.83 3.53 59M 27.68 61M ConvS2S** (MLE) (Gehring et al., 2017b) 32.31 1.35 21M 26.73 22M ConvS2S (MLE+SLE) (Edunov et al., 2018) 32.84 Pervasive Attention (this paper) 33.81 ± 0.03 3.51 7M…”

“Deep Convolutional Neural Networks (CNNs) have achieved remarkable accuracy for tasks in a wide range of application domains including image processing (He et al., 2016b), machine translation (Gehring et al., 2017), and speech recognition (Zhang et al., 2017).”

“Baseline Approaches: We employ two baseline approaches in our experiments, the convolutional neural network machine translation model (Gehring et al. 2017) and the attention-based sequence-to-sequence model in (Bahdanau, Cho, and Bengio 2014).”

“Additionally, [37, 43, 44, 45] considered CNN variants, such as pixelCNN [46], for text generation.”

“Ordering is defined by decreasing particle transverse momentum, and the index within the ordered sequence is embedded using learned positional embeddings [50, 51].”

“We start with the original Transformer architecture on the IWSLT-14 translation task (Cettolo et al., 2014) but use an SGD-based training routine as a baseline (Gehring et al., 2017).”

“41 Conv-Seq2Seq (Gehring et al., 2017) 24.”

“Here, we propose a novel model for video story QA task, Multimodal Dual Attention Memory (MDAM), which uses ResNet [6], GloVe [18], positional encoding [4], and casing features [19] to represent scene frames and captions of a video.”

“[17] proposed a sequence model based entirely on convolutional neural networks.”