Convolutional Sequence to Sequence Learning

“Our model is equipped with gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2015a).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Models with many layers often rely on shortcut or residual connections (He et al., 2015a; Zhou et al., 2016; Wu et al., 2016).”

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“…separately, the extracted feature vectors were concatenated and fed into a fully connected layer for prediction; and (3) CNN has also been widely employed in many different types of supervised tasks, e.g., image classification (He et al., 2016), and natural language processing (Gehring et al., 2017).”

“In previous studies, encoders and decoders were usually built using RNNs or CNNs [31].”

“They are first studied in the community of natural language processing (NLP), in which attention based encoder-decoder network is developed to benefit machine translation [4], [8], [48].”

“RNN-based structures fit the sequential nature of time series, but require iterative computing and subsequently a longer path when capturing long-range dependencies (Gehring et al., 2017).”

“And also, for machine translation, [22] replaces the encoder and [23] replaces the whole pipeline with a fully convolutional network.”

“30 n/a n/a (Gehring et al., 2017) CONVS2S n/a 26.”

“They can be based on Recurrent Neural Network (RNN) such as Long Short Term Memory (LSTM) or Gated Recurrent Unit (GRU) [1,10], or Convolutional Neural Network (CNN) [6,7].”

“In this work, we adopt learned positional embedding [41].”

“Gehring et al. (2017) proposed the CNN integrated positional attention architecture for sequence modeling, which computes position embedding to identify the context location and relies on gating and multi-level attention mechanism for sentiment classification.”

“The CNNs are now widely used not only in the time recognition but also in the fields of natural language processing [40, 41] and acoustic signal processing [42].”

“the relative or absolute position of each element, the triangular (sinusoidal) positional encoding (PE) [65] is added to the inputs”

“This is in contrast to the position embeddings (Gehring et al., 2017), a learned position vectors, which are not meant to attend to relative positions.”

“…the advanced modeling and training techniques, the RNMT+ (Chen et al. 2018) has shown its greater potential strength, even surpasses the convolutional seq2seq (ConvS2S) model (Gehring et al. 2017) and achieves comparable results with the well-known Trans-former model (Vaswani et al. 2017).”

“Recently, Convolutional Neural Network (CNN) based architectures have provided encouraging results in sequence-to-sequence tasks [10] like machine translation [11,12], image generation [13] and Image Captioning [14].”

“…2014; Bahdanau, Cho, and Bengio 2015), which directly models the translation process in an end-to-end way, has achieved state-of-the-art translation performance on resource-rich language pairs such as English-French and German-English (Johnson et al. 2016; Gehring et al. 2017; Vaswani et al. 2017).”

“Recently, convolutional sequence-to-sequence networks have been successfully applied to machine translation tasks [5, 8].”

“We also experimented with learned positional encoding [13] and did not observe any significant improvement in terms of performance.”

“Future directions include designing better approaches to evaluate the impact of components (e.g., from the perspective of information flow), and validating our findings on other NMT architectures such as RNMT (Chen et al., 2018) and ConvS2S (Gehring et al., 2017).”

“Although LSTMs are popular decoders in the encoder-decoder captioning models, language CNNs [49], [50] can alternatively be employed as decoders, and the advantages are (1) faster training [19], and (2) multi-level representations for sentences [18], [22].”

“Recently, 1D-CNN has been proven to be capable of handling sequence modeling tasks such as machine translation [12] and sentence classification [13], resulting in great advances in both computation efficiency and accuracy.”

“In CNN based encoder-decoders, the representations of words are the concatenation of word embeddings and position embeddings (Gehring et al., 2017).”

“, 2017) or with a convolutional neural network (CNN) (Gehring et al., 2017).”

“Besides, convolutional architectures initially developed for image classification such as Residual Networks are now routinely used for machine translation [13] and speech recognition [45].”

“positional information of the sequential input [58].”

“Those GFCN have proven their efficiency in many fields of language and image processing such as in language modeling [16], translation [17] and segmentation [18].”

“5 In our implementation, we used learned absolute positional embeddings [8] instead, as suggested by [26].”

“The gate mechanism is widely applied to NLP, such as LSTM[6], GRU[2], language modeling[4], sequence to sequence learning[5] and they utilize the gate to prevent the gradients vanishing and resolve the long-term dependency problem.”

“Just like Seq2Seq implemented using LSTM, ConvS2S involves encoder-decoder approach but contains convolution blocks containing layers of convolutional blocks in both encoder and decoder (Gehring et al., 2017 [5]).”

“Our model utilizes a learnable attention mechanism similar to those that have been widely adopted in the community [44, 50, 16], improving not only the performance for the task in question [37, 28], but also improving interpretability and robustness from various perspectives [3, 2, 51, 11].”

“These embeddings, combined with CNNs, have proved useful in computer vision (Gehring et al., 2017); they help to localize non-spatial features extracted by convolutional networks within an image.”

“Variants of network structures have been applied in NMT such as LSTM (Wu et al., 2016), CNN (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).”

“Various approaches could be used to aggregate historical context, such as RNNs, masked convolutions (Gehring et al., 2017) and attention-based methods (Vaswani et al., 2017).”

“The convolutional architecture proposed in Gehring et al. (2017) combines word embeddings with positional embeddings.”

“The Fairseq(-py) toolkit was used to model the ConvS2S model (Gehring et al., 2017).”

“Non-recurrent models, such as convolutional neural networks (CNNs, Kalchbrenner et al., 2016; Gehring et al., 2016, 2017) and self-attentive models (Vaswani et al., 2017; Chen et al., 2018) have recently reached comparable or better performance than RNNs on machine translation and other benchmarks.”

“Recent architectures using convolution (Gehring et al., 2017) and self attention (Vaswani et al., 2017) have proved to be useful, especially to facilitate efficient training.”

“With the success of RNNs in many NLP tasks, many scholars have proposed different structures to increase the speed of RNNs.”

“Until recently, CNNs are applied to natural language processing tasks and show competitive performance on several tasks, including sentence classification [16] and machine translation [7].”

“The temporal convolutional network (TCN) [3] is inspired by several convolutional architectures [6,8,14,24], but differs from these approaches, according to the authors, insofar as it combines simplicity, auto-regressive prediction, residual blocks and very long memory.”

“Attention based architectures, such as Transformers, have been effective for sequence modelling tasks such as machine translation [Gehring et al., 2017, Vaswani et al., 2017], question answering, sentence classification [Radford et al., 2018, Devlin et al., 2018] and document generation [Liu et al.,…”

“For example, previous works [11,36,41] indicated that PEs are useful to give the model a sense of which portion of the sequence in the input/output it is currently dealing with.”

“We add fixed positional encoding (Gehring et al. 2017) to the reduced features.”

“AAAB9XicbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaWDbbSbt0swm7E7WE/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqDg0ey1i3A2ZACgUNFCihnWhgUSChFYyup37rAbQRsbrDcQJ+xAZKhIIztNJ9F+EJdZTpVNHzSa9ccavuDHSZeDmpkBz1Xvmr2495GoFCLpkxHc9N0M+YRsElTErd1EDC+IgNoGOpYhEYP5tdPaEnVunTMNa2FNKZ+nsiY5Ex4yiwnRHDoVn0puJ/Xif…”

“…of TCNs on the kriging task includes: 1) TCNs can deal with input sequences with varying length, allowing us to perform kriging for test samples with different numbers of time points; 2) TCNs are superior to recurrent neural networks (RNN) with fast training and lightweight architecture [34].”

“On the other hand, Gehring [26] was not too far from Vaswani [22] in BLEU score in the English-French dataset.”

“We use convolutional networks (CNN) as the underlying model given their strong performance on numerous text classification tasks (Kim, 2014; Mou et al., 2016; Gehring et al., 2017).”

“For the purpose of this work, we will focus on two broad categories: (1) image processing for plasma tomography, which will be addressed with CNNs, and (2) time series analysis for disruption prediction, which will be addressed with RNNs.”

“We implement the Attribution method based on the Fairseq-py (Gehring et al., 2017) framework for the above models.”

“chine translation [1], [4], [9], [10] and abstractive summarization [5], [37], [43]–[45].”

“More recently, Gehring et al. [6] propose an architecture for sequence to sequence modeling that is entirely convolutional.”

“Several advanced attention mechanisms have been proposed one after another including hierarchical attention networks [26], attention over attention [27] and multi-step attention [28] et al.”

“They applied a convolutional encoder-decoder model [28] on PubMed open access subset5 to perform abstract generation task.”

“Inspired by the dynamical isometry property, various initialization methods are proposed to increase the convergence speed and stabilize the signal propagation, including Saxe et al. (2014); Bachlechner et al. (2020); Gehring et al. (2017); Balduzzi et al. (2017).”

“There are many workable choices for deciding the time representations [8].”

“Early systems of this kind stack 4-8 layers on both the encoder and decoder sides (Wu et al. 2016; Gehring et al. 2017), and the improvement often comes from the use of wider networks (a.”

“Many impressive progresses have been made in neural machine translation (NMT) in the past few years (Luong et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2019).”

“This task is also distinct to supervised/semi-supervised MT task that mainly depends on parallel sentences (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018; Sennrich et al., 2016a).”

“Neural Machine Translation (NMT) models trained on large, clean parallel corpora have reached impressive performance in translating clean texts following various architectures (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017).”

“NUS A multi-layer convolutional sequence-to-sequence neural network trained with the Fairseq-py toolkit (Gehring et al., 2017).”

“In future work, we plan to study how different units affect representation quality in non-recurrent models such as the Transformer (Vaswani et al., 2017) as well as in convolutional architectures (Gehring et al., 2017).”

“We opt to use learned positional embeddings (Gehring et al., 2017), and obtain the following after applying dropout: x′i = dropout (xi + embed [i]) .”

“DEEP neural networks (DNNs) have achieved great success in many applications of artificial intelligence such as object recognition [1]–[4], image captioning [5], [6], face detection [7], speech recognition [8], [9], and neural machine translation [10], [11].”

“, 2014), convolutional networks (Dauphin et al., 2016; Gehring et al., 2017) and selfattention (Sukhbaatar et al.”

“Recent studies have shown that convolutional neural networks (CNNs) (LeCun et al., 1998) can outperform RNNs in sequence transduction tasks (Kalchbrenner et al., 2016; Gehring et al., 2017).”

“Meanwhile, many sequence prediction studies have recently employed transformers [39], [49], [50] and convolutional neural networks (CNNs) [51], [52], [53].”

“[32] proposed the ConvSeq2Seq model, using a convolutional neural network instead of a recurrent neural network as the encoder and decoder.”

“The success of NeRF in capturing the details of scenes is due in part to the use of positional encoding [43,7] and periodic activations [36,5,37] that help to increase the model’s capacity.”

“In recent decades, various approaches have been widely applied to a variety of text generation tasks (Li et al., 2019; Gehring et al., 2017; Li et al., 2021a), especially the emergence of pretrained language models (PLMs) (Li et al.”

“While architectural innovations such as attention [8], convolutional layers [39], and Transformers [110] have led to significant improvements, this word-by-word modeling remains core to the approach, and with good reason—since any distribution over the output can be factorized autoregressively via…”

“State-of-the-art conditional sequence generation models (Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017) typically rely on an AutoRegressive (AR) factorization scheme to produce the output sequences.”

“Ja → Zh BLEU GNMT (Wu et al., 2016) 49.12 ConvS2S (Gehring et al., 2017) 50.32 Transformer (our implementation) 52.02 ST-NMT (our proposed) 52.84 We further demonstrate the effectiveness of our model on WMT14 English-German translation tasks, and we also compare our model with other competitive…”

“[8] simplified the gradient propagation using GLU and made a breakthrough on the WMT'14 English-German and WMT'14 English-French translation.”

“It is noted PoDA is simple, easy-to-implement and applicable to virtually all seq2seq architectures, including ConvS2S (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).”

“Different from regular operation which use RNN model to embedding the language [25] [31] [7], we adopt CNN to encode them with more efficient parallel convolution which has widely used in many tasks [15] [1].”

“’s 2014 list, language model probability of words using a convolutional sequence to sequence model from (Gehring et al., 2017), comparison methods using pre-”

“The embedding of a fixated word is calculated by the sum of the non-contextual BERT embedding (retrieved from the look-up table) of the fixated word wfk 1 and a trainable position embedding2 [15] that encodes the word index fk .”

“, ConvS2S [8] and transformer [36], etc.”

“Neural language models have been developed with various architectures during recent years (Graves, 2013; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017).”

“Typically, Seq2Seq learning can be implemented as various architectures (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2019), among which the Transformer (Vaswani et al., 2017) has advanced the state of the art.”

“Besides the widely applied RNN based NMT model [45], [47], [49], [39], the first CNN based NMT model on GEC became the first NMT based GEC systems outperforming SMT based GEC systems [50].”

“Because our work is end-to-end, it is also related to seq2seq learning (Sutskever et al., 2014; Bahdanau et al., 2014; Chan et al., 2017; Wiseman and Rush, 2016; Gehring et al., 2017), and in particular training with existing/external language models (Sriram et al., 2018).”

“DV3 [6] switched the architecture to a gated convolutional sequence to sequence architecture [16] and also incorporated the key-value attention mechanism of [17].”

“Comparing how semantic phenomena are represented in different NMT architectures, e.g. purely convolutional (Gehring et al., 2017) or attention-based (Vaswani et al., 2017), may shed light on whether different architectures may better capture semantic phenomena.”

“In NLP, there is little work on attention-based CNNs. Gehring et al. (2017) propose an attention-based convolutional seq-to-seq model for machine translation.”

“where ey and es are the embedding of y and s respectively, Concat(hy,hs) indicates the concatenation of hy and hs, and PE indicates the positional encoding (Gehring et al., 2017):”

“We do so by applying the positional embedding network [7].”

“Inspired by the study of FairSeq [7], a hybrid model is proposed. model incorporates a temporal convolution (T-Conv) unit and a dynamic hierarchical bidirectional GRU (DH-BiGRU) unit sequentially.”

“It relies on large quantities of parallel corpora for effective training of the unprecedented amount of parameters in each of its parts and to avoid overfitting (Bahdanau, Cho, and Bengio 2014; Gehring et al. 2017; Vaswani et al. 2017; Meng et al. 2019).”