Convolutional Sequence to Sequence Learning

“Our model is equipped with gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2015a).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”

“Models with many layers often rely on shortcut or residual connections (He et al., 2015a; Zhou et al., 2016; Wu et al., 2016).”

“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”

“According to (Gehring et al. 2017), it is hard to train a sequence-to-sequence model when the number of timesteps is too large.”

“, 2014), CNN (Gehring et al., 2017), and self-attention (Vaswani et al.”

“We implement the transformer model based on open source sequence-to-sequence software Fairseq-py [ Gehring et al. , 2017 ] For the transformer model, we use the same config-uration for both the encoder and decoder.”

“Convolutional networks have also been explored as an alternative, operating over fixed window sizes and allowing for more computation to be parallelized (Kim, 2014; Dauphin et al., 2017; Gehring et al., 2017).”

“Gehring et al. introduced a sequence-to-sequence model entirely based on CNNs, which enables fast and accurate text generation [22].”

“Starting from a sequence of learnable positional embeddings (Gehring et al., 2017), these latent representations are learned using another stack of transformer encoder layers with an added cross-attention component over the input representations in each block.”

“Deep learning has recently achieving state-of-the-art using convolutional neural network (CNN) [9] in many tasks including object detection [10], face recognition [11], sequence to sequence learning [12, 13], speech recognition [14], semantic segmentation [15], image classification [16], handwritten recognition [17, 1…”

“Each block consists of a bidirectional GRU and a subsequent Temporal Convolutional Network (TCN, Gehring et al. 2017), activated by the Gated Linear Uint (GLU, Dauphin et al. 2017).”

“1 Introduction Neural machine translation (NMT) based on the encoder-decoder architecture becomes a new state-ofthe-art approach due to its distributed representation and end-to-end learning (Luong et al., 2015; Gehring et al., 2017; Vaswani et al., 2017).”

“Channels of feature S would be further activated or deactivated by such channel-wise gates condition C, which shares the similar spirit with Squeeze and Excitation Network (Hu et al., 2018) and the Gated Convolution (Gehring et al., 2017).”

“Moreover, in some specific cases, like require running time guarantee, the hidden layers of Seq2Seq can be implemented by CNN instead of RNN [49].”

“Recurrent neural networks (RNNs) or its variants are often combined with the encoded spatial relationship to model temporal dependencies (e.g., road speed sequences) (Yu, Yin, and Zhu 2018; Gehring et al. 2017).”

“[18] proposed a CNN architecture for learning sequence by sequence.”

“Furthermore, we got the following conclusions: First, the baseline model ’In’ surpass the ’Out + In’ model which shows that the NMT model tends to fit out-of-domain features if we directly include The marks indicate whether the proposed methods were significantly better than the best performed contrast models(**: better…”

“Unsupervised NMT The current NMT systems (Sutskever et al., 2014; Cho et al., 2014a; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are known to easily over-fit and result in an inferior performance when the training data is limited (Koehn and Knowles, 2017; Isabelle et al.,…”

“Considering the sentiment polarity of a given target is usually determined by partial context words such as “is my favorite”, CNN which can process data parallel and extract the informative n-gram features seems to be a suitable model for our task and also the other natural language processing task [18, 19].”

“There are many recent efforts in improving the generation accuracy, e.g., ConvS2S (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).”

“…gated variants (Hochreiter and Schmidhuber, 1997; Cho et al., 2014), extensive efforts have been devoted to developing alternatives (Balduzzi and Ghifary, 2016; Miao et al., 2016; Zoph and Le, 2017; Lee et al., 2017; Lei et al., 2017a; Vaswani et al., 2017; Gehring et al., 2017, inter alia ).”

“…wide attention architectures, two practical considerations have to be addressed: (i) the d−1/2 scaling induced kernel in Equation (9) involves an analytically intractable integral E[ζ(G`1(x))ζ(G`1(x′))]; (ii) incorporation of positional encodings (Gehring et al., 2017; Vaswani et al., 2017).”

“…methods as baselines: Action-Plan (Fan et al., 2018) first predicts the premise of a story with the convolutional LM (Dauphin et al., 2017), then use the fusion mechanism (Sriram et al., 2018) to encourage a convolutional seq2seq model (Gehring et al., 2017) to generate the story from the premise.”

“In deep learning, positional encodings are helpful for recurrent networks [31] and transformers [32].”

“In the case of transformer model that was proposed newly [4], it has outperformed recurrent neural network (RNN)-based models [5–7] and convolutional neural network (CNN)-based models [8] on various translation tasks, drawing the attention of MT researchers.”

“Moreover, we evaluated two task-specific pretraining models including PlotMachines ( PM ) (Rashkin et al., 2020) and Plan&Write ( PW ) (Yao et al., 2019), and two typical non-pretrained models including ConvS2S (Gehring et al., 2017) and Fusion (Fan et al., 2018) on the generation tasks in LOT.”

“However, in recent years it has been repeatedly indicated [3, 16, 27, 44] that for many sequence processing tasks convolution-based methods are much faster than canonical recurrent layers such as LSTMs, and still show a similar or even superior performance.”

“geofmodel architectures. So far our evaluation has focused on systems with the Transformer architecture. In this section, we evaluate another two mainstream architectures: LSTM-Luong [27] and ConvS2S [19]. Our goal is to understand which architectural choices may be more vulnerable to attacks. LSTM-Luong [27] is imple…”

“Methodologies utilized for NMT include recurrent neural networks (RNNs) [25,26], convolutional neural networks (CNNs) [27,28], and the Transformer model [17].”

“ame order in this schema (unlike common RNN architectures) and thus losing the temporal dependencies in the input/output. Inspired by the position embedding technique widely used in sequence learning [12, 22], we add a similar position-dependent signal to each landmark feature, which helps capture comprehensive landma…”

“In this work, we present several simple & effective techniques, which could obtain accurate and stable alignment with the multi-step attention (Gehring et al., 2017).”

“In this model, we set h1:T = x1:T + a1:T , where a1:T is a learned absolute position representation, with one vector learned per absolute position, as used in (Gehring et al., 2017).”

“By summing the position encodings with the input embeddings and feeding the result the model, it can give the model a sense of the order in which the token is currently processed [22].”

“PE s and PE l denote the sine and cosine functions [10] and learned positional embeddings [61], respectively.”

“As mentioned before, gradient clipping has been widely used in training deep learning models such as large-scale language models to circumvent the exploding gradient challenge (Merity et al., 2017; Gehring et al., 2017; Peters et al., 2018).”

“Recent studies have also pointed out that PE can make improvements on models [21, 41].”

“Positional encoding layers [4, 11] are used to further augment Equal contribution the self-attention layers with sequence order information.”

“Convolutional Neural Networks (CNNs) [6,7] and Recurrent Neural Networks (RNNs) [8] have been successfully implemented to tackle complex problems such as computer vision and machine translation (e.g. [9] [10]).”

“Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are data driven models, which highly depend on the training corpus.”

“Deep neural networks (DNNs) have achieved remarkable accuracy for tasks in a wide range of applications, including image processing (He et al. 2016a), machine translation (Gehring et al. 2017), and speech recognition (Zhang et al. 2017).”

“To encode the question raising timestamp into a low-dimensional representation t ∈ Rd , we employ a traditional position encoding method [9], and the value of its k-th position in t is defined as follows:”

“Different from [5, 16] which utilize CNNs to address the above issues, in this work, we propose a model named Attention-based Densely Connected LSTM (DenseLSTM).”

“Convolutional neural networks are used in various fields, including image [33,34], object [18,35,36] and handwriting recognition [36,37], face verification [38], natural language processing [39,40] and machine translation [21].”

“BIFT performs best12 among all the systems including GNMT [20], Conv [3] and AttIsAll [4].”

“Related work includes the CNN models that first made a series of breakthroughs in the computer vision field and subsequently showed excellent performance in NLP tasks such as machine translation [26], sentence classification [27,28], and sentence modelling [29].”

“Convolutional Neural Networks (CNN) originally popularized for text classification by employing convolutions directly to the text sentences are also effective for NLP [5], such as sentence modeling [8], named entity recognition [13] and sequence to sequence learning [10].”

“The second system, FConv , is a smaller version of the convolutional Seq2Seq model with an attention mechanism from (Gehring et al., 2017).”

“Fully convolutional architectures have performed well against encoder–decoder networks on neural machine translation tasks [30], handwriting generation [31, 32], and HTR tasks [33–38].”

“0‡ [ Convolutional Seq-to-Seq (Gehring et al., 2017) 40.”

“, 2012; Collobert, Puhrsch, and Synnaeve, 2016), computer vision (Krizhevsky, Sutskever, and Hinton, 2012), and machine translation (Sutskever, Vinyals, and Le, 2014; Gehring et al., 2017).”

“Combined with our first result, these observation helps to explain why an increasingly large body of empirical research succeeds in replacing recurrent models with feed-forward models in important applications, including translation (Vaswani et al., 2017; Gehring et al., 2017), speech synthesis (Van Den Oord et al.”

“Similar to how positional encoding enhances performance in models that already incorporate positional information [19], predicting k using FVEs”

“The encoder and decoder can be specialized using different types of neural networks, such as GRU [3], CNN [16], and Trans-former [43].”

“Besides the one-hot embedding, we also experimented with sinusoidal wave embedding [38], learned embedding [13], and stochastic embedding.”

“Inspired by BERT’s [class] token [54], [CLS] tokens are preset at the head of every token sequence, for integrating the spatial information of a BVP into one representation vector, Additionally, a learnable encoding Plearnable1 with (Ns + 1, Ds) dimensions [55], as the spatial position embedding, are added to the patc…”

“Self-attention directly quanti fi es the attention weight between each positional pair of feature vectors by their dot product (Gehring et al., 2017).”

“Second, we utilize the Convolutional sequence-to-sequence CONVS2S model (Gehring et al., 2017) following the same settings.”

“…parameter effi-ciency by weight sharing, there is research that studies lightweight model architecture for seq2seq learning where early work (Gehring et al., 2017; Wu et al., 2019) mainly focuses on CNNs, while recent efforts have tended to switch to attention-based models such as Mehta et…”

“me order in the modeling process. To this end, we add an order-aware embedding oi ∈RD p to each product-part feature, which is similar to the position embedding strategy employed in sequence learning [8, 20]. oi stands for the embedding for the frame order i. Sharing similar spirit, we further obtain the type-aware pr…”

“For word representation, different architectures— including, but not limited to, recurrence-based (Chen et al., 2018), convolution-based (Gehring et al., 2017) and transformation-based (Vaswani et al., 2017) NMT models—have been taking advantage of the distributed word embeddings to capture the…”

“To enhance the accuracy and diversity of the generated description, we fed the source token vectors of the product title and product properties into coordinate encoders (Transformers [41] feature attention networks with multiple attention heads and convolutional layers equipped with gated linear units [15]).”

“…systems are based on recurrent networks, usually LSTMs or GRUs (Bahdanau et al., 2014), although recently new approaches to neural machine translation have been proposed (Vaswani et al., 2017; Gehring et al., 2017) which are not based on recurrent networks, but keep the encoder-decoder structure.”

“Moreover, because our model is one type of convolutional neural networks (CNNs), it is supposed to run faster than RNN-based models and is more easily parallelized (van den Oord et al. 2016; Gehring et al. 2017).”

“This indicates”

“Inspired by Transformer that incorporates position encoding [9,33,23,27,7,25] into the attention layer, we introduce placement encoding to encapsulate both location and size information of f bg .”

“Recent years have witnessed a breakthrough in automatic machine translation, thanks to the advances in neural machine translation (NMT) [1, 2].”

“These works aim to improve the expressiveness of transformers using different transformations (e.g., convolutions [21, 24] and gated linear units [25]) or multi-branch feature extractors (e.g., [22, 23]).”

“…proposed for modeling NMT: Recurrent (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014), Convolutional (Gehring et al., 2017), and recently Transformer (TNN) by Vaswani et al. (2017) that have shown better performance and efficient processing of input…”

“…For the utterance encoding, we adopt the directional self-attention (Shen et al., 2017) based encoder (DISAN), adding absolute position embedding (Gehring et al., 2017) to further improve the encoding performance.2 DISAN unit consists of two types of multi-dimensional attention - word level…”

“…integrated into the state-of-the-art Transformer model (Vaswani et al., 2017), the implementation is architecture-neutral, and can therefore be applied to RNN-based models (Schwenk, 2012; Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) or fully convolutional models (Gehring et al., 2017).”

“Wu [40] presented a sliding convolutional attention network for scene text recognition, based on the convolutional sequence-to-sequence learning framework [10].”

“The size of embeddings d is often set between 100 and 1000 (Bahdanau et al., 2014; Vaswani et al., 2017; Gehring et al., 2017).”

“Vanilla Transformer employs Sinusoidal positional encoding [14] or Learned positional encoding [49].”

“…2020; Liu et al., 2020b; Wang et al., 2022) attempt to leverage sequence-to-sequence PT for neural machine translation (NMT; Bahdanau et al., 2015a; Gehring et al., 2017; Vaswani et al., 2017a) by using a large amount of unlabeled ( i.e. monolingual) data, Zan et al. (2022b) show that it usually…”

“Compared to a chain-structured RNN network, a hierarchical CNN can capture long-distance relationships in a shorter path and its computations can be parallelized.”

“For the normalization and initialization of the convolutional architectures, we follow Gehring et al. (2017).”

“Other Approaches: ConvSeq (Gehring et al., 2017b), part of our encoder network, yields performance below all the other competitors, hence we do not include its results here.”

“ConvS2S [14] is also built upon a CNN and encodes the positions of words in the sentence to explicitly model the order of words.”

“We use 16 Telsa V100 GPUs and accelerate training by utilizing mixed precision floating point (Micikevicius et al., 2018), and implement all models with fairseq (Gehring et al., 2017).”

“Concerning the choice of non-embedding or softmax layers, Rikters et al. [113] and Lakew et al. [83] have compared RNN, CNN and the self-attention based architectures for MNMT.”

“Recently several encoder-decoder architectures, such as Convs2s [3] and Trans-former [16] are exploited.”

“in which W0 ∈ Rd×h is the linear transformation matrix shared across all the time slots, b0 ∈ R is the bias, h is the hidden size, and P ∈ RT×h is the matrix of positional embeddings [29].”

“The widely used deep learning model has greatly accelerated the research progress in various fields, especially in the fields of computer version [27,28] and natural language processing (NLP) [29,30].”

“Specifically, CNN is widely used for processing grid-like structured data [12], and made great success in processing image data with applications like image classification [20] and face recognition [22], as well as textual data [11].”

“For WMT14 English-German, from Table 5, we can see that DenseNMT outperforms ConvS2S model by 0.36 BLEU score using 35% fewer training iterations and 20% fewer parameters.”

“Gehring et al. (2017) introduce source word embeddings to predict target words.”

“For WMT14 En→Fr, we borrow the setup of Gehring et al. (2017), validate on newstest2012+2013 and test on newstest2014.”

“Convolution-based sequence learning models [Gehring et al., 2017] are used as alternatives [Yu et al., 2017; Guo et al., 2019]; however, numerous hidden layers are needed to cover broad contexts with a limited size of receptive fields.”

“Wu et al. (2016) also uses the NMT approach over the existing SMT approach and show better results than SMT. Facebook in 2017 proposed the implementation of NMT using Convolutional Neural Networks and claimed faster performance than the work presented by Gehring et al. (2016, 2017).”

“PE was first introduced to represent word positions in a sentence [ Gehring et al. , 2017 ] .”

“Neural machine translation (NMT) models (Kalch-brenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are data-driven and hence require large-scale training data to achieve good performance (Zhang et al., 2019a).”

“Spatio-temporal Graph Convolutional Network (ST-GCN) [39]: ST-GCN introduces a gated linear unit GLU [45] in full convolution layer to capture the time relationship, and uses graph convolution to capture the spatial relationship for mid-and long-term traffic prediction.”

“Inspired by literature [9], we embed their corresponding orders in the low dimensional space”

“In addition, our model is equipped with a multi-step attention mechanism [29,30] relying on a quadratic attentional scoring function introduced in [3].”