Learned in Translation: Contextualized Word Vectors

“[2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”

“Qi et al. [2016] propose a CNN-based object tracking framework, which uses hierarchical features from a pretrained CNN (VGG-19 by Simonyan and Zisserman [2014]).”

“In computer vision, convolutional neural networks (CNNs) pretrained on ImageNet [Krizhevsky et al., 2012, Deng et al., 2009] have become the de facto initialization for more complex and deeper models.”

“The MT-LSTM is used to compute a sequence of hidden states h = MT-LSTM ( GloVe ( w x )) .”

“Girshick et al. [2014] show that using a pretrained CNN to extract features from region proposals improves object detection and semantic segmentation models.”

“…representation for all inputs. x pool = max( X | y ); mean( X | y ); min( X | y ); x self (15) We feed this joined representation through a three-layer, batch-normalized [Ioffe and Szegedy, 2015] maxout network [Goodfellow et al., 2013] to produce a probability distribution over possible classes.”

“Sutskever et al. [2014] investigate sequence-to-sequence models that consist of a neural network encoder and decoder for machine translation.”

“Bahdanau et al. [2015] propose the augmenting sequence to sequence models with an attention mechanism that gives the decoder access to the encoder representations of the input sequence at each step of sequence generation.”

“For tasks where direct comparisons are possible, ELMo outperforms CoVe (McCann et al., 2017), which computes contextualized representations using a neural machine translation encoder.”

“Model pretraining (McCann et al., 2017; Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2018) allows models to be trained on generic corpora and subsequently be easily adapted to specific tasks with strong performance.”

“For instance, ELMo (Peters et al. , 2018) uses a bidirectional language model while CoVe (McCann et al. , 2017) uses machine translation to embed context information into word representations.”

“[126] pre-trained a deep LSTM encoder from an attentional sequence-to-sequence model with machine translation (MT).”

“, 2017) and machine translation (McCann et al., 2017).”

“Although some supervised pre-training methods like CoVE have achieved promising results on NLP tasks, it is nearly impossible to annotate a textual dataset as large as ImageNet, considering annotating textual data is far more complex than annotating images.”

“…(Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Vinyals et al., 2015; Vaswani et al., 2017), pre-training and transfer learning (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2018), and availability of large-scale supervised datasets (Sandhaus, 2008; Nallapati et…”

“These models allow us to capture the syntax and semantics of text via representations learned in an unsupervised manner, before fine-tuning the model to downstream tasks (Melamud et al., 2016; McCann et al., 2017; Peters et al., 2018; Liu et al., 2019b; Yang et al., 2019; Goldberg, 2019).”

“Also in natural language processing, representations pre-trained on web-crawled corpus via Mask Language Model [10] achieves leading performance on machine translation [38] and natural language inference [8].”

“In addition to biLM-based representations, McCann et al. (2017) learned contextualized vectors with a neural machine translation system (CoVe).”

“A newly emerging type of contextualized word embeddings (Peters et al., 2017; McCann et al., 2017) is gaining popularity in various fields of NLP.”

“duce the performance degradation when increasing the number of languages, or using such models for multilingual transfer learning (McCann et al., 2017; Eriguchi et al., 2018; Artetxe and Schwenk, 2018).”

“Pre-training on other tasks such as machine translation has also been studied (McCann et al., 2017).”

“McCann et al. (2017) exploits language model pre-trained on supervised translation corpus in the target task.”

“…been evaluated in a wide range of NLP tasks, including sentiment analysis, question answering and classification, textual entailment, semantic role labeling, reading comprehension, named entity extraction, and coreference resolution (Peters et al., 2018; Salant & Berant, 2018; McCann et al., 2017).”

“We compare with Existing Models , including tailor-made E2E-ABSA models (Li et al., 2019a; Luo et al., 2019; He et al., 2019), and competitive LSTM-CRF sequence labeling models (Lample et al., 2016; Ma and Hovy, 2016; Liu et al., 2018).”

“It is worth noting that we initially used a pretrained twolayer Bi-LSTM that yields Context Vectors (CoVe) in [18], which we found does not contribute to performance.”

“A CoVe model pretrained on translation was then used on other NLP tasks.”

“, 2014) and contextualized word embeddings (Peters et al., 2018; Devlin et al., 2018; Radford et al., 2018; McCann et al., 2017; Radford et al., 2019) have led to huge performance improvement on many NLP tasks.”

“As seen in Figure 1, the recent works based on neural architectures (Le & Mikolov, 2014; Dai & Le, 2015; Johnson & Zhang, 2016; Miyato et al., 2017; McCann et al., 2017; Howard & Ruder, 2018; Xie et al., 2019; Thongtan & Phienthrakul, 2019) have dominated over traditional machine learning models (Maas et al.”

“, 2014a,b) and words-in-context (McCann et al., 2017).”

“, [70,71,60]) it is argued that the”

“The dynamic embed-dings, such as C2V [15], CoVe [16], ELMo [17], ULMFiT [18], UNILM [19], etc., being context-sensitive, were able to address the polysemy aspect of words.”

“Deep learning using Neural Networks (NNs) has achieved state-of-the-art performance in a wide variety of domains, including computer vision [30], speech recognition [21], machine translation [40], and game playing [47].”

“(Parikh et al., 2016) 86.8 NTI (Yu and Munkhdalai, 2017b) 87.3 Re-read LSTM (Sha et al., 2016) 87.5 BiMPM (Wang et al., 2017) 87.5 DIIN (Gong et al., 2017) 88.0 BCN + CoVe (McCann et al., 2017) 88.1 CAFE (Tay et al., 2018) 88.5 ESIM (Chen et al., 2017a) 88.0 KIM (This paper) 88.6 Table 2: Accuracies of models on SNLI.”

“[21], we train a neural machine translation model to translate from the low-resource language (Spanish or Thai) to English.”

“, 2013) and CoVe (McCann et al., 2017) did not show any gains.”

“Pretraining objectives include context prediction (Mikolov et al., 2013), autoencoding (Dai & Le, 2015), machine translation (McCann et al., 2017), and more recently, variants of language modeling (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019) objectives.”

“Reference [47] proposed context-Vectors (CoVe) derived from deep LSTM encoder of an attentional seq-to-seq model trained for machine translation task.”

“Likewise, in natural language processing, universal text representations can be extracted from networks trained for machine translation [32] or language inference [33], [34].”

“CoVe (McCann et al., 2017) pre-trained a encoder.”

“To avoiding over-fitting, we concatenate the output of a pre-trained 600-dimensional contextualized embedding 1 (McCann et al., 2017), which is trained on English-German machine translation dataset, with the aforementioned lexicon embeddings as the final input of contextual encoding layer.”

“Schematic diagram of the process about combining other tasks with the target task [48].”

“The works in [62] learn contextualized word vectors (CoVe) by training an encoder in the supervised machine translation system.”

“ral language processing (NLP) [35], automatic speech recognition (ASR) [1], etc.”

“While MUSE operates over token based embeddings, we are interested in aligning contextual embeddings, which have shown their benefits in several monolingual applications (Peters et al., 2018; McCann et al., 2017; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018).”

“The recent proposed feature-based approaches mainly focus on learning contextualized word representations such as CoVe (McCann et al., 2017) and ELMo (Peters et al.”

“FISHQA has achieved an accuracy of 0.9446 and an F1-Score of 0.9449, followed by HAN, a model that adopts hierarchical networks with an attention mechanism of random initialization, having an accuracy of 0.9177 and an F1-Score of 0.9166.”

“Pre-trained contextualized word representations learned from language modeling objectives, such as CoVe (McCann et al., 2017), ELMo (Peters et al., 2018), ULMFit (Howard and Ruder, 2018b), GPT (Radford et al., 2018, 2019), and BERT (Devlin et al., 2018), have shown strong performance on a variety…”

“Some previous natural language processing works use only the last layer (17; 12) while others use a linear combination across layers (18).”

“, 2018) and machine translation (McCann et al., 2017) have shown significant improvement on various natu-”

“[24] try to bring the benefit of transfer learning to NLP tasks.”

“Subsequent research works enhanced learning to capture contextualized word representations and transferred them to neural models [53],[54].”

“[21] inputs the static word embeddings GloVe into the supervised neural machine translation task to get the context-based embeddings.”

“…the question by concate-nating 300 -dimensional Glove embeddings pre-trained on the 840B Common Crawl corpus (Pen-nington et al., 2014), 100 -dimensional character n-gram embeddings by Hashimoto et al. (2017), and 300 -dimensional contextualized embeddings pre-trained on WMT (McCann et al., 2017).”

“Recently, neural contextualised word representations have gained widespread use in NLP, thanks to deep learning models which learn usage-dependent representations while optimising tasks such as machine translation (CoVe; McCann et al., 2017) and language modelling (Dai and Le, 2015, ULMFiT; Howard and Ruder, 2018, ELM…”

“The authors of (McCann et al. 2017) stated that utilizing pretraining and character n-gram embeddings improves validation accuracy by 2.”

“A big corpus-based pre-training model (PTM) has been used in a variety of NLP tasks [4], [5], [6], [7].”

“For example, contextual word vectors (CoVe) [29] capture contextual information by an encoder in an attentional seq-to-seq machine translation model; and embeddings from language model (ELMO) [30] extracts context-sensitive features from a bidirectional language model (biLM).”

“Besides training a neural model using only NLI datasets from scratch, approaches leveraging external resources (Chen et al. 2017a; Peters et al. 2018; Radford et al. 2018; McCann et al. 2017; Pan et al. 2018) emerged recently and further improved the performance of NLI.”

“Recently, CoVe (McCann et al., 2017) introduced (conditional) language models pre-trained on paired machine translation corpus.”

“Although there are many other well-developed neural network models, such as C-LSTM [36], bi-attentional classification network [20], etc.”

“CoVe (McCann et al., 2017) generates contextualized representations for a word based on pre-trained MT models, More recently, ELMo (Peters et al.”

“…representations have been used in neural machine translation as well, e.g. for learning contextual word vectors and applying them in other tasks (McCann et al., 2017) or for learning context-dependent representations to solve disambiguation problems in machine translation Choi et al. (2016).”

“Jiangnan (Xia, 2018) applies a BiLSTM over GloVe and CoVe embeddings (McCann et al., 2017) with an additional attention mechanism.”

“These findings begin to shed light on why pretrained representations, like ELMo and CoVe, are so beneficial for neural language understanding models (Peters et al., 2018; McCann et al., 2017).”

“…encoder in an unsupervised way; in [35], sentiment tree banks and recursive neural tensor networks are used to represent language features; in [27], a LSTM encoder from attentional sequence-to-sequence model is used to contextualize word vectors; in [44], XL-Net, a generalized auto-regressive…”

“Table 1 lists 8 popular pre-training models and their main differences (Kiros et al., 2015; Lo-geswaran and Lee, 2018; McCann et al., 2017; Conneau et al., 2017; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018).”

“This method was developed by McCann et al. [140].”

“The use of contextualized word representations (Peters et al., 2017; McCann et al., 2017) greatly reduces the amount of data needed to train linguistic models.”

“Recently, there is a trend of learning contextualized word embeddings (Dai and Le, 2015; McCann et al., 2017; Peters et al., 2018; Howard and Ruder, 2018).”

“Early pre-trained language models (PLMs) like CoVe (McCann et al., 2017) and ELMo (Peters et al.”

“…the sense that they do not change with the context.2 Recently, contextualized word embeddings derived from large-scale pre-trained language models (McCann et al., 2017; Peters et al., 2017, 2018; ∗Email corresponding 1Our code is released at https://github.com/ WangYuxuan93/CLBT 2In this paper,…”

“However, for the sake of completeness, we still report their scores 2 (McCann et al., 2017; Chen et al., 2017a; Peters et al., 2018).”

“More recent reusable sentence encoders such as CoVe (McCann et al., 2017) and GPT (Radford et al., 2018) instead represent sentences as sequences of vectors.”

“Varying these factors can significantly impact performance on a broad range of NLP tasks (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2019).”

“Language pre-training has been highly successful, with state-of-the-art models such as GPT, BERT, XLNet, ERNIE, ELECTRA, and T5, among many others [8,19,20,23–25].”

“The aforementioned token vectors are concatenated with pre-trained 600-dimensional CoVe vectors (McCann et al., 2017), and then fed to a BiL- 2https://github.com/kevinduh/san_mrc Embedding FFN Bi-LSTM +CoVe Embedding FFN Bi-LSTM +CoVe Bi-LSTMSelf-Attn ... fell several meters…”

“Many of the top performing NLI models (Ghaeini et al., 2018b; Tay et al., 2018; Peters et al., 2018; McCann et al., 2017; Gong et al., 2017; Wang et al., 2017; Chen et al., 2017), are variants of the ESIM model (Chen et al.”

“2 we show the expected performance of the biattentive classification network (BCN; McCann et al., 2017) with three embedding approaches (GloVe only, GloVe + ELMo frozen, and GloVe”

“consider word embeddings as a function of the word’s context in which the word is appeared (McCann et al., 2017; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018).”

“Recently, pre-trained contextualized word representations (Melamud et al., 2016; McCann et al., 2017; Peters et al., 2018; Devlin et al., 2019) have been shown to improve downstream NLP tasks.”

“Recurrent Networks: Recurrent networks are a powerful technique in many challenging procedural [11, 22] and temporal [4, 29, 31] computer vision problems.”

“Learning representations of natural languages is useful for a variety of NLP tasks (McCann et al., 2017; Liu et al., 2019).”

“…as this would be akin to comparing a text classification model and a question answering model. ious approaches for leveraging knowledge from a set of source tasks to target tasks, such as recent transfer learning methods in natural language processing (Peters et al., 2018; McCann et al., 2017).”

“In this line of work, McCann et al. [71] firstly proposed an inspiring context vector (CoVe) model where a deep long short-term memory (LSTM) encoder trained in a sequence-to-sequence machine translation (MT) model is used to generate contextualized embeddings which are then applied to a wide range…”

“Contextualized word embeddings like CoVe, ElMo, and BERT (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2018) improve a wide variety of natural language tasks (Wang et al., 2018; Rajpurkar et al., 2016; Socher et al., 2013; Conneau et al., 2018).”

“They further report that the representations obtained from the NMT model are better than those obtained from LMs. McCann et al. (2017) showed that using the representations obtained from the encoder of an NMT system as context vectors in downstream NLP tasks significantly improves performance over using only unsupervis…”

“Recently, the authors in [25] used the context vectors (CoVe) from a deep LSTM encoder in a bilingual NMT system to complement GloVe word vectors [26] and improve the performance on several tasks: sentiment analysis, question classification, entailment, and question answering.”

“(McCann et al., 2017).”

“Inspired by the success of Transformer (Vaswani et al. 2017) on a wide range of tasks in natural language processing (NLP) (McCann et al. 2017; Howard and Ruder 2018), Vision Transformer (ViT) (Dosovitskiy et al. 2021) first employed a pure Transformer architecture for image classifi-cation, which…”