Mimicking Word Embeddings using Subword RNNs
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Word embeddings improve generalization over lexical features by placing each word in a lower-dimensional space, using distributional information obtained from unlabeled data.However, the effectiveness of word embeddings for downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which embeddings do not exist.In this paper, we present MIM-ICK, an approach to generating OOV word embeddings compositionally, by learning a function from spellings to distributional embeddings.Unlike prior work, MIMICK does not require re-training on the original word embedding corpus; instead, learning is performed at the type level.Intrinsic and extrinsic evaluations demonstrate the power of this simple approach.On 23 languages, MIMICK improves performance over a word-based baseline for tagging part-of-speech and morphosyntactic attributes.It is competitive with (and complementary to) a supervised characterbased model in low-resource settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, it is hard to extract contextual information from characters to form meaningful words [50], [51], [52].”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [50]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Therefore, Log2Vec could not effectively handle some domain-specific words, such as technical terms or entity names [50], [51], [52].”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [50]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“(2015) propose a multi-level long shortterm memory (LSTM; Hochreiter and Schmidhuber, 1997), in which word embeddings are built compositionally from an LSTM over characters, and then tagging is performed by an LSTM over”
BackgroundEnriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 5×
“compare to FastText (Bojanowski et al., 2016), a high-vocabulary, high-dimensionality embedding benchmark.”
MethodResultInfluentialCharacter-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530 · cited 1×
“Character-based approaches avoid these problems: for example, Kim et al. (2016) train a recurrent neural network over words, whose embeddings are constructed by convolution over character embeddings; Wiet-ing et al. (2016) learn embeddings of character n-grams, and then sum them into word…”
BackgroundCharacter-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,491 · cited 1×
“Character-based approaches avoid these problems: for example, Kim et al. (2016) train a recurrent neural network over words, whose embeddings are constructed by convolution over character embeddings; Wiet-ing et al. (2016) learn embeddings of character n-grams, and then sum them into word…”
BackgroundBetter Word Representations with Recursive Neural Networks for Morphology
Luong, M.-T., Socher, R., & Manning, C. D. (2013) · Conference on Computational Natural Language Learning · Cited by 1,020
A Universal Part-of-Speech Tagset
Petrov, S., Das, D., & McDonald, R. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3pjyez8kmkhz · Cited by 722
Learning Character-level Representations for Part-of-Speech Tagging
dos Santos, C., & Zadrozny, B. (2014) · Cited by 519
Universal Stanford dependencies: A cross-linguistic typology
de Marneffe, M.-C., Dozat, T., Silveira, N., Haverinen, K., Ginter, F., Nivre, J., & Manning, C. D. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4ok3zdjx8t96 · Cited by 456 · cited 5×
“phosyntactic annotations through the Universal Dependencies (UD) corpus (De Marneffe et al., 2014). For example, Faruqui et al. (2016) propose a graph-based technique for propagating typelevel morphological information across a lexicon, improving token-level morphosyntactic tagging in 11 languages, using an SVM tagger.”
BackgroundMethodInfluentialEnriching Word Vectors with Subword Information
Bojanowski, P., Grave, É., Joulin, A., & Mikolov, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.04606 · Cited by 384 · cited 5×
“compare to FastText (Bojanowski et al., 2016), a high-vocabulary, high-dimensionality embedding benchmark.”
MethodResultInfluentialDyNet: The Dynamic Neural Network Toolkit
Neubig, G., Dyer, C., Goldberg, Y., Matthews, A., Ammar, W., Anastasopoulos, A., Ballesteros, M., Chiang, D., Clothiaux, D., Cohn, T., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.03980 · Cited by 302 · cited 2×
“We initialize character embeddings randomly, and use DyNet to implement the model (Neubig et al., 2017).”
Method
Research works printed but not linked: 8
- Željko Agić, Nikola Ljubešić, and Danijela Merkler. 2013. Lemmatization and morphosyntactic tagging of Croatian and Serbian. In 4th Biennial Interna- tional Workshop on Balto-Slavic Natural Language Processing (BSNLP 2013).
- Marta R Costa-jussà, David Aldón, and José AR Fonol- losa. 2017. Chinese-spanish neural machine trans- lation enhanced with character and word bitmap fonts. Machine Translation, pages 1-13.
- Marie-Catherine De Marneffe, Timothy Dozat, Na- talia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D Manning. 2014. Univer- sal stanford dependencies: A cross-linguistic typol- ogy. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 4585-4592.
- Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), Denver, CO.
- Jan Hajič and Barbora Hladká. 1998. Tagging inflec- tive languages: Prediction of morphological cate- gories for a rich, structured tagset. In Proceed- ings of the Association for Computational Linguis- tics (ACL), pages 483-490.
- Wang Ling, Tiago Luís, Luís Marujo, Ramón Fernan- dez Astudillo, Silvio Amir, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of Empirical Methods for Natural Language Process- ing (EMNLP).
- Thomas Müller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient higher-order CRFs for morpholog- ical tagging. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 322-332.
- Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the Association for Computational Linguistics (ACL), Berlin.
Cited by
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
“[38] provided an interesting approach Fig.”
Log-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
MethodBPE-Dropout: Simple and Effective Subword Regularization
Provilkov, I., Emelianenko, D., & Voita, E. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.170 · Cited by 218
Towards generalisable hate speech detection: a review on obstacles and solutions
Yin, W., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.598 · Cited by 207
“MIMICK (Pinter, Guthrie, & Eisenstein, 2017)-based model displayed similar performances (P. Mishra, Yannakoudakis, & Shutova, 2018).”
Naseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3434237 · Cited by 194
“raining and they are assigned UNK token and same vector for all words and degrades results if the number of OOV is large. Different methods to handle OOV words have been proposed in different studies [31, 48, 127] But still these models does not capture the polysemy issues. This issue of words with different meanings…”
BackgroundCanine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00448 · Cited by 140
WATSON: Abstracting Behaviors from Audit Logs via Aggregation of Contextual Semantics
Zeng, J., Chua, Z. L., Chen, Y., Ji, K., Liang, Z., & Mao, J. (2021) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24549 · Cited by 176
“Especially, we can further leverage NLP techniques [19], [44], [69] to learn the semantics of "outof-vocabulary" (unseen) audit events from their morphological information (e.”
BackgroundMultimodal deep networks for text and image-based document classification
Audebert, N., Herold, C., Slimani, K., & Vidal, C. (2019) · HAL · Cited by 125
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPPreprint
Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W. Y., Sagot, B., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10508 · Cited by 384
“Specifically, Pinter et al. (2017) learn a model that is trained to mimic the embedding of a word given its spelling using a helper RNN model that is called whenever an unknown word appears during test time.”
BackgroundEARL: Joint Entity and Relation Linking for Question Answering over Knowledge Graphs
Dubey, M., Banerjee, D., Chaudhuri, D., & Lehmann, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00671-6_7 · Cited by 111
DeepAG: Attack Graph Construction and Threats Prediction With Bi-Directional Deep Learning
Li, T., Jiang, Y. L., Lin, C., Obaidat, M. S., Shen, Y., & Ma, J. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3143551 · Cited by 97
“With the strategies of online update and trained OOV processor in the offline stage with MIMICK [27], DeepAG assigns a new embedding vector for unseen words when a new word appears in the online stage, and thus adapts new patterns of capricious intrusion.”
MethodProceedings of the BioNLP 2018 workshop
(2018) · doi:10.18653/v1/w18-23 · Cited by 71
A Semantic-aware Representation Framework for Online Log Analysis
Meng, W., Liu, Y., Huang, Y., Zhang, S., Zaiter, F., Chen, B., & Pei, D. (2020) · 2020 29th International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn49398.2020.9209707 · Cited by 63
“By training a model over the existing vocabulary, MIMICK can later use that model to predict the embedding of an unseen word.”
InfluentialReplicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
Dror, R., Baumer, G., Bogomolov, M., & Reichart, R. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00074 · Cited by 63
“Pinter et al. (2017) choose their languages so that they reflect a variety of typological, and particularly morphological, properties.”
BackgroundMethodInfluentialFew-Shot Representation Learning for Out-Of-Vocabulary Words
Hu, Z., Chen, T., Chang, K.-W., & Sun, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1402 · Cited by 94
“The first group of methods derives embedding vectors of OOV words based on their morphological information (Bojanowski et al., 2017; Kim et al., 2016; Pinter et al., 2017).”
BackgroundMethodOn the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling
Gerz, D., Vulić, I., Ponti, E. M., Reichart, R., & Korhonen, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1029 · Cited by 97
“For instance, a promising line of current and future research is to (learn to) exploit subword-level patterns captured in an unsupervised manner (Pinter et al., 2017; Herbelot and Baroni, 2017) or integrate existing morphological generation and inflection tools and regularities (Cotterell et al.,…”
BackgroundLanguage-Agnostic Visual-Semantic Embeddings
Wehrmann, J., Lopes, M. A., Souza, D. M., & Barros, R. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00590 · Cited by 54
“Unlike previous work [29, 17, 16] that generate word embeddings using characters or similar sub-word information together with RNNs, our approach is simple to implement and allows for fast word-embedding computation.”
Neural IR Meets Graph Embedding: A Ranking Model for Product Search
Zhang, Y., Wang, D., & Zhang, Y. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313468 · Cited by 71
“3A similar approach [35] is used by Pinter et al.”
MethodPota, M., Marulli, F., Esposito, M., Pietro, G. D., & Fujita, H. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.11.003 · Cited by 59
Misspelling Oblivious Word Embeddings
Piktus, A., Edizel, B., Bojanowski, P., Grave, É., Ferreira, R., & Silvestri, F. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1326 · Cited by 73
“An alternative approach, also capable of yielding representations for OOV words, is MIMIK (Pinter et al., 2017).”
Towards Locality-Aware Meta-Learning of Tail Node Embeddings on Networks
Liu, Z., Zhang, W., Fang, Y., Zhang, X., & Hoi, S. C. H. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411910 · Cited by 76
“Nevertheless, some studies also deal with sparse observations in other domains, such as cold-start recommendation [20, 33, 36] and rare word embedding [1, 15, 27].”
BackgroundAtzeni, M., & Reforgiato Recupero, D. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2019.10.012 · Cited by 50
Generalizing Word Embeddings using Bag of Subwords
Zhao, J., Mudgal, S., & Liang, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1059 · Cited by 53
“604 ICK (Pinter et al., 2017) is a character-level bidirectional LSTM word embedder trained against pre-trained word vectors (here Polyglot vectors 5).”
InfluentialDessì, D., Recupero, D. R., & Sack, H. (2021) · Electronics · doi:10.3390/electronics10070779 · Cited by 48
“More precisely, in one set of experiments, domain generated word embeddings are created through mimicking techniques; this allows to face slang, misspellings, or obfuscated contents not represented within pre-trained word embedding representations [24,44].”
BackgroundMethodAdvanced Rich Transcription System for Estonian Speech
Tanel, A. A. E., Tilk, O., & Asad, U. (2018) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-61499-912-6-1 · Cited by 44
“The approach is inspired from [19], but our model is unidirectional as our experiments did not show improvements with a bidirectional model, and uses gated recurrent units [20] instead of LSTM.”
MethodBERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance
Schick, T., & Schütze, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.368 · Cited by 40
Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts
Schick, T., & Schütze, H. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1048 · Cited by 60
“The comparably poor performance of Mim-ick is consistent with the observation of Pinter et al. (2017) that this method captures mostly syntactic information.”
BackgroundMethodResultInfluentialPilehvar, M. T., Kartsaklis, D., Prokhorov, V., & Collier, N. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1169 · Cited by 45
“Subword models alleviate this limitation by breaking the word into its subword (Pinter et al., 2017; Bojanowski et al., 2017) or morphological units (Luong et al.”
BackgroundResultInfluentialA Systematic Study of Leveraging Subword Information for Learning Word Representations
Zhu, Y., Vulić, I., & Korhonen, A. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1097 · Cited by 47
“Qiu et al., 2014; Cotterell and Sch¨utze, 2015; Wi-eting et al., 2016; Avraham and Goldberg, 2017; Vania and Lopez, 2017; Pinter et al., 2017; Cot-terell and Sch¨utze, 2018).”
BackgroundImproving Pre-Trained Multilingual Model with Vocabulary Expansion
Wang, H., Yu, D., Sun, K., Chen, J., & Yu, D. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1030 · Cited by 42
“, 2017) or develop hybrid word/subword-character architectures (Chung et al., 2016; Luong and Manning, 2016; Pinter et al., 2017; Bahdanau et al., 2017; Matthews et al., 2018; Li et al., 2018).”
BackgroundMethodCharacters or Morphemes: How to Represent Words?
Üstün, A. B., Kurfalı, M., & Can, B. (2018) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w18-3019 · Cited by 32
Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model
Mielke, S. J., & Eisner, J. M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016843 · Cited by 24
Multimodal Deep Networks for Text and Image-Based Document Classification
Audebert, N., Herold, C., Slimani, K., & Vidal, C. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_35 · Cited by 59
Learning Semantic Representations for Novel Words: Leveraging Both Form and Context
Schick, T., & Schütze, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016965 · Cited by 20
Subword-level Composition Functions for Learning Word Embeddings
Li, B., Валентинович, Д. А., Liu, T., & Du, X. (2018) · doi:10.18653/v1/w18-1205 · Cited by 25
“Pinter et al. (2017) also utilize BiLSTM to construct word embeddings.”
MethodA novel approach for text summarization using optimal combination of sentence scoring methods
Verma, P., & Om, H. (2019) · Sadhana · doi:10.1007/s12046-019-1082-4 · Cited by 26
“Hence, we handle these words by a mimicking approach using sub-word RNNs [33].”
MethodDeep Learning Models for Representing Out-of-Vocabulary Words
Lochter, J. V., Silva, R. M., & Almeida, T. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61377-8_29 · Cited by 25
“ding to Hu et al. [11], the representation methods able to learning new representations to handle OOV usually employ two different sources of information: the morphological structure of the OOV words [21, 4], or the context in which the OOV is inserted [10, 11]. FastText [4] A popular distributed representation model…”
BackgroundMethodLearning to Generate Word Representations using Subword Information
Kim, Y., Kim, K.-M., Lee, J.-M., & Lee, S. (2018) · International Conference on Computational Linguistics · Cited by 22
Wang, X., Dou, S., Xiong, L., Zou, Y., Zhang, Q., Gui, T., Qiao, L., Cheng, Z., & Huang, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.383 · Cited by 23
Predicting and interpreting embeddings for out of vocabulary words in downstream tasks
Garneau, N., Leboeuf, J.-S., & Lamontagne, L. (2018) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w18-5439 · Cited by 22
“An interesting way to handle OOV words is the Mimick model (Pinter et al., 2017).”
MethodNguyen, M., Ngo, G. H., & Chen, N. F. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2955246 · Cited by 23
“[68]–[70] proposed building embeddings of new words from pre-trained embeddings by learning mapping from characters to embeddings.”
BackgroundPredicting Semantic Relations using Global Graph Properties
Pinter, Y., & Eisenstein, J. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1201 · Cited by 25
“5 To handle these cases, we trained a character-based model called M IMICK , which learns to predict embed-dings for out-of-vocabulary items based on their spellings (Pinter et al., 2017).”
MethodSelf-citationToo Much in Common: Shifting of Embeddings in Transformer Language Models and its Implications
Biś, D., Podkorytov, M., & Liu, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.403 · Cited by 73
Cold Item Recommendations via Hierarchical Item2vec
Barkan, O., Caciularu, A., Rejwan, I., Katz, O., Weill, J., Malkiel, I., & Koenigstein, N. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00101 · Cited by 30
“Indeed, it was repeatedly shown that language models suffer from poor representations of long-tail entities [7]–[10] that cannot be solely mitigated by co-occurrence statistics (due to the persistence of rare and out-of-vocabulary word and phrases).”
BackgroundEnML: Multi-label Ensemble Learning for Urdu Text Classification
Mehmood, F., Shahzadi, R., Ghafoor, H., Asim, M., Ghani, M., Mahmood, W., & Dengel, A. (2023) · ACM Trans. Asian Low Resour. Lang. Inf. Process. · doi:10.1145/3616111 · Cited by 20
“Most widely used embedding dimensions suggested in the literature are 64, 128, and 256 [64, 82, 84].”
MethodGraph-Based Bilingual Word Embedding for Statistical Machine Translation
Wang, R., Zhao, H., Ploux, S., Lu, B., Utiyama, M., & Sumita, E. (2018) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3203078 · Cited by 21
“(iii) The third category uses subword unit such as characters or the dictionary as the “context” information [62, 75, 85].”
BackgroundDo, Q., Moriyani, M. A., Le, C., & Le, T. (2023) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-13023 · Cited by 19
Proceedings of The Third Workshop on Representation Learning for NLP
(2018) · doi:10.18653/v1/w18-30 · Cited by 20
Hasni, S., & Faïz, S. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00777-5 · Cited by 17
A Comparison of Character and Word Embeddings in Bidirectional LSTMs for POS Tagging in Italian
Marulli, F., Pota, M., & Esposito, M. (2018) · Smart innovation, systems and technologies · doi:10.1007/978-3-319-92231-7_2 · Cited by 18
Robust Word Vectors: Context-Informed Embeddings for Noisy Texts
Malykh, V., Logacheva, V., & Khakhulin, T. (2018) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/w18-6108 · Cited by 17
On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages
Zhu, Y., Heinzerling, B., Vulić, I., Strube, M., Reichart, R., & Korhonen, A. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1021 · Cited by 30
“each word as an atomic unit, subword-informed neural architectures reduce data sparsity by relying on parameterization at the level of subwords (Bojanowski et al., 2017; Pinter et al., 2017; Chaudhary et al., 2018; Kudo, 2018).”
MethodMulti-level out-of-vocabulary words handling approach
Lochter, J. V., Silva, R. M., & de Almeida, T. D. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108911 · Cited by 17
Glyph2Vec: Learning Chinese Out-of-Vocabulary Word Embedding from Glyphs
Chen, H.-Y., YU, S.-H., & Lin, S.-D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.256 · Cited by 20
“M is for Mimick baseline (Pinter et al., 2017) based on the authors’ code.”
MethodInfluentialRobust Backed-off Estimation of Out-of-Vocabulary Embeddings
Fukuda, N., Yoshinaga, N., & Kitsuregawa, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.434 · Cited by 23
“…surface information in computing OOV word embeddings basically learn the embeddings of characters or subwords to reconstruct pre-trained word embeddings from them and then use the obtained embeddings to compute embeddings for OOV words (Pinter et al., 2017; Zhao et al., 2018; Sasaki et al., 2019).”
MethodYuan, L.-P., Liu, P., & Zhu, S. (2021) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3433210.3453098 · Cited by 18
“The other option is to leverage a pre-trained Word2Vec embedding in building a new embedding model (e.g. Mimick embeding [40]).”
BackgroundDual Long Short-Term Memory Networks for Sub-Character Representation Learning
He, H. W., Wu, L., Yang, X., Yan, H., Gao, Z., Feng, Y., & Townsend, G. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-77028-4_55 · Cited by 14
Viability of Neural Networks for Core Technologies for Resource-Scarce Languages
Loubser, M., & Puttkammer, M. J. (2020) · Information · doi:10.3390/info11010041 · Cited by 20
Handling Out-Of-Vocabulary Problem in Hangeul Word Embeddings
Kwon, O., Kim, D., Lee, S.-R., Choi, J., & Lee, S. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.280 · Cited by 24
“This demonstrates the importance of pre-trained embedding knowledge when creating new embeddings (Pinter et al., 2017).”
BackgroundMulti-level embeddings for processing Arabic social media contents
Moudjari, L., Benamara, F., & Akli‐Astouati, K. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101240 · Cited by 19
Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation
Liu, X., Yang, B., Liu, D., Zhang, H., Luo, W., Zhang, M., Zhang, H., & Su, J. (2021) · doi:10.18653/v1/2021.acl-long.468 · Cited by 23
“In this context, researchers use embeddings of characters or subwords to predict those of unseen words (Pinter et al., 2017; Zhao et al., 2018; Sasaki et al., 2019; Fukuda et al., 2020).”
BackgroundQuantifying the Contextualization of Word Representations with Semantic Class Probing
Zhao, M., Dufter, P., Yaghoobzadeh, Y., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.109 · Cited by 24
Imputing Out-of-Vocabulary Embeddings with LOVE Makes LanguageModels Robust with Little Cost
Chen, L., Varoquaux, G. P., & Suchanek, F. M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.245 · Cited by 21
Enhanced Highway Project Clustering Framework to Support Project Cost Estimation
Do, Q., Moriyani, M. A., Le, T., Le, C., & Piratla, K. R. (2024) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-15194 · Cited by 13
Subword-based Compact Reconstruction of Word Embeddings
Sasaki, S., Suzuki, J., & Inui, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1353 · Cited by 14
“Another study that shares the same motivation and goal is that of Pinter et al. (2017).”
BackgroundMethodNYTWIT: A Dataset of Novel Words in the New York Times
Pinter, Y., Jacobs, C. L., & Bittker, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.572 · Cited by 14
Self-citationContextual Generation of Word Embeddings for Out of Vocabulary Words in Downstream Tasks
Garneau, N., Leboeuf, J.-S., & Lamontagne, L. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18305-9_60 · Cited by 11
Exploiting Cross-Lingual Subword Similarities in Low-Resource Document Classification
Zhang, M., Fujinuma, Y., & Boyd-Graber, J. L. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6500 · Cited by 11
Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality
Aguilar, G., McCann, B., Niu, T., Rajani, N. F., Keskar, N. S., & Solorio, T. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.141 · Cited by 23
“Pinter et al. (2017) proposed learning OOV words from a pre-trained word embedding dictionary.”
BackgroundRelational data embeddings for feature enrichment with background information
Cvetkov-Iliev, A., Allauzen, A., & Varoquaux, G. P. (2023) · Machine Learning · doi:10.1007/s10994-022-06277-7 · Cited by 34
“Further work should explore automated techniques, for instance using fuzzy or similarity joins [29,42], or adapting NLP techniques used to create embeddings robust to out-of-vocabulary entities [8,35,12].”
MethodRepresentation Learning via Variational Bayesian Networks
Barkan, O., Caciularu, A., Rejwan, I., Katz, O., Weill, J., Malkiel, I., & Koenigstein, N. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482363 · Cited by 11
“However, it was shown that even when increasing the amount of co-occurrence data, the existence of rare, out-of-vocabulary entities persists [26, 50, 52, 53].”
BackgroundRare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking
Schick, T., & Schütze, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6403 · Cited by 83
Yu, H., An, J., Yoon, J., Kim, H., & Ko, Y. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.04.009 · Cited by 11
Guo, D., Dong, G., Fu, D., Wu, Y., Zeng, C., Hui, T., Wang, L., Li, X., Wang, Z., He, K., et al. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094766 · Cited by 11
“MIMICK[6] and LOVE[7] focused on the surface form of words to generate vectors for OOV words.”
BackgroundCharacter Eyes: Seeing Language through Character-Level Taggers
Pinter, Y., Marone, M., & Eisenstein, J. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4811 · Cited by 10
“previously been found to affect performance in character-level models (Pinter et al., 2017; Gerz et al., 2018; Chaudhary et al., 2018), and thus we select representatives of each group.”
BackgroundMethodSelf-citationNeural morphosyntactic tagging for Rusyn
Scherrer, Y., & Rabus, A. (2019) · Natural Language Engineering · doi:10.1017/s1351324919000287 · Cited by 14
“Therefore, most recent work reports averages over per-feature F1-scores, either macro-averages (Buys and Botha 2016; Cotterell and Heigold 2017) or micro-averages (Pinter, Guthrie, and Eisenstein 2017).”
BackgroundMethodLearning Task-Specific Representation for Novel Words in Sequence Labeling
Peng, M., Zhang, Q., Xing, X., Gui, T., Fu, J., & Huang, X. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/715 · Cited by 12
Learning to Learn Words from Visual Scenes
Surís, D., Epstein, D., Ji, H., Chang, S., & Vondrick, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58526-6_26 · Cited by 12
A Systematic Study of Leveraging Subword Information for Learning Word RepresentationsPreprint
Zhu, Y., Vulić, I., & Korhonen, A. (2019) · arXiv (Cornell University) · doi:10.17863/cam.39780 · Cited by 8
Galea, D., Laponogov, I., & Veselkov, K. (2018) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/w18-2307 · Cited by 11
“It would be interesting to assess how embedding models such as MIMICK, where the word2vec space can be pre-served while still being able to generate character - based vectors for OOV terms, compare.”
BackgroundMethodInfluentialWon, M.-S., Choi, Y., Kim, S., Na, C., & Lee, J.-H. (2021) · ACM Symposium on Applied Computing · doi:10.1145/3412841.3441982 · Cited by 11
Multi-task Learning Model for Detecting Internet Slang Words with Two-Layer Annotation
Seki, Y., & Liu, Y. (2022) · International Conference on Asian Language Processing · doi:10.1109/ialp57159.2022.9961254 · Cited by 7
“[7] proposed the Mimick model, which learned a function from spelling to embedding based on subword information, using existing embeddings to generate embeddings for OOV words and obtaining excellent results with part-of-speech (POS) tagging tasks in 23 languages.”
BackgroundMisspelling Oblivious Word EmbeddingsPreprint
Edizel, B., Piktus, A., Bojanowski, P., Ferreira, R., Grave, É., & Silvestri, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09755 · Cited by 26
“An alternative approach, also capable of yielding representations for OOV words, is MIMIK (Pinter et al., 2017).”
Graph-based Relation Mining for Context-free Out-of-vocabulary Word Embedding Learning
Liang, Z., Lu, Y., Chen, H., & Rao, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.790 · Cited by 8
“They represented the word form information through characters (Pinter et al., 2017), sub-units (Zhao et al., 2018; Zhang et al., 2019; Sasaki et al., 2019; Fukuda et al., 2020; Chen et al., 2022), images (Chen et al., 2020a), and so forth.”
BackgroundMethodInfluentialOut-of-vocabulary word embedding learning based on reading comprehension mechanism
Zhuang, Z., Liang, Z., Rao, Y., Xie, H., & Wang, F. L. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100038 · Cited by 12
Efficient Contextual Representation Learning With Continuous Outputs
Li, L. H., Chen, P. H., Hsieh, C., & Chang, K.-W. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00289 · Cited by 10
“Open-vocabulary word embeddings, such as the FastText embedding and the MIMICK model (Pinter et al., 2017), utilize character or subword information to provide word embeddings.”
MethodQuestion Classification by Convolutional Neural Networks Embodying Subword Information
Pota, M., & Esposito, M. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489406 · Cited by 6
“Some works [8,9,19,31-34] studied the effect of these decisions on accuracy of text classification tasks.”
BackgroundMethodAdaptation of Morphosyntactic Taggers
Zampieri, M., & Nakov, P. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108565080.011 · Cited by 6
Huang, J., Wu, Z., Mahowald, K., & Potts, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.770 · Cited by 24
Locality-Aware Tail Node Embeddings on Homogeneous and Heterogeneous Networks
Liu, Z., Fang, Y., Zhang, W., Zhang, X., & Hoi, S. C. H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3313355 · Cited by 7
“Meanwhile, some studies deal with sparse observations in other domains, such as cold-start recommendation [5], [33], [34] and rare word embedding [6], [10], [35].”
BackgroundQuantifying the Contextualization of Word Representations with Semantic Class Probing
Zhao, M., Dufter, P., Yaghoobzadeh, Y., & Schütze, H. (2020) · Open access LMU (Ludwid Maxmilian's Universitat Munchen) · doi:10.5282/ubm/epub.74039 · Cited by 8
Subwords-Only Alternatives to fastText for Morphologically Rich Languages
Ghukasyan, T., Yeshilbashian, Y. M., & Avetisyan, K. (2021) · Programming and Computer Software · doi:10.1134/s0361768821010059 · Cited by 5
“Earlier attempts concentrated on reconstruction of pretrained embeddings [1–4].”
BackgroundMethodVirk, S. M., Foster, D. G., Muhammad, A. S., & Saleem, R. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_166 · Cited by 5
“The embedding inferring technique has been proposed by Pinter et al. (2017) in which a character-based bidirectional LSTM was used to infer vectors for a list of unknown words and to add them to the existing word-embeddings.”
Embedding for Out of Vocabulary Words Considering Contextual and Morphosyntactic Information
Won, M.-S., & Lee, J.-H. (2018) · International conference on Fuzzy Theory and Its Applications · doi:10.1109/ifuzzy.2018.8751687 · Cited by 5
“Yuval Pinter trained a characterbased model with Bidirectional Recurrent Neural Network (Bi-RNN) that captures morphosyntactic information of the word [7].”
MethodSecond-order contexts from lexical substitutes for few-shot learning of word representations
Liu, Q., McCarthy, D., & Korhonen, A. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1007 · Cited by 8
“…word representations can be categorized into the following three approaches: (1) constructing target word embeddings from the subword components (Pinter et al., 2017; Bojanowski et al., 2017), (2). leveraging definitions or relational structures from ex-ternal resources such as Wordnet (Bahdanau…”
BackgroundInfluentialEstimator Vectors: OOV Word Embeddings based on Subword and Context Clue Estimates
Patel, R. N., & Domeniconi, C. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207711 · Cited by 7
“this problem is to estimate OOV word representations using word roots or subwords [3], [19].”
BackgroundPBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding
Zhao, J., Zhong, S., Zhang, X., & Liang, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.53 · Cited by 5
A Novel Bilingual Word Embedding Method for Lexical Translation Using Bilingual Sense CliquePreprint
Wang, R., Zhao, H., Ploux, S., Lu, B., Utiyama, M., & Sumita, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.08692 · Cited by 6
“(iii) The third category uses subword unit such as characters or the dictionary as the “context” information [62, 75, 85].”
Background
Metadata
- Authors
- Pinter, Y., Guthrie, R., & Eisenstein, J.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Pinter, Yuval
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 102-112
- OpenAlex
- W2737092125
- Cited by
- 177
- References
- 20
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 6 · 2023: 14 · 2022: 17 · 2021: 35 · 2020: 29 · 2019: 40 · 2018: 27 · 2017: 2
- FWCI
- 12.22
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Threat Reduction Agency (HDTRA1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- word embeddings, out-of-vocabulary words, distributional semantics, part-of-speech tagging, morphosyntactic tagging, low-resource languages, multilingual word representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

