The DOI matches its Crossref record; 68 of the 69 references it cites were found in a registry or an index.
Unsupervised Domain Clusters in Pretrained Language Models
Open accessAbstract
The notion of "in-domain data" in NLP is often over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality.In addition, domain labels are many times unavailable, making it challenging to build domainspecific systems.We show that massive pretrained language models implicitly learn sentence representations that cluster by domains without supervision -suggesting a simple datadriven definition of domains in textual data.We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data.We evaluate our data selection methods for neural machine translation across five diverse domains, where they outperform an established approach as measured by both BLEU and by precision and recall of sentence selection with respect to an oracle.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-4009 · Cited by 2,475
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4828 · Cited by 1,953
(2018) · International Conference on Learning Representations · Cited by 1,621
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Parallel Data, Tools and Interfaces in OPUS
Tiedemann, J. (2012) · Language Resources and Evaluation · Cited by 1,421
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
KenLM: Faster and Smaller Language Model Queries
Heafield, K. (2011) · Cited by 1,145
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2025 · Cited by 729
Transfer Learning in Natural Language Processing
Ruder, S., Peters, M. E., Swayamdipta, S., & Wolf, T. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-5004 · Cited by 712
Intelligent Selection of Language Model Training Data
Moore, R. C., & Lewis, W. D. (2010) · Cited by 530
Findings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa‐jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5301 · Cited by 519
Domain Adaptation via Pseudo In-Domain Data Selection
Axelrod, A. E., He, X., & Gao, J. (2011) · Cited by 473
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual GeneralizationPreprint
Hu, J., Ruder, S., Siddhant, A., Neubig, G., Fırat, O., & Johnson, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11080 · Cited by 400
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385
Self-citationFacebook FAIR’s WMT19 News Translation Task Submission
Ng, N., Yee, K., Baevski, A., Ott, M., Auli, M., & Edunov, S. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5333 · Cited by 308
A Survey of Domain Adaptation for Neural Machine TranslationPreprint
Chu, C.-H., & Wang, R. (2018) · International Conference on Computational Linguistics · doi:10.48550/arxiv.1806.00258 · Cited by 301
The Frequency and Use of Lexical Bundles in Conversation and Academic Prose
Conrad, S., & Biber, D. (2005) · Lexicographica · doi:10.1515/9783484604674.56 · Cited by 145 · cited 1×
“This may be expected as the subtitles corpus mainly includes transcriptions of spoken, conversational language, and “conversation tends to have more verbs, more personal pronouns, and more questions” (Conrad and Biber, 2005).”
BackgroundCurriculum Learning for Domain Adaptation in Neural Machine Translation
Zhang, X., Shapiro, P., Kumar, G., McNamee, P., Carpuat, M. J., & Duh, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1189 · Cited by 115
Multi-Domain Neural Machine Translation through Unsupervised Adaptation
Farajian, M. A., Turchi, M., Negri, M., & Federico, M. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4713 · Cited by 115
(2018) · Cited by 115
Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation
Duh, K., Neubig, G., Sudoh, K., & Tsukada, H. (2013) · Cited by 107
Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora
Junczys-Dowmunt, M. (2018) · Proceedings of the Third Conference on Machine Translation: Shared Task Papers · doi:10.18653/v1/w18-6478 · Cited by 106
Domain Adaptation with BERT-based Domain Classification and Data Selection
Ma, X., Xu, P., Wang, Z., Nallapati, R. M., & Xiang, B. (2019) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/d19-6109 · Cited by 99
Active learning for statistical phrase-based machine translation
Haffari, G., Roy, M., & Sarkar, A. (2009) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1620754.1620815 · Cited by 97
Sentence Embedding for Neural Machine Translation Domain Adaptation
Wang, R., Finch, A., Utiyama, M., & Sumita, E. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2089 · Cited by 90
Non-Parametric Adaptation for Neural Machine Translation
Bapna, A., & Fırat, O. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1191 · Cited by 88
Domain Robustness in Neural Machine Translation
Müller, M., Rios, A., & Sennrich, R. (2019) · arXiv · doi:10.48550/arxiv.1911.03109 · Cited by 68
Domain Adaptation of Neural Machine Translation by Lexicon Induction
Hu, J., Xia, M., Neubig, G., & Carbonell, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1286 · Cited by 63
Dynamic Topic Adaptation for Phrase-based MT
Hasler, E., Blunsom, P., Koehn, P., & Haddow, B. (2014) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1035 · Cited by 61
A Study of Style in Machine Translation: Controlling the Formality of Machine Translation Output
Niu, X., Martindale, M. J., & Carpuat, M. J. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1299 · Cited by 56
When and Why is Unsupervised Neural Machine Translation Useless?Preprint
Kim, Y., Graça, M., & Ney, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10581 · Cited by 54
Wang, W., Caswell, I. R., & Chelba, C. I. (2019) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p19-1123 · Cited by 49
Survey of data-selection methods in statistical machine translation
Eetemadi, S., Lewis, W. D., Toutanova, K., & Radha, H. (2015) · Machine Translation · doi:10.1007/s10590-015-9176-1 · Cited by 47
Does more data always yield better translations
Gascó, G., Rocha, M.-A., Sanchis-Trilles, G., Andrés-Ferrer, J., & Casacuberta, F. (2012) · Cited by 46
Split and Rephrase: Better Evaluation and Stronger Baselines
Aharoni, R., & Goldberg, Y. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2114 · Cited by 46
Self-citationNeural Machine Translation Training in a Multi-Domain ScenarioPreprint
Sajjad, H., Durrani, N., Dalvi, F., Belinkov, Y., & Vogel, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.08712 · Cited by 46
When Does Unsupervised Machine Translation Work?
Marchisio, K., Duh, K., & Koehn, P. (2020) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2020.wmt-1.68 · Cited by 37
Learning to select data for transfer learning with Bayesian Optimization
Ruder, S., & Plank, B. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1038 · Cited by 31
Unsupervised Domain Adaptation for Neural Machine Translation with Domain-Aware Feature Embeddings
Dou, Z.-Y., Hu, J., Anastasopoulos, A., & Neubig, G. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1147 · Cited by 30
What’s in a Domain? Analyzing Genre and Topic Differences in Statistical Machine Translation
van der Wees, M., Bisazza, A., Weerkamp, W., & Monz, C. (2015) · doi:10.3115/v1/p15-2092 · Cited by 22
Extracting In-domain Training Corpora for Neural Machine Translation Using Data Selection Methods
Silva, C. C., Liu, C.-H., Poncelas, A., & Way, A. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6323 · Cited by 22
Emerging Cross-lingual Structure in Pretrained Language ModelsPreprint
Wu, S., Conneau, A., Li, H., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01464 · Cited by 21
Domain Differential Adaptation for Neural Machine Translation
Dou, Z.-Y., Wang, X., Hu, J., & Neubig, G. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5606 · Cited by 17
Neural Networks Classifier for Data Selection in Statistical Machine Translation
Peris, Á., Chinea-Ríos, M., & Casacuberta, F. (2017) · The Prague Bulletin of Mathematical Linguistics · doi:10.1515/pralin-2017-0027 · Cited by 12
Data Selection with Feature Decay Algorithms Using an Approximated Target SidePreprint
Poncelas, A., Wenniger, G. M. D. B., & Way, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03039 · Cited by 12
Aulamo, M., & Tiedemann, J. (2019) · DSpace repository (University of Tartu) · Cited by 12
Adapting to All Domains at Once: Rewarding Domain Invariance in SMT
Cuong, H., Sima’an, K., & Titov, I. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00086 · Cited by 11
Data selection for NMT using Infrequent n-gram Recovery
Parcheta, Z., Sanchis-Trilles, G., & Nolla, F. C. (2018) · RUA, Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante) · Cited by 11
Cynical Selection of Language Model Training DataPreprint
Axelrod, A. E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02279 · Cited by 11
Data Selection for IT Texts using Paragraph Vector
Duma, M.-S., & Menzel, W. (2016) · doi:10.18653/v1/w16-2331 · Cited by 8
Data Selection with Cluster-Based Language Difference Models and Cynical SelectionPreprint
Santamaría, L., & Axelrod, A. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04900 · Cited by 8
Scalable Evaluation and Improvement of Document Set Expansion via Neural Positive-Unlabeled Learning
Jacovi, A., Niu, G., Goldberg, Y., & Sugiyama, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.47 · Cited by 4
Self-citationWhat’s in a domain?: Towards fine-grained adaptation for machine translation
van der Wees, M. (2017) · UvA-DARE (University of Amsterdam) · Cited by 4
Research works printed but not linked: 8
- Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Ses- sions, pages 177-180, Prague, Czech Republic. As- sociation for Computational Linguistics.
- Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proceed- ings of the First Workshop on Neural Machine Trans- lation, pages 28-39, Vancouver. Association for Computational Linguistics. · DOI · Paper page
- Christopher D Manning, Prabhakar Raghavan, and Hin- rich Schütze. 2008. Introduction to information re- trieval. Cambridge university press.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119.
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguistics. · DOI · Paper page
- Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, R'emi Louf, Morgan Funtow- icz, and Jamie Brew. 2019. Huggingface's trans- formers: State-of-the-art natural language process- ing. ArXiv, abs/1910.03771. · arXiv
Cited by
Sorted by citations
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios
Hedderich, M. A., Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.201 · Cited by 310
Neural Unsupervised Domain Adaptation in NLP—A Survey
Ramponi, A., & Plank, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.603 · Cited by 311
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisPreprint
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03545 · Cited by 154
Metadata
- Authors
- Aharoni, R., & Goldberg, Y.
- Affiliations
- Google (United States) (US)
- Bar-Ilan University (IL)
- ORCID
- Goldberg, Yoav
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 7747-7763
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3014390292
- Cited by
- 181
- References
- 69
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 21 · 2023: 52 · 2022: 34 · 2021: 43 · 2020: 12
- FWCI
- 13.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- pre-trained language models, domain adaptation, neural machine translation, domain-specific translation, sentence selection, monolingual data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
