Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 16

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.

Nonparametric Masked Language Modeling

Min, Sewon, Shi, Weijia, Lewis, Mike, Chen, Xilun, Yih, Wen-tau, Hajishirzi, Hannaneh, Zettlemoyer, Luke·2023·Findings of the Association for Computational Linguistics: ACL 2023
DOI 10.18653/v1/2023.findings-acl.132
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-acl.132

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 An illustration of NPM. The encoder maps a masked sentence into a dense vector, and retrieves the nearest phrase from a reference corpus . NPM can fill in the [MASK] with multiple tokens, e.g., Thessaloniki (4 BPE tokens) and unseen words, e.g., ᄇ ᅡ ᆫ ᄑ ᅩᄃ ᅢᄀ ᅭ (12 BPE tokens).
Figure 1 An illustration of NPM. The encoder maps a masked sentence into a dense vector, and retrieves the nearest phrase from a reference corpus . NPM can fill in the [MASK] with multiple tokens, e.g., Thessaloniki (4 BPE tokens) and unseen words, e.g., ᄇ ᅡ ᆫ ᄑ ᅩᄃ ᅢᄀ ᅭ (12 BPE tokens).CC-BY · From the PDF
Figure 2 Inference of NPM (Section 3.1). Each token in the reference corpus C is mapped into a dense vector space. At test time, a query is represented as two vectors, q start and q end , each in the same vector space. We use a nearest neighbor search to retrieve the start and the end of the phrase using q start and q end , respectively.
Figure 2 Inference of NPM (Section 3.1). Each token in the reference corpus C is mapped into a dense vector space. At test time, a query is represented as two vectors, q start and q end , each in the same vector space. We use a nearest neighbor search to retrieve the start and the end of the phrase using q start and q end , respectively.CC-BY · From the PDF
Figure 3 Training of NPM. (Section 3.2). [MASK s ][MASK e ] indicates the masked span whose original phrase is the Seattle Seahawks . We maximize the similarity scores between ... [MASK s ] [MASK e ] ... and ... the Seattle Seahawks... , and between ... [MASK s ] [MASK e ] ... and ...the Seattle Seahawks ... .
Figure 3 Training of NPM. (Section 3.2). [MASK s ][MASK e ] indicates the masked span whose original phrase is the Seattle Seahawks . We maximize the similarity scores between ... [MASK s ] [MASK e ] ... and ... the Seattle Seahawks... , and between ... [MASK s ] [MASK e ] ... and ...the Seattle Seahawks ... .CC-BY · From the PDF
Figure 5 Predictions from RoBERTa (baseline) and NPM. The bottom indicates the context NPM retrieves to fill in [MASK] . Note that the fuzzy verbalizer maps broken to Negative and awesome to Positive .
Figure 5 Predictions from RoBERTa (baseline) and NPM. The bottom indicates the context NPM retrieves to fill in [MASK] . Note that the fuzzy verbalizer maps broken to Negative and awesome to Positive .CC-BY · From the PDF
Figure 6 Zero-shot results on knowledge tasks. The x -axis indicates the relative number of model parameters in log scale compared to RoBERTa large (354M). NPM outperforms significantly larger parameters models, either with or without BM25. See Table 8 in Appendix C for the raw numbers.
Figure 6 Zero-shot results on knowledge tasks. The x -axis indicates the relative number of model parameters in log scale compared to RoBERTa large (354M). NPM outperforms significantly larger parameters models, either with or without BM25. See Table 8 in Appendix C for the raw numbers.CC-BY · From the PDF
Figure 7 Ablation on the size of the reference corpus, from 41M tokens (5%) to 810M tokens (100%). There is a strong correlation between the size of the corpus and downstream performance.
Figure 7 Ablation on the size of the reference corpus, from 41M tokens (5%) to 810M tokens (100%). There is a strong correlation between the size of the corpus and downstream performance.CC-BY · From the PDF
Figure 8 Performance on LAMA and TempLAMA tasks, broken down based on the number of BPE splits of the target entity, which is an indication of rarity of the entities (L:Frequent → R:Rare). NPM outperforms GPT-3 or T5 more significantly when the target entities are rare.
Figure 8 Performance on LAMA and TempLAMA tasks, broken down based on the number of BPE splits of the target entity, which is an indication of rarity of the entities (L:Frequent → R:Rare). NPM outperforms GPT-3 or T5 more significantly when the target entities are rare.CC-BY · From the PDF
Image 8
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 24 references

Research works printed but not linked: 57

  1. Uri Alon, Frank Xu, Junxian He, Sudipta Sengupta, Dan Roth, and Graham Neubig. 2022. Neuro-symbolic 2106 language modeling with automaton-augmented re- trieval. In Proceedings of the International Confer- ence of Machine Learning.
  2. Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question an- swering. In Conference of the North American Chap- ter of the Association for Computational Linguistics.
  3. Bogdan Babych and Anthony F. Hartley. 2003. Improv- ing machine translation quality with automatic named entity recognition. Proceedings of the International EAMT workshop on MT and other Language Tech- nology Tools, Improving MT through other Language Technology Tools Resources and Tools for Building MT.
  4. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In Proceedings of the International Conference of Machine Learning.
  5. Wenlin Chen, David Grangier, and Michael Auli. 2016. Strategies for training large vocabulary neural lan- guage models. In Proceedings of the Association for Computational Linguistics.
  6. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in ty- pologically diverse languages. Transactions of the Association for Computational Linguistics.
  7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop.
  8. Xiang Deng, Yu Su, Alyssa Lees, You Wu, Cong Yu, and Huan Sun. 2021. ReasonBERT: Pre-trained to reason with distant supervision. In Proceedings of Empirical Methods in Natural Language Processing.
  9. Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguistics.
  10. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the International Confer- ence on Language Resources and Evaluation.
  11. William T Freeman, Thouis R Jones, and Egon C Pasz- tor. 2002. Example-based super-resolution. IEEE Computer graphics and Applications.
  12. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vin- cent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2022. Attributed text generation via post-hoc research and revision. arXiv preprint arXiv:2210.08726. · arXiv
  13. Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2014. Optimized product quantization. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  14. Édouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for GPUs. In Proceedings of the International Conference of Machine Learning.
  15. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In Proceedings of the Interna- tional Conference of Machine Learning.
  16. Ahmed Hassan, Haytham Fahmy, and Hany Hassan. 2007. Improving named entity translation by exploit- ing comparable and parallel corpora. In Proceedings of the International Workshop on Acquisition and Management of Multilingual Lexicons.
  17. Junxian He, Graham Neubig, and Taylor Berg- Kirkpatrick. 2021. Efficient nearest neighbor lan- guage models. In Proceedings of Empirical Methods in Natural Language Processing.
  18. Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competi- tion: Why the highest probability answer isn't always right. In Proceedings of Empirical Methods in Natu- ral Language Processing.
  19. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299. · arXiv
  20. Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. 2020. A memory efficient baseline for open domain ques- tion answering. arXiv preprint arXiv:2012.15156. · arXiv
  21. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022. Temporalwiki: A lifelong bench- mark for training and evaluating ever-evolving lan- guage models. In Proceedings of Empirical Methods in Natural Language Processing.
  22. Jan-Christoph Kalo and Leandra Fichtel. 2022. Kamel: Knowledge analysis with multitoken entities in lan- guage models. In Proceedings of the Conference on Automated Knowledge Base Construction.
  23. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. arXiv preprint arXiv:2211.08411. · arXiv
  24. Nora Kassner and Hinrich Schütze. 2020. BERT-kNN: Adfding a kNN search component to pretrained lan- guage models for better QA. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020.
  25. Mayank Kulkarni, Debanjan Mahata, Ravneet Arora, and Rajarshi Bhowmik. 2022. Learning rich repre- sentation of keyphrases from text. In Findings of the Association for Computational Linguistics: NAACL 2022.
  26. Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. 2023. Copy is all you need. In Pro- ceedings of the International Conference on Learning Representations.
  27. Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021. Learning dense representations of phrases at scale. In Proceedings of the Association for Computational Linguistics.
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  29. Joel Mackenzie, Rodger Benham, Matthias Petri, Jo- hanne R Trippas, J Shane Culpepper, and Alistair Moffat. 2020. Cc-news-en: A large english news corpus. In Proceedings of the ACM International Conference on Information and Knowledge Manage- ment.
  30. Julian McAuley and Jure Leskovec. 2013. Hidden fac- tors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the ACM conference on Recommender systems.
  31. Robert C Moore. 2003. Learning translations of named- entity phrases from parallel corpora. In Proceedings of the European Chapter of the Association for Com- putational Linguistics.
  32. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Pro- ceedings of the International workshop on artificial intelligence and statistics.
  33. Mohammad Norouzi, Ali Punjani, and David J Fleet. 2012. Fast search in hamming space with multi- index hashing. In 2012 IEEE conference on computer vision and pattern recognition, pages 3108-3115. IEEE.
  34. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. · arXiv
  35. Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the Association for Computational Linguistics.
  36. Nikolaos Pappas, Phoebe Mulcaire, and Noah A. Smith. 2020. Grounded compositional outputs for adaptive language modeling. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1252-1267, Online. As- sociation for Computational Linguistics. · DOI
  37. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Proceed- ings of Advances in Neural Information Processing Systems.
  38. Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison- Burch. 2022. Bidirectional language models are also few-shot learners. arXiv preprint arXiv:2209.14500. · arXiv
  39. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of Empirical Methods in Natural Language Processing.
  40. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. Bert is not a knowledge base (yet): Factual knowl- edge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv:1911.03681. · arXiv
  41. Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Glober- son, and Omer Levy. 2021. Few-shot question an- swering by pretraining span selection. In Proceed- ings of the Association for Computational Linguis- tics.
  42. Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. · arXiv
  43. Minjoon Seo, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2018. Phrase- indexed question answering: A new challenge for scalable document comprehension. In Proceedings of Empirical Methods in Natural Language Process- ing.
  44. Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. In Proceedings of the Association for Computational Linguistics.
  45. Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. 2022. Nearest neighbor zero-shot inference. In Proceedings of Empirical Methods in Natural Language Processing.
  46. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of Empirical Methods in Natural Lan- guage Processing.
  47. Zequn Sun, Wei Hu, and Chengkai Li. 2017. Cross- lingual entity alignment via joint attribute-preserving embedding. In Proceedings of the International Se- mantic Web Conference.
  48. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of Empirical Methods in Natural Lan- guage Processing.
  49. Frank F. Xu, Junxian He, Graham Neubig, and Vin- cent Josua Hellendoorn. 2022. Capturing structural locality in non-parametric language models. In Pro- ceedings of the International Conference on Learning Representations.
  50. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Conference of the North American Chapter of the Association for Computational Linguistics.
  51. Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. 2021. Efficient passage retrieval with hashing for open-domain question answering. In Proceedings of the Association for Computational Linguistics.
  52. Jinghui Yan, Jiajun Zhang, JinAn Xu, and Chengqing Zong. 2018. The impact of named entity translation for neural machine translation. In Proceedings of the China Workshop on Machine Translation.
  53. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2018. Breaking the softmax bot- tleneck: A high-rank rnn language model. In Pro- ceedings of the International Conference on Learning Representations.
  54. Dani Yogatama, Cyprien de Masson d'Autume, and Lingpeng Kong. 2021. Adaptive semiparametric lan- guage models. Proceedings of the Association for Computational Linguistics.
  55. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. · arXiv
  56. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Proceedings of Advances in Neural Information Processing Systems.
  57. Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Stein- hardt. 2022a. Describing differences between text distributions with natural language. In Proceedings of the International Conference of Machine Learning.

Cited by

Metadata

Authors
Min, S., Shi, W., Lewis, M., Chen, X., Yih, W.-T., Hajishirzi, H., & Zettlemoyer, L.
Venue
Findings of the Association for Computational Linguistics: ACL 2023
Publisher
Association for Computational Linguistics
Published
2023
Pages
2097-2118
DOI
10.18653/v1/2023.findings-acl.132
Cited by
16
References
24

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.