Relational memory-augmented language models
Open accessAbstract
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model and a knowledge graph for more coherent and logical generation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
(2014) · Cited by 33,419 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
Advances in Neural Information Processing Systems 14
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 9,184 · see citing works
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251 · see citing works
Modeling Relational Data with Graph Convolutional Networks
Schlichtkrull, M., Kipf, T. N., Bloem, P., van den Berg, R., Titov, I., & Welling, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93417-4_38 · Cited by 5,570 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1× · see citing works
“While we can also use prompts as intervention for vanilla language models, it remains challenging about selecting the appropriate prompts in different applications (Liu et al., 2021a).”
MethodDense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,646 · see citing works
Open Information Extraction from the Web
Etzioni, O., Banko, M., Cafarella, M. J., Soderland, S., & Weld, D. S. (2007) · CACM · doi:10.1145/1409360.1409378 · Cited by 2,620 · see citing works
A survey of named entity recognition and classification
Nadeau, D., & Sekine, S. (2007) · Lingvisticae Investigationes · doi:10.1075/li.30.1.03nad · Cited by 2,507 · see citing works
Complex Embeddings for Simple Link Prediction
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., & Bouchard, G. (2016) · arXiv · doi:10.48550/arxiv.1606.06357 · Cited by 2,100 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
Collaborative Knowledge Base Embedding for Recommender Systems
Zhang, F., Yuan, N. J., Lian, D., Xie, X., & Ma, W. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939673 · Cited by 1,820 · see citing works
Design Challenges and Misconceptions in Named Entity Recognition
Ratinov, L., & Roth, D. (2009) · Conference on Computational Natural Language Learning · doi:10.3115/1596374.1596399 · Cited by 1,640 · see citing works
Reading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
Wang, H., Zhang, F., Xie, X., & Guo, M. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186175 · Cited by 1,034 · see citing works
ERNIE: Enhanced Representation through Knowledge IntegrationPreprint
Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., & Wu, H. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09223 · Cited by 983 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 1× · see citing works
“We note that question answering is also often used to evaluate coherence (Guu et al., 2020; Lin et al., 2021).”
MethodTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899 · see citing works
K-BERT: Enabling Language Representation with Knowledge Graph
Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5681 · Cited by 797 · cited 1× · see citing works
“Using knowledge information for pretraining language models (Peters et al., 2019; Sun et al., 2019; Liu et al., 2020; Guu et al., 2020; Wang et al., 2021b; Agarwal et al., 2021; Verga et al., 2021) has recently grown in popularity and has achieved substantial improvements on knowledge-driven tasks such as question ans…”
BackgroundLeveraging Linguistic Structure For Open Domain Information Extraction
Angeli, G., Johnson Premkumar, M. J., & Manning, C. D. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1034 · Cited by 725 · see citing works
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00360 · Cited by 676 · see citing works
Knowledge Enhanced Contextual Word Representations
Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1005 · Cited by 661 · see citing works
KG-BERT: BERT for Knowledge Graph CompletionPreprint
Yao, L., Mao, C., & Luo, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03193 · Cited by 638 · see citing works
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.45 · Cited by 620 · see citing works
Knowledge Graph Embedding Based Question Answering
Huang, X. X., Zhang, J., Li, D., & Li, P. (2019) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3289600.3290956 · Cited by 598 · see citing works
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574 · see citing works
Findings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa‐jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5301 · Cited by 461 · see citing works
Improving Neural Language Models with a Continuous CachePreprint
Grave, E., Joulin, A., & Usunier, N. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1612.04426 · Cited by 316 · see citing works
Tying Word Vectors and Word Classifiers: A Loss Framework for Language ModelingPreprint
Inan, H., Khosravi, K., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01462 · Cited by 314 · see citing works
OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs
Moon, S., Shah, P., Kumar, A., & Subba, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1081 · Cited by 306 · see citing works
YAGO: A Multilingual Knowledge Base from Wikipedia, Wordnet, and Geonames
Rebele, T., Suchanek, F. M., Hoffart, J., Biega, J., Kuzey, E., & Weikum, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46547-0_19 · Cited by 285 · see citing works
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285 · see citing works
Globally Coherent Text Generation with Neural Checklist Models
Kiddon, C., Zettlemoyer, L., & Choi, Y. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1032 · Cited by 219 · see citing works
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
Logan, R., Liu, N. F., Peters, M. E., Gardner, M., & Singh, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1598 · Cited by 168 · see citing works
Nearest Neighbor Machine Translation
Khandelwal, U., Fan, A., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2010.00710 · Cited by 165 · see citing works
Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
Agarwal, O., Ge, H., Shakeri, S., & Al-Rfou, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.278 · Cited by 161 · see citing works
Barzilay, R., & Lapata, M. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219858 · Cited by 136 · see citing works
Word meaning in minds and machines.
Lake, B. M., & Murphy, G. L. (2023) · Psychological Review · doi:10.1037/rev0000297 · Cited by 131 · see citing works
Neural Text Generation in Stories Using Entity Representations as Context
Clark, E., Ji, Y., & Smith, N. A. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1204 · Cited by 115 · see citing works
A Neural Knowledge Language ModelPreprint
Ahn, S., Choi, H., Pärnamaa, T., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.00318 · Cited by 103 · see citing works
Episodic Memory in Lifelong Language Learning
d'Autume, C. D. M., Ruder, S., Kong, L., & Yogatama, D. (2019) · arXiv · doi:10.48550/arxiv.1906.01076 · Cited by 95 · see citing works
Self-citationLanguage Models are Open Knowledge Graphs
Wang, C., Liu, X., & Song, D. (2020) · arXiv · doi:10.48550/arxiv.2010.11967 · Cited by 79 · see citing works
Differentiable Reasoning on Large Knowledge Bases and Natural Language
Minervini, P., Bošnjak, M., Rocktäschel, T., Riedel, S., & Grefenstette, E. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5962 · Cited by 71 · see citing works
Reference-Aware Language Models
Yang, Z., Blunsom, P., Dyer, C., & Wang, L. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1197 · Cited by 68 · see citing works
Self-citationLearning to Compute Word Embeddings On the FlyPreprint
Bahdanau, D., Bosc, T., Jastrzȩbski, S., Grefenstette, E., Vincent, P., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00286 · Cited by 62 · see citing works
Augmenting Transformers with KNN-Based Composite Memory for Dialog
Fan, A., Gardent, C., Braud, C., & Bordes, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00356 · Cited by 58 · see citing works
Adaptive Semiparametric Language Models
Yogatama, D., d’Autume, C. D. M., & Kong, L. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00371 · Cited by 58 · see citing works
Self-citationAdaptable and Interpretable Neural MemoryOver Symbolic Knowledge
Verga, P., Sun, H., Soares, L. B., & Cohen, W. W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.288 · Cited by 47 · see citing works
Wang, Q., Pan, X., Huang, L., Zhang, B., Jiang, Z., Ji, H., & Knight, K. K. (2018) · International Conference on Natural Language Generation · doi:10.18653/v1/w18-6502 · Cited by 46 · see citing works
Latent Relation Language Models
Hayashi, H., Hu, Z., Xiong, C., & Neubig, G. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6298 · Cited by 44 · see citing works
Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented DialogPreprint
Thulke, D., Daheim, N., Dugast, C., & Ney, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04643 · Cited by 41 · see citing works
Nye, M., Tessler, M. H., Tenenbaum, J. B., & Lake, B. M. (2021) · arXiv · doi:10.48550/arxiv.2107.02794 · Cited by 40 · see citing works
Building Language Models for Text with Named Entities
Parvez, M. R., Chakraborty, S., Ray, B., & Chang, K.-W. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1221 · Cited by 36 · see citing works
Dynamic Evaluation of Transformer Language Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2019) · arXiv · doi:10.48550/arxiv.1904.08378 · Cited by 30 · see citing works
Leveraging Knowledge Bases in LSTMs for Improving Machine ReadingPreprint
Yang, B., & Mitchell, T. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09091 · Cited by 28 · see citing works
Pretraining the Noisy Channel Model for Task-Oriented Dialogue
Liu, Q., Yu, L., Rimell, L., & Blunsom, P. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00390 · Cited by 22 · see citing works
Self-citationSchneider, N., Flanigan, J., & O’Gorman, T. (2015) · Cited by 20 · see citing works
WikiGraphs: A Wikipedia Text - Knowledge Graph Paired Dataset
Wang, L., Li, Y., Aslan, Ö., et al. (2021) · doi:10.18653/v1/2021.textgraphs-1.7 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 23
- K. M. Annervaz, Somnath Basu Roy Chowdhury, and Ambedkar Dukkipati. 2018. Learning be- yond datasets: Knowledge graph augmented neural networks for natural language process- ing. In Proceedings of the 2018 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 313-322. Asso- ciation for Computational Linguistics.
- Ankur Bapna and Orhan Firat. 2019. Non- parametric adaptation for neural machine trans- lation. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 1921-1931. Association for Computational Linguistics.
- Kurt D. Bollacker, Robert P. Cook, and Patrick Tufts. 2007. Freebase: A shared database of structured general human knowledge. In Pro- ceedings of the Twenty-Second AAAI Con- ference on Artificial Intelligence, July 22-26, 2007, Vancouver, British Columbia, Canada, pages 1962-1963. AAAI Press.
- Antoine Bordes, Nicolas Usunier, Alberto García- Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural In- formation Processing Systems 26: 27th Annual Conference on Neural Information Process- ing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2787-2795.
- Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In 9th International Con- ference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Conference of the As- sociation for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 2978-2988. Association for Computational Linguistics.
- Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge- powered conversational agents. In 7th Interna- tional Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
- Édouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017. Ef- ficient softmax approximation for GPUs. In Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1302-1310. PMLR.
- Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2018. Dialog-to-action: Conver- sational question answering over a large-scale knowledge base. In Advances in Neural Infor- mation Processing Systems 31: Annual Confer- ence on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2946-2955.
- Frederick Jelinek. 1980. Interpolated estimation of markov source parameters from sparse data. In Proceedings of Workshop on Pattern Rec- ognition in Practice, 1980.
- Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A. Smith. 2017. Dynamic entity representations in neural language mod- els. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1830-1839. Asso- ciation for Computational Linguistics.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Repre- sentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
- Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval- augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
- Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mix- ture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
- Malte Ostendorff, Peter Bourgonje, Maria Berger, Julián Moreno Schneider, Georg Rehm, and Bela Gipp. 2019. Enriching BERT with knowl- edge graph embeddings for document classifi- cation. In Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019, Erlangen, Germany, October 9-11, 2019.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Juan Ramos. 2003. Using tf-idf to determine word relevance in document queries. In Proceed- ings of the First Instructional Conference on Machine Learning, 242, pages 29-48. Citeseer.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Process- ing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
- Hongwei Wang, Fuzheng Zhang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2019. Multi-task feature learning for knowledge graph enhanced recommendation. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 2000-2010. ACM.
- Muhan Zhang and Yixin Chen. 2018. Link pre- diction based on graph neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural In- formation Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5171-5181.
- Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Ap- plications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 6069-6076. AAAI Press.
- Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chatbots with deep attention matching network. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1118-1127. Association for Computational Linguistics.
Cited by
Sorted by citations
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
- Authors
- Liu, Q., Yogatama, D., & Blunsom, P.
- Affiliations
- University of Oxford (GB)
- Google DeepMind (United Kingdom) (GB)
- Corresponding author
- Liu, Qi
- Yogatama, Dani
- Blunsom, Phil
- ORCID
- Liu, Qi
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2022
- Volume
- 10
- Pages
- 555-572
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4226365369
- Cited by
- 23
- References
- 67
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 6 · 2023: 5 · 2022: 3
- FWCI
- 1.48
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- DeepMind
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- knowledge graphs, relational memory, autoregressive language models, text generation, perplexity, coherence, causal intervention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
