CoChat

Semantic Structure in Deep Learning

Open access
Pavlick, Ellie · 2022 · Annual Review of Linguistics
Cited by 66

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

Abstract

Deep learning has recently come to dominate computational linguistics, leading to claims of human-level performance in a range of language processing tasks. Like much previous computational work, deep learning–based linguistic representations adhere to the distributional meaning-in-use hypothesis, deriving semantic representations from word co-occurrence statistics. However, current deep learning methods entail fundamentally new models of lexical and compositional meaning that are ripe for theoretical analysis. Whereas traditional distributional semantics models take a bottom-up approach in which sentence meaning is characterized by explicit composition functions applied to word meanings, new approaches take a top-down approach in which sentence representations are treated as primary and representations of words and syntax are viewed as emergent. This article summarizes our current understanding of how well such representations capture lexical semantics, world knowledge, and composition. The goal is to foster increased collaboration on testing the implications of such representations as general-purpose models of semantics.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1146/annurev-linguistics-031120-122924

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 37 references

Research works printed but not linked: 103

  1. Adi Y, Kermany E, Belinkov Y, Lavi O, Goldberg Y. 2017.Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. Paper presented at the International Conference on Learning Representations (ICLR 2017), Toulon, Fr., Apr. 24–26
  2. Adiwardana D, Luong MT, So DR, Hall J, Fiedel N, et al. 2020. Towards a human-like open-domain chatbot. arXiv:2001.09977 [cs.CL] · arXiv
  3. Baroni M, 2014, Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1, 238
  4. Baroni M, 2010, Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1183
  5. 10.1162/tacl_a_00254 · DOI · Paper page
  6. Bengio Y, 2003, J. Mach. Learn. Res., 3, 1137
  7. Berant J, 2011, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 610
  8. 10.1146/annurev-linguistics-011619-030303 · DOI · Paper page
  9. 10.1162/COLI_a_00261 · DOI · Paper page
  10. Bowman SR, Dahl GE. 2021. What will it take to fix benchmarking in natural language understanding? arXiv:2104.02145 [cs.CL] · DOI · Paper page
  11. 10.18653/v1/2021.acl-long.146 · DOI · Paper page
  12. Carlson A, 2010, Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, 1306
  13. 10.18653/v1/2020.conll-1.17 · DOI
  14. 10.18653/v1/2021.acl-long.258 · DOI · Paper page
  15. 10.18653/v1/P18-1198 · DOI · Paper page
  16. Cooper R, 1996, Using the framework
  17. 10.1007/11736790_9 · DOI · Paper page
  18. Dasgupta I, Guo D, Stuhlmüller A, Gershman SJ, Goodman ND. 2018. Evaluating compositionality in sentence embeddings. arXiv:1802.04302 [cs.CL] · arXiv
  19. De Marneffe MC, 2019, Proceedings of Sinn und Bedeutung 23, 107
  20. Dhillon PS, 2011, NIPS'11: Proceedings of the 24th International Conference on Neural Information Processing Systems, 199
  21. Dumais ST, 1988, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 281, 10.1145/57167.57214 · DOI · Paper page
  22. 10.1207/s15516709cog1402_1 · DOI · Paper page
  23. 10.18653/v1/2020.acl-main.663 · DOI
  24. Erk K., 2007, Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 216
  25. 10.1002/lnco.362 · DOI · Paper page
  26. Ettinger A, 2018, Proceedings of the 27th International Conference on Computational Linguistics, 1790
  27. 10.18653/v1/W16-2524 · DOI · Paper page
  28. 10.1145/1409360.1409378 · DOI · Paper page
  29. 10.1007/978-90-481-8847-5_10 · DOI · Paper page
  30. Firth JR., 1957, Studies in Linguistic Analysis, 1
  31. Forbes M, Holtzman A, Choi Y. 2019. Do neural language representations learn physical commonsense? arXiv:1908.02899 [cs.CL] · arXiv
  32. Frege G., 1884, Die Grundlagen der Arithmetik: Eine logisch mathematische Untersuchung über den Begriff der Zahl
  33. 10.18653/v1/N19-1004 · DOI · Paper page
  34. 10.18653/v1/2021.acl-long.212 · DOI
  35. 10.18653/v1/2020.acl-main.365 · DOI
  36. 10.18653/v1/2020.acl-main.177 · DOI
  37. Grefenstette E, Sadrzadeh M. 2011. Experimenting with transitive verbs in a DisCoCat. arXiv:1107.3119 [cs.CL] · arXiv
  38. 10.1098/rstb.2007.2089 · DOI · Paper page
  39. 10.18653/v1/E17-1006 · DOI
  40. Hewitt J, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4129
  41. 10.1162/tacl_a_00250 · DOI
  42. Hupkes D, Singh A, Korrel K, Kruszewski G, Bruni E. 2018a. Learning compositionally through attentive guidance. arXiv:1805.09657 [cs.CL] · arXiv
  43. 10.1613/jair.1.11196 · DOI · Paper page
  44. Hwang JD, Bhagavatula C, Bras RL, Da J, Sakaguchi K, et al. 2020. COMET-ATOMIC 2020: on symbolic and neural commonsense knowledge graphs. arXiv:2010.05953 [cs.CL] · DOI · Paper page
  45. 10.18653/v1/2020.blackboxnlp-1.18 · DOI
  46. 10.18653/v1/W18-5424 · DOI · Paper page
  47. Jurafsky D, 2009, Speech and Language Processing
  48. Kann K, 2019, Proceedings of the Society for Computation in Linguistics (SCiL) 2019, 287
  49. 10.18653/v1/2020.acl-main.698 · DOI · Paper page
  50. 10.18653/v1/S19-1026 · DOI · Paper page
  51. Lake B, 2018, Proceedings of the 35th International Conference on Machine Learning (ICML 2018), 2873
  52. 10.1162/089120103322145324 · DOI
  53. Lenci A., 2008, Ital. J. Linguist., 20, 1
  54. 10.1146/annurev-linguistics-030514-125254 · DOI · Paper page
  55. 10.18653/v1/2020.emnlp-main.557 · DOI · Paper page
  56. 10.1162/tacl_a_00115 · DOI · Paper page
  57. 10.1023/B:BTTJ.0000047600.45421.6d · DOI · Paper page
  58. 10.18653/v1/N19-1112 · DOI · Paper page
  59. Lovering C, 2021, Predicting inductive biases of pre-trained models
  60. Lund K, 1995, Proceedings of the 17th Annual Conference of the Cognitive Science Society, 660
  61. McKenna N, 2020, Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, 137
  62. 10.18653/v1/2020.blackboxnlp-1.4 · DOI · Paper page
  63. Mickus T, 2020, Proceedings of the Society for Computation in Linguistics 2020, 279
  64. Mikolov T, Chen K, Corrado G, Dean J. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781 [cs.CL] · arXiv
  65. 10.1111/j.1551-6709.2010.01106.x · DOI · Paper page
  66. 10.18653/v1/2020.blackboxnlp-1.7 · DOI
  67. Nair S, 2020, Proceedings of the Workshop on the Cognitive Aspects of the Lexicon, 129
  68. 10.18653/v1/W19-2004 · DOI
  69. 10.18653/v1/W16-2504 · DOI
  70. Partee B, 1995, Invitation to Cognitive Science, 1, 311
  71. 10.18653/v1/P16-1204 · DOI
  72. 10.3115/v1/D14-1162 · DOI · Paper page
  73. 10.18653/v1/D18-1007 · DOI · Paper page
  74. 10.18653/v1/S18-2023 · DOI · Paper page
  75. Potts C., 2020, Medium
  76. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, et al. 2021. Learning transferable visual models from natural language supervision. arXiv:2103.00020 [cs.CV] · arXiv
  77. Radford A, 2020, OpenAI Blog
  78. Radford A, 2019, Language models are unsupervised multitask learners
  79. 10.18653/v1/D16-1264 · DOI · Paper page
  80. Ravichander A, 2020, Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, 88
  81. Reif E, 2019, Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
  82. 10.18653/v1/P18-1079 · DOI · Paper page
  83. 10.18653/v1/D19-1228 · DOI
  84. Saxton D, 2019, Analysing mathematical reasoning abilities of neural models
  85. 10.1162/tacl_a_00277 · DOI
  86. 10.18653/v1/2021.acl-long.569 · DOI · Paper page
  87. 10.1109/ICCV.2019.00756 · DOI · Paper page
  88. Szabó ZG., 2020, Stanford Encyclopedia of Philosophy
  89. Tenney I, 2019, What do you learn from context? Probing for sentence structure in contextualized word representations
  90. 10.18653/v1/2020.blackboxnlp-1.25 · DOI
  91. 10.1613/jair.2934 · DOI · Paper page
  92. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, et al. 2017. Attention is all you need. arXiv:1706.03762 [cs.CL] · arXiv
  93. Veldhoen S, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches (CoCo 2016)
  94. 10.18653/v1/2020.emnlp-main.586 · DOI · Paper page
  95. 10.18653/v1/D19-1221 · DOI · Paper page
  96. 10.18653/v1/P19-1439 · DOI · Paper page
  97. Wang A, 2019, Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
  98. 10.18653/v1/D19-1286 · DOI · Paper page
  99. 10.18653/v1/2020.emnlp-main.16 · DOI · Paper page
  100. White AS, 2018, Proceedings of the 48th Annual Meeting of the North East Linguistic Society (NELS 48), 221
  101. 10.18653/v1/W19-4804 · DOI
  102. 10.18653/v1/2020.blackboxnlp-1.15 · DOI · Paper page
  103. 10.18653/v1/2021.findings-acl.201 · DOI

Cited by

See all 66

Metadata

Authors
Pavlick, E.
Venue
Annual Review of Linguistics
Published
2022-01-14
Volume
8
Issue
1
Pages
447-471
ISSN
  • 2333-9683 (ISSN-L)
  • 2333-9691
OpenAlex
W3215599159
Cited by
66
References
37
FWCI
0.78
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Semantic Structure in Deep Learning. https://app.cochat.ai/papers/10.1146/annurev-linguistics-031120-122924
The paper
Pavlick, E. (2022). Semantic Structure in Deep Learning. Annual Review of Linguistics. https://doi.org/10.1146/annurev-linguistics-031120-122924

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.