Graph Convolutional Encoders for Syntax-aware Neural Machine Translation
Open accessAbstract
We present a simple and effective approach to incorporating syntactic structure into neural attention-based encoderdecoder models for machine translation.We rely on graph-convolutional networks (GCNs), a recent class of neural networks developed for modeling graph-structured data.Our GCNs use predicted syntactic dependency trees of source sentences to produce representations of words (i.e.hidden states of the encoder) that are sensitive to their syntactic neighborhoods.GCNs take word representations as input and produce word representations as output, so they can easily be incorporated as layers into standard encoders (e.g., on top of bidirectional RNNs or convolutional neural networks).We evaluate their effectiveness with English-German and English-Czech translation experiments for different types of encoders and observe substantial improvements over their syntax-agnostic versions in all the considered setups.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · see citing works
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Neural Message Passing for Quantum ChemistryPreprint
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01212 · Cited by 5,414 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Advances In Deep Learning On Graphs (Gsp'18 Workshop)
Defferrard, M. (2016) · Figshare · doi:10.5281/zenodo.1286817 · Cited by 4,977 · see citing works
A Study of Translation Edit Rate with Targeted Human Annotation
Snover, M., Dorr, B. J., Schwartz, R. M., Micciulla, L., & Makhoul, J. I. (2006) · Cited by 2,380 · see citing works
Molecular graph convolutions: moving beyond fingerprints
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., & Riley, P. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9938-8 · Cited by 1,631 · see citing works
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1176 · Cited by 1,277 · see citing works
On the Properties of Neural Machine Translation: Encoder-Decoder ApproachesPreprint
Cho, K., van Merriënboer, B., Bahdanau, D., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.1259 · Cited by 1,112 · see citing works
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
Marcheggiani, D., & Titov, I. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1159 · Cited by 907 · see citing works
Self-citationMulti-task Sequence to Sequence LearningPreprint
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, Ł. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06114 · Cited by 606 · see citing works
Recurrent Neural Network Grammars
Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1024 · Cited by 517 · see citing works
The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies
Surdeanu, M., Johansson, R., Meyers, A., et al. (2008) · doi:10.3115/1596324.1596352 · Cited by 500 · see citing works
A Convolutional Encoder Model for Neural Machine Translation
Gehring, J., Auli, M., Grangier, D., & Dauphin, Y. (2016) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p17-1012 · Cited by 470 · see citing works
Neural Machine Translation in Linear Time
Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · see citing works
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · cited 1× · see citing works
“To not only summarize the past of a word, but also its future, a bidirectional RNN (Schuster and Paliwal, 1997; Irsoy and Cardie, 2014) is often used.”
MethodSyntax augmented machine translation via chart parsing
Zollmann, A., & Venugopal, A. (2006) · Workshop on Statistical Machine Translation · doi:10.3115/1654650.1654671 · Cited by 306 · see citing works
Tree-to-Sequence Attentional Neural Machine Translation
Eriguchi, A., Hashimoto, K., & Tsuruoka, Y. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1078 · Cited by 271 · see citing works
Modeling Relational Data with Graph Convolutional NetworksPreprint
Schlichtkrull, M., Kipf, T., Bloem, P., van den Berg, R., Titov, I., & Welling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06103 · Cited by 181 · see citing works
Self-citationTowards String-To-Tree Neural Machine Translation
Aharoni, R., & Goldberg, Y. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2021 · Cited by 147 · see citing works
Learning to Compose Words into Sentences with Reinforcement LearningPreprint
Yogatama, D., Blunsom, P., Dyer, C., Grefenstette, E., & Wang, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.09100 · Cited by 139 · see citing works
Learning to Translate with Source and Target Syntax
Chiang, D. (2010) · Cited by 109 · see citing works
Smith, D. A., & Eisner, J. M. (2006) · Workshop on Statistical Machine Translation · doi:10.3115/1654650.1654655 · Cited by 90 · see citing works
Syntactically Guided Neural Machine Translation
Stahlberg, F., Hasler, E., Waite, A., et al. (2016) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p16-2049 · Cited by 72 · see citing works
Fitting Sentence Level Translation Evaluation with Many Dense Features
Stanojević, M., & Sima’an, K. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1025 · Cited by 56 · see citing works
Self-citationNeural Machine Translation with Source-Side Latent Graph Parsing
Hashimoto, K., & Tsuruoka, Y. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1012 · Cited by 53 · see citing works
A Convolutional Encoder Model for Neural Machine TranslationPreprint
Gehring, J., Auli, M., Grangier, D., & Dauphin, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02344 · Cited by 52 · see citing works
Neural Monkey: An Open-source Tool for Sequence Learning
Helcl, J., & Libovický, J. (2017) · The Prague Bulletin of Mathematical Linguistics · doi:10.1515/pralin-2017-0001 · Cited by 47 · see citing works
Learning to Parse and Translate Improves Neural Machine TranslationPreprint
Eriguchi, A., Tsuruoka, Y., & Cho, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03525 · Cited by 20 · see citing works
Syntax-aware Neural Machine Translation Using CCG.Preprint
Nădejde, M., Reddy, S., Sennrich, R., et al. (2017) · arXiv (Cornell University) · Cited by 13 · see citing works
Evaluating MT systems with BEER
Stanojević, M., & Sima’an, K. (2015) · The Prague Bulletin of Mathematical Linguistics · doi:10.1515/pralin-2015-0010 · Cited by 6 · see citing works
Self-citationNeural Machine Translation with Source-Side Latent Graph ParsingPreprint
Hashimoto, K., & Tsuruoka, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02265 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 10
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Represen- tations (ICLR).
- Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2012. Abstract meaning representation (amr) 1.0 specification. In Conference on Empiri- cal Methods in Natural Language Processing, pages 1533-1544.
- Michaël Defferrard, Xavier Bresson, and Pierre Van- dergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Sys- tems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3837-3845.
- David K Duvenaud, Dougal Maclaurin, Jorge Ipar- raguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. 2015. Convo- lutional networks on graphs for learning molecular fingerprints. In Advances in neural information pro- cessing systems, pages 2224-2232.
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax- sensitive dependencies. Transactions of the Associ- ation for Computational Linguistics, 4:521-535.
- Rico Sennrich and Barry Haddow. 2016. Linguistic In- put Features Improve Neural Machine Translation. In Proceedings of the First Conference on Machine Translation (WMT16), volume abs/1606.02892. · arXiv
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation sys- tems for wmt 16. In Proceedings of the First Conference on Machine Translation, pages 371- 376, Berlin, Germany. Association for Computa- tional Linguistics.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguistics.
- Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1526- 1534, Austin, Texas. Association for Computational Linguistics.
- Andreas Zollmann and Ashish Venugopal. 2006. Syn- tax augmented machine translation via chart pars- ing. In Proceedings of the Workshop on Statistical Machine Translation, StatMT '06, pages 138-141, Stroudsburg, PA, USA. Association for Computa- tional Linguistics.
Cited by
Sorted by citations
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works
Graph neural networks: A review of methods and applications
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020) · AI Open · doi:10.1016/j.aiopen.2021.01.001 · Cited by 6,880 · see citing works
- Authors
- Bastings, J., Titov, I., Aziz, W., Marcheggiani, D., & Simaan, K.
- Affiliations
- University of Amsterdam (NL)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 1957-1967
- OpenAlex
- W2612881151
- Cited by
- 97
- References
- 41
- Citations by year
- 2025: 3 · 2024: 10 · 2023: 6 · 2022: 6 · 2021: 24 · 2020: 25 · 2019: 14 · 2018: 9
- FWCI
- 5.86
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (277-89-002, 639.022.518)
- H2020 European Research Council (678254)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- graph convolutional networks, neural machine translation, syntactic dependency trees, English-German translation, English-Czech translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
