CoChat

Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter

Open access
Wu, Lijun, Tan, Xu, He, Di, Tian, Fei, Qin, Tao, Lai, Jianhuang, Liu, Tie-Yan · 2018 · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Cited by 54

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Abstract

Neural machine translation usually adopts autoregressive models and suffers from exposure bias as well as the consequent error propagation problem.Many previous works have discussed the relationship between error propagation and the accuracy drop (i.e., the left part of the translated sentence is often better than its right part in left-to-right decoding models) problem.In this paper, we conduct a series of analyses to deeply understand this problem and get several interesting findings.(1) The role of error propagation on accuracy drop is overstated in the literature, although it indeed contributes to the accuracy drop problem.(2) Characteristics of a language play a more important role in causing the accuracy drop: the left part of the translation result in a right-branching language (e.g., English) is more likely to be more accurate than its right part, while the right part is more accurate for a left-branching language (e.g., Japanese).Our discoveries are confirmed on different model structures including Transformer and RNN, and in other sequence generation tasks such as text summarization.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d18-1396

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Curriculum learning

    Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×

    “In this section, we study different branching languages from two aspects, including the n-gram statistics of a target language, which has been used as a kind of characterization of hardness of learning (Bengio et al., 2009), and the dependency statistics in parse trees.”
    Background
  • A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

    Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369 · cited 2×

    “Second, to further investigate the influence of error propagation on accuracy drop, we conduct a set of experiments with teacher forcing (Williams and Zipser, 1989) during inference, in which we feed the ground-truth preceding words to predict the next target word.”
    BackgroundMethod
  • Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

    Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621

  • Sequence Level Training with Recurrent Neural Networks

    Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426 · cited 4×

    “Shen et al. (2016); Ranzato et al. (2015) improve the basic maximum likelihood estimation (MLE) with reinforcement learning or minimum risk training and aim to address the limitation of MLE training and exposure bias problem.”
    BackgroundInfluential
  • A Closer Look at Memorization in Deep Networks

    Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307 · cited 1×

    “As previous work (Arpit et al., 2017) shows, neural networks are easy to learn and memorize simple patterns but difficult to make a correct prediction on noise examples.”
    Background

Cited by

See all 54

Metadata

Authors
Wu, L., Tan, X., He, D., Tian, F., Qin, T., Lai, J., & Liu, T.-Y.
Venue
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Published
2018
Pages
3602-3611
OpenAlex
W2889606145
Cited by
54
References
5
FWCI
4.58
Type
conference paper
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.