CoChat

Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement

Open access
Lee, Jason, Mansimov, Elman, Cho, Kyunghyun · 2018 · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

The DOI matches its Crossref record; 41 of the 46 references it cites were found in a registry or an index.

Abstract

We propose a conditional non-autoregressive neural sequence model based on iterative refinement.The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task.We extensively evaluate the proposed model on machine translation (En$De and En$Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d18-1149

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. Andrej Karpathy and Fei-Fei Li. 2015. Deep visualsemantic alignments for generating image descriptions. In CVPR .
  2. Yoon Kim and Alexander M Rush. 2016. Sequencelevel knowledge distillation. arXiv preprint arXiv:1606.07947 . · arXiv
  3. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL .

Cited by

  • Going deeper with Image Transformers

    Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259

    “Note, transforming only the class token with frozen trunk embeddings in CaiT is reminiscent of nonautoregressive encoder-decoders [25, 41], where a whole sequence (we have only one prediction) is produced at once by iterative refinements.”
    Background
  • Mask-Predict: Parallel Decoding of Conditional Masked Language Models

    Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1633 · Cited by 464

    “Mask-predict repeatedly masks out and re-predicts the subset of words in the current translation that the model is least confident about, in contrast to recent parallel decoding translation approaches that repeatedly predict the entire sequence (Lee et al., 2018).”
    BackgroundMethodResultInfluential
  • WaveGrad: Estimating Gradients for Waveform Generation

    Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., & Chan, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00713 · Cited by 44

    “Edit-based models can also iteratively refine the outputs during inference (Lee et al., 2018; Ghazvininejad et al., 2019; Chan et al., 2020), while they do not rely on a (continuous) gradient-based sampler, they rely on a (discrete) edit-based sampler.”
    BackgroundMethodInfluential
  • Challenges in building intelligent open-domain dialog systems

    Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288

    “onditioned on the words that are previously generated. To make the decoding parallelizable, non-autoregressive models based on Transformer have been proposed to generate all the tokens simultaneously [55, 59]. Nonautoregressive modeling factorizes the distribution over a target sequence given a query into a product of…”
    Method
  • ClariNet: Parallel Wave Generation in End-to-End Text-to-SpeechPreprint

    Ping, W., Peng, K., & Chen, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07281 · Cited by 351

    “Similar techniques are applied in non-autoregressive models for machine translation (Gu et al., 2018; Kaiser et al., 2018; Lee et al., 2018; Roy et al., 2018).”
    Method

See all 386

Metadata

Authors
Lee, J., Mansimov, E., & Cho, K.
Affiliations
New York University (US)
Venue
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Published
2018
Pages
1173-1182
OpenAlex
W2963434219
Cited by
386
References
46
Citations by year
2026: 10 · 2025: 18 · 2024: 24 · 2023: 55 · 2022: 57 · 2021: 88 · 2020: 86 · 2019: 43 · 2018: 5
FWCI
22.08
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • Samsung Advanced Institute of Technology
  • Tencent
  • Samsung
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
iterative refinement, deterministic decoding, latent variable models, denoising autoencoder, machine translation, image captioning, decoding speed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.