Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Open accessThe DOI matches its Crossref record; 41 of the 46 references it cites were found in a registry or an index.
Abstract
We propose a conditional non-autoregressive neural sequence model based on iterative refinement.The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task.We extensively evaluate the proposed model on machine translation (En$De and En$Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“It reduces memory costs by avoiding auxiliary models but relies on fixed-size drafting blocks. Verification is strict, and confidence awareness is minimal or implicit.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While recent advances in speculative decoding and non-autoregressive generation have made substantial strides toward accelerating sequence generation, most existing approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5×
“Each image is pre-transformed into a set of 49 512-dimensional feature vectors, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al.”
MethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“Training and Inference We use Adam (Kingma and Ba, 2014) and use L = 3 in Eq.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We extract 49 512-dimensional feature vectors for each image, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009).”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 6×
“…compare the proposed non-autoregressive model against the autoregressive counterpart both in terms of generation quality, measured in terms of BLEU (Papineni et al., 2002), and generation efficiency, measured in terms of (source) tokens and images per second for translation and image captioning,…”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodSelf-citationDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2×
“…Gu et al. (2017), in the context of machine translation, and Oord et al. (2017), in the context of speech generation, have recently discovered that it is important to use knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016) to successfully train a non-autoregressive sequence model.”
Background
Research works printed but not linked: 3
- Andrej Karpathy and Fei-Fei Li. 2015. Deep visualsemantic alignments for generating image descriptions. In CVPR .
- Yoon Kim and Alexander M Rush. 2016. Sequencelevel knowledge distillation. arXiv preprint arXiv:1606.07947 . · arXiv
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL .
Cited by
Going deeper with Image Transformers
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259
“Note, transforming only the class token with frozen trunk embeddings in CaiT is reminiscent of nonautoregressive encoder-decoders [25, 41], where a whole sequence (we have only one prediction) is produced at once by iterative refinements.”
BackgroundMask-Predict: Parallel Decoding of Conditional Masked Language Models
Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1633 · Cited by 464
“Mask-predict repeatedly masks out and re-predicts the subset of words in the current translation that the model is least confident about, in contrast to recent parallel decoding translation approaches that repeatedly predict the entire sequence (Lee et al., 2018).”
BackgroundMethodResultInfluentialWaveGrad: Estimating Gradients for Waveform Generation
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., & Chan, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00713 · Cited by 44
“Edit-based models can also iteratively refine the outputs during inference (Lee et al., 2018; Ghazvininejad et al., 2019; Chan et al., 2020), while they do not rely on a (continuous) gradient-based sampler, they rely on a (discrete) edit-based sampler.”
BackgroundMethodInfluentialChallenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288
“onditioned on the words that are previously generated. To make the decoding parallelizable, non-autoregressive models based on Transformer have been proposed to generate all the tokens simultaneously [55, 59]. Nonautoregressive modeling factorizes the distribution over a target sequence given a query into a product of…”
MethodClariNet: Parallel Wave Generation in End-to-End Text-to-SpeechPreprint
Ping, W., Peng, K., & Chen, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07281 · Cited by 349
“Similar techniques are applied in non-autoregressive models for machine translation (Gu et al., 2018; Kaiser et al., 2018; Lee et al., 2018; Roy et al., 2018).”
MethodJoint Entity and Relation Extraction With Set Prediction Networks
Sui, D., Zeng, X., Chen, Y., Liu, K., & Zhao, J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264735 · Cited by 214
“Due to the autoregressive decoder featured by lower latency during inference, there is rich literature devoted to the topic, such as [34], [35], [36], [37], and [38].”
BackgroundGu, J., Wang, C., & Zhao, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11006 · Cited by 176
“We follow closely a standard pipeline established by Lee et al. (2018). Table 2b shows this comparison.”
BackgroundMethodResultInfluentialContextualized Perturbation for Textual Adversarial Attack
Li, D., Zhang, Y., Peng, H., Chen, L., Brockett, C., Sun, M.-T., & Dolan, B. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.400 · Cited by 178
Parallel Iterative Edit Models for Local Sequence Transduction
Awasthi, A., Sarawagi, S., Goyal, R., Ghosh, S., & Piratla, V. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1435 · Cited by 190
“We create special compounds edits that merge token inserts with preceding edits that yield higher accuracy than earlier methods of independently predicting inserts (Ribeiro et al., 2018; Lee et al., 2018).”
BackgroundMethodInfluentialFlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
Ma, X., Zhou, C., Li, X., Neubig, G., & Hovy, E. H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1437 · Cited by 188
“We use the data provided in Lee et al. (2018)”
MethodNeural machine translation: A review of methods, resources, and tools
Tan, Z., Wang, S., Yang, Z., Chen, G., Huang, X., Sun, M., & Liu, Y. (2020) · AI Open · doi:10.1016/j.aiopen.2020.11.001 · Cited by 140
Non-Autoregressive Machine Translation with Auxiliary Regularization
Wang, Y., Tian, F., He, D., Qin, T., Zhai, C., & Liu, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015377 · Cited by 138
Semantic-Conditional Diffusion Networks for Image Captioning*
Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02237 · Cited by 182
Understanding Knowledge Distillation in Non-autoregressive Machine TranslationPreprint
Zhou, C., Neubig, G., & Gu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02727 · Cited by 191
Non-Autoregressive Transformer for Speech Recognition
Chen, N., Watanabe, S., Villalba, J., Żelasko, P., & Dehak, N. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3044547 · Cited by 162
“To decide the length, previous studies predicted either the length of each word [8] or the length of the output sequence [9].”
BackgroundResultEnd-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification
Libovický, J., & Helcl, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1336 · Cited by 124
“Lee et al. (2018) propose an architecture with two decoders.”
MethodMask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict
Higuchi, Y., Watanabe, S., Chen, N., Ogawa, T., & Kobayashi, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2404 · Cited by 173
Gao, Z., Zhang, S., McLoughlin, I. V., & Yan, Z. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-9996 · Cited by 248
Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
Guo, J., Tan, X., He, D., Qin, T., Xu, L., & Liu, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013723 · Cited by 132
Non-Autoregressive Machine Translation with Latent Alignments
Saharia, C., Chan, W., Saxena, S., & Norouzi, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.83 · Cited by 106
Glancing Transformer for Non-Autoregressive Neural Machine Translation
Qian, L., Zhou, H., Bao, Y., Wang, M., Qiu, L., Zhang, W., Yu, Y., & Li, L. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.155 · Cited by 108
“As for IWSLT16 DE-EN, we adopt linear annealing (from 3 e − 4 to 1 e − 5 ) as in Lee et al. (2018).”
MethodInfluentialShu, R., Lee, J., Nakayama, H., & Cho, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6413 · Cited by 141
Self-citationText Embeddings Reveal (Almost) As Much As Text
Morris, J., Kuleshov, V., Shmatikov, V., & Rush, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.765 · Cited by 90
Single Document Summarization as Tree Induction
Liu, Y., Titov, I., & Lapata, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1173 · Cited by 114
“The idea of structure refinement is conceptually related to recently proposed models for solving iterative inference problems (Marino et al., 2018; Putzky and Welling, 2017; Lee et al., 2018).”
BackgroundA Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond
Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., & Liu, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3277122 · Cited by 179
“To release the inherent uncertainty of the data itself, length parallel decoding (LPD) [23], [29] and noise parallel decoding”
BackgroundMethodInfluentialMulti-modal Text Recognition Networks: Interactive Enhancements Between Visual and Semantic Features
Na, B., Kim, Y., & Park, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_26 · Cited by 139
Distilling Translations with Visual Awareness
Ive, J., Madhyastha, P., & Specia, L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1653 · Cited by 98
“Iterative refinement approaches have also been proposed that start with a draft translation and then predict discrete substitutions based on an attention mechanism (Novak et al., 2016), or using non-autoregressive methods with a focus on speeding up decoding (Lee et al., 2018).”
MethodDeep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine TranslationPreprint
Kasai, J., Pappas, N., Peng, H., Cross, J., & Smith, N. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10369 · Cited by 193
“We follow the preprocessing and data splits of previous work (EN-DE: Vaswani et al., 2017; EN-RO: Lee et al., 2018; EN-ZH: Hassan et al., 2018; Wu et al., 2019; EN-FR: Gehring et al., 2017).”
MethodFully Non-autoregressive Neural Machine Translation: Tricks of the Trade
Gu, J., & Kong, X. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.11 · Cited by 80
NAOMI: Non-Autoregressive Multiresolution Sequence ImputationPreprint
Liu, Y., Yu, R., Zheng, S., Zhan, E., & Yue, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10946 · Cited by 136
“Similarly, [17, 18] propose a fully deterministic model to reduce the amount of supervision.”
BackgroundMethodInfluentialInsertion-based Decoding with Automatically Inferred Generation Order
Gu, J., Liu, Q., & Cho, K. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00292 · Cited by 84
“For the NL2Code task, We use the Django dataset (Oda et al., 2015)5 and the MS COCO (Lin et al., 2014) with the standard split (Karpathy and Fei-Fei, 2015) for the NL2Code task and image captioning, respectively.”
MethodSelf-citationNon-Autoregressive Coarse-to-Fine Video Captioning
Yang, B., Zou, Y., Liu, F., & Zhang, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16421 · Cited by 108
“Another is that we pursue the works in NMT (Lee, Mansimov, and Cho 2018; Guo et al. 2019) to enhance decoder inputs by integrating the copied source information (the dashed line in Fig.”
BackgroundMethodInfluentialAligned Cross Entropy for Non-Autoregressive Machine Translation
Ghazvininejad, M., Karpukhin, V., Zettlemoyer, L., & Levy, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01655 · Cited by 74
“Some of these techniques include iterative refinement of translations based on previous predictions (Lee et al., 2018; Ghazvininejad et al., 2019; 2020; Gu et al., 2019; Kasai et al., 2020) and combining a lighter autoregressive decoder with a non-autoregressive one (Sun et al., 2019).”
BackgroundMethodInfluentialPOINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
Zhang, Y., Wang, G., Li, C., Gan, Z., Brockett, C., & Dolan, B. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.698 · Cited by 74
“Generation can be further refined with a post-processing step to remedy the conditional independence of the parallel decoding process (Lee et al., 2018; Ghazvininejad et al., 2019; Ma et al., 2019; Sun et al., 2019; Kasai et al., 2020).”
BackgroundSemi-Autoregressive Neural Machine Translation
Wang, C., Zhang, J., & Chen, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1044 · Cited by 97
“This phenomenon is echoing with observations by Gu et al. (2017); Oord et al. (2017); Lee et al. (2018).”
BackgroundMethodInfluentialSlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling
Wu, D., Ding, L., Lü, F., & Xie, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.152 · Cited by 98
Parallel Tacotron: Non-Autoregressive and Controllable TTS
Elias, I., Zen, H., Shen, J., Zhang, Y., Ye, J., Weiss, R. J., & Wu, Y. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414718 · Cited by 123
“The fine-grained phoneme-level VAE model was trained differ-ently.”
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2025) · ACM Computing Surveys · doi:10.1145/3754448 · Cited by 238
“GPUs also come equipped with high-bandwidth memory (HBM), which allows for faster data transfer rates, significantly reducing the bottleneck associated with memory access during large-scale computations.”
Towards Accurate Text-based Image Captioning with Content Diversity Exploration
Xu, G., Niu, S., Tan, M., Luo, Y., Du, Q., & Wu, Q. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01245 · Cited by 62
Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
Guo, J., Tan, X., Xu, L., Qin, T., Chen, E., & Liu, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6289 · Cited by 93
Challenges in End-to-End Neural Scientific Table Recognition
Deng, Y., Rosenberg, D. S., & Mann, G. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00148 · Cited by 94
“In addition to using self-attention mechanism to alleviate this issue, we hypothesize that iterative refinement might help [47], [48], [49].”
MethodNeural machine translation: Challenges, progress and future
Zhang, J., & Zong, C. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1632-x · Cited by 119
Imitation Learning for Non-Autoregressive Neural Machine Translation
Wei, B., Wang, M., Zhou, H., Lin, J., & Sun, X. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1125 · Cited by 65
“(Gu et al., 2017), (Lee et al., 2018) and (Kaiser et al., 2018) respectively. imitate-NAT is our proposed NAT with imitation learning.”
BackgroundMethodInfluentialLength-Controllable Image Captioning
Deng, C., Ding, N., Tan, M., & Wu, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_42 · Cited by 85
Transformer: A General Framework from Machine Translation to Others
Zhao, Y., Zhang, J., & Zong, C. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1393-5 · Cited by 93
Progressive Generation of Long Text with Pretrained Language Models
Tan, B., Yang, Z., Al-Shedivat, M., Xing, E., & Hu, Z. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.341 · Cited by 69
“, 2020), or refinement (Lee et al., 2018; Novak et al., 2016; Mansimov et al., 2019; Kasai et al., 2020) that differs from the restricted left-toright generation in conventional LMs.”
BackgroundAcquiring Knowledge from Pre-Trained Model to Neural Machine Translation
Weng, R., Yu, H., Huang, S., Cheng, S., & Luo, W. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6465 · Cited by 72
Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
Ran, Q., Lin, Y., Li, P., & Zhou, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17618 · Cited by 68
Non-Autoregressive Machine Translation with Disentangled Context Transformer
Kasai, J., Cross, J., Ghazvininejad, M., & Gu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.05136 · Cited by 56
“As in this work, several prior work proposed methods to iteratively refine output predictions (Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019; Mansimov et al., 2019).”
Discrete Flows: Invertible Generative Models of Discrete DataPreprint
Tran, D., Vafa, K., Agrawal, K. K., Dinh, L., & Poole, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10347 · Cited by 116
“Lee et al. (2018) adds an iterative refinement stage to initial parallel generations.”
BackgroundUnderstanding and Improving Lexical Choice in Non-Autoregressive TranslationPreprint
Ding, L., Wang, L., Liu, X., Wong, D. F., Tao, D., & Tu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14583 · Cited by 83
“Well-performed NAT models are generally trained on synthetic data distilled by autoregressive translation (AT) teachers instead of the raw training data (Figure 1(a)) (Stern et al., 2019; Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019).”
InfluentialHard-Coded Gaussian Attention for Neural Machine Translation
You, W., Sun, S., & Iyyer, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.687 · Cited by 57
FELIX: Flexible Text Editing Through Tagging and Insertion
Mallinson, J., Severyn, A., Malmi, E., & Garrido, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.111 · Cited by 122
Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation
Guo, J., Xu, L., & Chen, E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.36 · Cited by 63
“…with Curriculum Learning (FCL-NAT) (Guo et al., 2020), NAT with Dynamic Conditional Random Field (NAT-DCRF) (Sun et al., 2019); and two iterative decoding based works: NAT with Iterative Refinement (NAT-IR) (Lee et al., 2018) and Conditional Masked NAT (CM-NAT) (Ghazvininejad et al., 2019).”
BackgroundMethodInfluentialHint-Based Training for Non-Autoregressive Machine Translation
Li, Z., Lin, Z. J., He, D., Tian, F., Qin, T., Wang, L., & Liu, T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1573 · Cited by 68
“Comparing to the more than 15 × speed up in Gu et al. (2017), which uses a relatively simpler design choice of z , the speedup of Kaiser et al. (2018) is reduced to about 5 × , and the speedup of Lee et al. (2018) is reduced to about 2 × .”
BackgroundMethodResultInfluentialLexically Constrained Neural Machine Translation with Levenshtein Transformer
Susanto, R. H., Chollampatt, S., & Tan, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.325 · Cited by 56
KERMIT: Generative Insertion-Based Modeling for Sequences
Chan, W., Kitaev, N., Guu, K., Stern, M., & Uszkoreit, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01604 · Cited by 57
“We also follow prior work (Gu et al., 2018; Stern et al., 2018, 2019; Lee et al., 2018) in using distillation (Hinton et al., 2015; Kim and Rush, 2016) to train our models.”
MethodInfluentialLearning to Efficiently Sample from Diffusion Probabilistic Models
Watson, D., Ho, J., Norouzi, M., & Chan, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03802 · Cited by 48
“…very closely related to DDPMs, there has also been work on non-autoregressive modeling of text sequences that can be regarded as discrete-space DDPMs with a forward process that masks or remove tokens [Lee et al., 2018, Gu et al., 2019, Stern et al., 2019, Chan et al., 2020, Saharia et al., 2020].”
BackgroundCommonGen: A constrained text generation challenge for generative commonsense reasoning
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.165 · Cited by 239
“Recent advances (Lee et al., 2018; Stern et al., 2019) in conditional sentence generation have an emerging interest on (edit-based) non-autoregressive generation models, which iteratively refine generated sequences.”
BackgroundMinimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
Shao, C., Zhang, J., Feng, Y., Meng, F., & Zhou, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5351 · Cited by 91
A Study of Non-autoregressive Model for Sequence Generation
Ren, Y., Liu, J., Tan, X., Zhao, Z., Zhao, S., & Liu, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.15 · Cited by 80
Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
Guo, L., Liu, J., Zhu, X., He, X., Jiang, J., & Lu, H. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/107 · Cited by 78
Bai, Y., Yi, J., Tao, J., Tian, Z., Wen, Z., & Zhang, S. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3082299 · Cited by 80
“proposed an iterative refinement algorithm for MT [39].”
MethodErdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
Syntactically Supervised Transformers for Faster Neural Machine Translation
Akoury, N., Krishna, K., & Iyyer, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1122 · Cited by 64
“, 2014; Luong et al., 2015) focused on architectures and optimization. Sennrich and Haddow (2016) demonstrate that augmenting word embeddings with dependency relations helps NMT, while Shi et al.”
BackgroundMethodInfluentialImproved Mask-CTC for Non-Autoregressive End-to-End ASR
Higuchi, Y., Inaguma, H., Watanabe, S., Ogawa, T., & Kobayashi, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414198 · Cited by 75
Non-Autoregressive Dialog State TrackingPreprint
Le, H., Socher, R., & Hoi, S. C. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08024 · Cited by 55
“Several non-and semi-autoregressive approaches aim to generate tokens of the target language independently (Gu et al., 2018; Lee et al., 2018; Kaiser et al., 2018).”
BackgroundSpike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition
Tian, Z., Yi, J., Tao, J., Bai, Y., Zhang, S., & Wen, Z. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2086 · Cited by 66
Semantic Coding for Text Transmission: An Iterative Design
Yao, S., Niu, K., Wang, S., & Dai, J. (2022) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2022.3192407 · Cited by 61
“Inspired by source-only refinement [19] and channel-only iterative decoding [24], we explore the complementary nature of the semantic neural decoder and the channel decoder.”
MethodTheory and Experiments on Vector Quantized Autoencoders
Roy, A., Vaswani, A., Neelakantan, A., & Parmar, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11063 · Cited by 49
“Additionally, we see a large improvement in the performance of the model by using sequence-level distillation [12], as has been observed previously in non-autoregressive models [6, 15].”
MethodSemi-Autoregressive Transformer for Image Captioning
Zhou, Y., Zhang, Y., Hu, Z., & Wang, M. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00350 · Cited by 58
Learning towards conversational AI: A survey
Fu, T., Gao, S., Zhao, X., Wen, J.-R., & Yan, R. (2022) · AI Open · doi:10.1016/j.aiopen.2022.02.001 · Cited by 46
“Motivated by this, some researchers make efforts to study non-auto-aggressive generation paradigm (Kaiser et al., 2018; Lee et al., 2018), a new generation paradigm that relies on the hypothesis that every word is conditionally independent in per-step distribution.”
BackgroundNon-autoregressive Translation with Layer-Wise Prediction and Deep Supervision
Huang, C., Zhou, H., Zai͏̈ane, O. R., Mou, L., & Li, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21323 · Cited by 51
Seamless: Multilingual Expressive and Streaming Speech Translation
Communication, S., Barrault, L., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.-A., Ellis, B., Elsahar, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.05187 · Cited by 45
PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation
Hua, X., & Wang, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.57 · Cited by 42
A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation
Higuchi, Y., Chen, N., Fujita, Y., Inaguma, H., Komatsu, T., Lee, J., Nozaki, J., Wang, T., & Watanabe, S. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688157 · Cited by 58
Self-citationSemi-Autoregressive Training Improves Mask-Predict DecodingPreprint
Ghazvininejad, M., Levy, O., & Zettlemoyer, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08785 · Cited by 66
“Multi-Iteration SMART Lee et al. (2018) also proposed a semi-autoregressive training regime, in which the training process imitated the iterative refinement decoding algorithm.”
BackgroundMethodInfluentialDeeCap: Dynamic Early Exiting for Efficient Image Captioning
Fei, Z., Yan, X., Wang, S., & Tian, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01190 · Cited by 83
“Numerous works follow this line, e.g. , iterative refinement [13, 16, 23], la-12206 tent variables [13], multi-agent optimization [20], and semi-autoregressive [15, 57, 64].”
BackgroundParallel Refinements for Lexically Constrained Text Generation with BART
He, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.681 · Cited by 69
“To alleviate this problem, recent NAT models (Lee et al., 2018; Stern et al., 2019; Gu et al., 2019; Ghazvininejad et al., 2019) generate the output with several steps, making a trade-off between the decoding speed and generation quality.”
MethodSpeculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., & Sui, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.257 · Cited by 40
“For WMT16 EN ↔ RO translation, we use the dataset released by Lee et al. (2018), where newsdev2016 and newstest2016 are taken as validation and test sets.”
MethodRetrieving Sequential Information for Non-Autoregressive Neural Machine Translation
Shao, C., Feng, Y., Zhang, J., Meng, F., Chen, X., & Zhou, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1288 · Cited by 42
“IRNAT: iterative refinement for NAT (Lee et al., 2018).”
BackgroundMethodResultInfluentialBai, Y., Yi, J., Tao, J., Tian, Z., Wen, Z., & Zhang, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1600 · Cited by 46
Song, J., Kim, S., & Yoon, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.1 · Cited by 49
“For WMT16 En-Ro dataset, we use the preprocessed corpus provided by Lee et al. (2018).”
BackgroundMethodLearning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
Ran, Q., Lin, Y., Li, P., & Zhou, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.277 · Cited by 50
Xu, W., & Carpuat, M. J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00368 · Cited by 66
“Algorithm The reposition and insertion operations used in EDITOR are designed so that the Levenshtein edit distance algorithm (Levenshtein, 1966) can be used as the oracle.”
MethodFastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition
Leng, Y., Tan, X., Wang, R., Zhu, L., Xu, J., Liu, W., Liu, L., Li, X., Qin, T., Lin, E. C., et al. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.367 · Cited by 59
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
Tu, L., Pang, R. Y., Wiseman, S., & Gimpel, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.251 · Cited by 46
Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
Liu, J., Ren, Y., Tan, X., Zhang, C., Qin, T., Zhao, Z., & Liu, T. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/534 · Cited by 37
“To improve the inference speed of AT models, a series of works develop non-autoregressive translation (NAT) models based on Transformer [Gu et al., 2018; Lee et al., 2018; Li et al., 2019; Wang et al., 2019; Guo et al., 2019a], which generate all the target tokens in parallel.”
InfluentialAlign-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment
Chi, E. A., Salazar, J., & Kirchhoff, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.154 · Cited by 64
Progressive Multi-Granularity Training for Non-Autoregressive Translation
Ding, L., Wang, L., Liu, X., Wong, D. F., Tao, D., & Tu, Z. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.247 · Cited by 39
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
Ding, L., Wang, L., Liu, X., Wong, D. F., Tao, D., & Tu, Z. (2021) · doi:10.18653/v1/2021.acl-long.266 · Cited by 55
“Some researchers proposed new model architectures (Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019; Kasai et al., 2020), aided with additional signals (Wang et al., 2019; Ran et al., 2019; Ding et al., 2020), introduced sequential information (Wei et al., 2019; Shao et al., 2019; Guo…”
BackgroundResultA CTC Alignment-Based Non-Autoregressive Transformer for End-to-End Automatic Speech Recognition
Fan, R., Chu, W., Chang, P., & Alwan, A. A. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3263789 · Cited by 41
“generating output tokens in parallel [9], [10], [11], [12], [13], [14].”
BackgroundA Generalized Framework of Sequence Generation with Application to Undirected Sequence ModelsPreprint
Mansimov, E., Wang, A., Welleck, S., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12790 · Cited by 46
“Non-autoregressive neural sequence modeling by iterative refinement We next consider the other extreme in which we replace the symbols in all positions at every single generation step (Lee et al., 2018).”
BackgroundMethodResultSelf-citationRobustifying models against adversarial attacks by Langevin dynamics
Srinivasan, V., Rohrer, C., Marbán, A., Müller, K., Samek, W., & Nakajima, S. (2021) · Neural Networks · doi:10.1016/j.neunet.2020.12.024 · Cited by 53
“We propose to train a supervised denoising autoencoder (sDAE) [38, 39] by minimizing the following functional with respect to the function r : R 7→ R: Ep′(x,y)p′(ν) [ ‖r(x+ ν)− x‖(2) + 2σ(2)J (r(x+ ν),y) ] .”
MethodResultInfluentialA Voyage on Neural Machine Translation for Indic Languages
Sheshadri, S. K., Gupta, D., & Costa‐jussà, M. R. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.242 · Cited by 38
EdiT5: Semi-Autoregressive Text Editing with T5 Warm-Start
Mallinson, J., Adamek, J., Malmi, E., & Severyn, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.156 · Cited by 62
Context-Aware Cross-Attention for Non-Autoregressive Translation
Ding, L., Wang, L., Wu, D., Tao, D., & Tu, Z. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.389 · Cited by 45
Token-level Adaptive Training for Neural Machine Translation
Gu, S., Zhang, J., Meng, F., Feng, Y., Xie, W., Zhou, J., & Yu, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.76 · Cited by 42
Learning to Rewrite for Non-Autoregressive Neural Machine Translation
Geng, X., Feng, X., & Qin, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.265 · Cited by 42
“As an early alternative, Lee et al. (2018) corrected the original non-autoregressive output by passing it multiple times through a denoising autoencoder.”
MethodInfluentiallatent-GLAT: Glancing at Latent Variables for Parallel Text Generation
Bao, Y., Zhou, H., Huang, S., Wang, D., Qian, L., Dai, X., Chen, J., & Li, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.575 · Cited by 49
Metadata
- Authors
- Lee, J., Mansimov, E., & Cho, K.
- Affiliations
- New York University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1173-1182
- OpenAlex
- W2963434219
- Cited by
- 386
- References
- 46
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 24 · 2023: 55 · 2022: 57 · 2021: 88 · 2020: 86 · 2019: 43 · 2018: 5
- FWCI
- 22.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Samsung Advanced Institute of Technology
- Tencent
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- iterative refinement, deterministic decoding, latent variable models, denoising autoencoder, machine translation, image captioning, decoding speed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.