Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Open accessThe DOI matches its Crossref record; 41 of the 46 references it cites were found in a registry or an index.
Abstract
We propose a conditional non-autoregressive neural sequence model based on iterative refinement.The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task.We extensively evaluate the proposed model on machine translation (En$De and En$Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“It reduces memory costs by avoiding auxiliary models but relies on fixed-size drafting blocks. Verification is strict, and confidence awareness is minimal or implicit.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While recent advances in speculative decoding and non-autoregressive generation have made substantial strides toward accelerating sequence generation, most existing approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5×
“Each image is pre-transformed into a set of 49 512-dimensional feature vectors, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al.”
MethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“Training and Inference We use Adam (Kingma and Ba, 2014) and use L = 3 in Eq.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We extract 49 512-dimensional feature vectors for each image, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009).”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 6×
“…compare the proposed non-autoregressive model against the autoregressive counterpart both in terms of generation quality, measured in terms of BLEU (Papineni et al., 2002), and generation efficiency, measured in terms of (source) tokens and images per second for translation and image captioning,…”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodSelf-citationDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2×
“…Gu et al. (2017), in the context of machine translation, and Oord et al. (2017), in the context of speech generation, have recently discovered that it is important to use knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016) to successfully train a non-autoregressive sequence model.”
Background
Research works printed but not linked: 3
- Andrej Karpathy and Fei-Fei Li. 2015. Deep visualsemantic alignments for generating image descriptions. In CVPR .
- Yoon Kim and Alexander M Rush. 2016. Sequencelevel knowledge distillation. arXiv preprint arXiv:1606.07947 . · arXiv
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL .
Cited by
Items 101–200 of 376
Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation
He, X., Haffari, G., & Norouzi, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.275 · Cited by 62
Jointly Trained Transformers Models for Spoken Language Translation
Vydana, H. K., Karafiát, M., Žmolíková, K., Burget, L., & Černocký, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414159 · Cited by 37
“As all the models are transformers, they could be replaced with Non-Auto regressive models [19], [20],[21], which could reduce the latency of decoding.”
Han, X., Kumar, S., & Tsvetkov, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.647 · Cited by 266
Non-Autoregressive Translation by Learning Target Categorical Codes
Bao, Y., Huang, S., Xiao, T., Wang, D., Dai, X., & Chen, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.458 · Cited by 30
Partially Non-Autoregressive Image Captioning
Fei, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16219 · Cited by 42
“Recently, a flurry of nonautoregressive image captioning methods has been investigated to mitigate the slow decoding speed problem by generating all words independently in parallel (Jason et al. 2018; Guo et al. 2020; Fei 2019).”
BackgroundMethodInfluentialSubformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
Reid, M., Marrese-Taylor, E., & Matsuo, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.344 · Cited by 70
Zou, Y., Liu, Z., Hu, X., & Zhang, Q. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.169 · Cited by 36
“Another relevant line of research is refinement-based generation (Lee et al., 2018; Kasai et al., 2020; Hua and Wang, 2020; Tan et al., 2021), which gradually improves generation quality by iterative refinement on the draft instead of one-pass generation.”
BackgroundIterative Back Modification for Faster Image Captioning
Fei, Z. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413901 · Cited by 30
“Our approach bears similarity to the non-autoregressive captioning model in [20, 40], which also employs iterative refinement to improve the output sentence.”
MethodO2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Liu, F., Ren, X., Wu, X., Yang, B., Ge, S., & Sun, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.24 · Cited by 40
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image SynthesisPreprint
Zhang, Z., Ma, J., Zhou, C., Men, R., Li, Z., Ding, M., Tang, J., Zhou, J., & Yang, H. (2021) · arXiv (Cornell University) · Cited by 42
Multi-Task Semantic Dependency Parsing with Policy Gradient for Learning Easy-First Strategies
Kurita, S., & Søgaard, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1232 · Cited by 26
“Iterative refinement approaches have been proposed in the context of sentence generation (Lee et al., 2018).”
BackgroundDecoding Methods in Neural Language Generation: A Survey
Zarrieß, S., Voigt, H., & Schüz, S. (2021) · Information · doi:10.3390/info12090355 · Cited by 51
“However, they are much faster and in some domains, such as speech synthesis or machine translation, good quality can be reached by using techniques of knowledge distillation [49], probability density distillation [48], or iterative refinement [50].”
BackgroundSequence Generation: From Both Sides to the Middle
Zhou, L., Zhang, J., Zong, C., & Yu, H. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/760 · Cited by 26
Modeling Fluency and Faithfulness for Diverse Neural Machine Translation
Feng, Y., Xie, W., Gu, S., Shao, C., Zhang, W., Yang, Z., & Yu, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5334 · Cited by 39
Automatic Speech Recognition: A Comprehensive Survey
Rista, A., & Kadriu, A. (2020) · SEEU Review · doi:10.2478/seeur-2020-0019 · Cited by 29
“The encoder-decoder architecture has three encoders for handling three kinds of inputs: the acoustic feature sequence Xn and the latent variables Z for phoneme, the grapheme sequences and the auto-regressive recurrent neural net (RNN) decoders (Lee et al.,2018).”
BackgroundMethodInfluentialNon-autoregressive Transformer by Position LearningPreprint
Bao, Y., Zhou, H., Feng, J., Wang, M., Huang, S., Chen, J., & Li, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.10677 · Cited by 30
“We follow the common choice of approximating decoding algorithms (Gu et al., 2018; Lee et al., 2018) to reduce the search space of latent variable model.”
BackgroundMethodInfluentialNeural Language Generation: Formulation, Methods, and Evaluation
Gârbacea, C., & Mei, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15780 · Cited by 24
“Parallel decoding models include iterative refinement (Lee et al., 2018), noisy parallel decoding (Gu et al.”
BackgroundMethodInfluentialSpan Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing
Shrivastava, A., Chuang, P., Babu, A., Desai, S., Arora, A., Zotov, A., & Aly, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.161 · Cited by 23
A Non-Autoregressive Edit-Based Approach to Controllable Text Simplification
Agrawal, S., Xu, W., & Carpuat, M. J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.330 · Cited by 44
“Non-autoregressive Seq2Seq Models They have primarily been used to speed up Machine Translation by allowing parallel edit operations on the output sequence (Lee et al., 2018; Gu et al., 2018; Ghazvininejad et al., 2019; Stern et al., 2019; Chan et al., 2020; Xu and Carpuat, 2020).”
MethodSequence-Level Training for Non-Autoregressive Neural Machine Translation
Shao, C., Feng, Y., Zhang, J., Meng, F., & Zhou, J. (2021) · Computational Linguistics · doi:10.1162/coli_a_00421 · Cited by 35
Semantic Role Labeling with Iterative Structure Refinement
Lyu, C., Cohen, S. B., & Titov, I. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1099 · Cited by 25
“Refinement strategies have been successful in machine translation (Lee et al., 2018; Novak et al., 2017; Xia et al., 2017; Hassan et al., 2018), but their effectiveness in other NLP tasks is yet to be demonstrated.”
BackgroundMethodInfluentialCapturing Argument Interaction in Semantic Role Labeling with Capsule Networks
Chen, X., Lyu, C., & Titov, I. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1544 · Cited by 22
“Unlike standard refinement methods (Belanger et al., 2017; Lee et al., 2018) which sum losses across all refinement iterations, our loss is only based on the prediction made at the last iteration.”
MethodORTHROS: non-autoregressive end-to-end speech translation With dual-decoder
Inaguma, H., Higuchi, Y., Duh, K., Kawahara, T., & Watanabe, S. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415093 · Cited by 28
“However, iterative refinement methods generally achieve better quality than pure NAR methods at the cost of speed because of multiple forward passes [20, 23, 39].”
MethodSemi-Autoregressive Image Captioning
Yan, X., Fei, Z., Li, Z., Wang, S., Huang, Q., & Tian, Q. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475179 · Cited by 29
“Cho et al. [19] and Gao et al. [12] introduce an iterative mask refinement strategy to learn the position matching information.”
InfluentialImage Captioning With Controllable and Adaptive Length Levels
Ding, N., Deng, C., Tan, M., Du, Q., Ge, Z., & Wu, Q. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3328298 · Cited by 25
“More recently, several NAR image captioning models have been developed [43], [44], however, they usually suffer from a large performance degradation compared with the SOTA AR image captioning models.”
MethodResultIterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation
Lee, J., Shu, R., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.73 · Cited by 32
Self-citationImproving Non-autoregressive Neural Machine Translation with Monolingual Data
Zhou, J., & Keung, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.171 · Cited by 24
Learning to Model Editing Processes
Reid, M., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2205.12374 · Cited by 21
Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy
Zhang, S., & Feng, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.581 · Cited by 42
“6M pairs) (Lee et al., 2018) We use newsdev2016 (1999 pairs) as the validation set and news-test2016 (1999 pairs) as the test set.”
MethodA Primer on Seq2Seq Models for Generative Chatbots
Scotti, V., Sbattella, L., & Tedesco, R. (2023) · ACM Computing Surveys · doi:10.1145/3604281 · Cited by 38
“In fact, auto-encoders have been employed for neural machine translation [65, 81, 89, 96], and generic text generation [192].”
BackgroundMethodResultInfluentialCopy That! Editing Sequences by Copying Spans
Panthaplackel, S., Allamanis, M., & Brockschmidt, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17606 · Cited by 25
Contextualized Perturbation for Textual Adversarial AttackPreprint
Li, D., Zhang, Y., Peng, H., Chen, L., Brockett, C., Sun, M., & Dolan, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07502 · Cited by 20
“…from lexical substitution (Wu et al., 2019a; Zhou et al., 2019a; Qiang et al., 2020; Wu et al., 2019b, inter alia) to non-autoregressive generation (Gu et al., 2018; Lee et al., 2018; Ghazvininejad et al., 2019; Ma et al., 2019; Sun et al., 2019; Ren et al., 2020; Zhang et al., 2020b, inter alia).”
BackgroundMethodFunction-guided protein design by deep manifold samplingPreprint
Gligorijević, V., Berenberg, D., Ra, S., Watkins, A. M., Kelow, S. P., Cho, K., & Bonneau, R. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.12.22.473759 · Cited by 26
“We also consider known issues with autoregressive models including decoding latency [22, 23], difficulty of parallelization at inference time [24, 25, 26], and exposure bias at test-time generation [27, 28].”
BackgroundSelf-citationChuang, S.-P., Chuang, Y.-S., Chang, C.-C., & Lee, H.-Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.92 · Cited by 31
Learning to Model Editing Processes
Reid, M., & Neubig, G. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.280 · Cited by 46
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Feng, S., Ren, X., Chen, H., Sun, B., Li, K., & Sun, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.534 · Cited by 24
Imitation Learning for Non-Autoregressive Neural Machine TranslationPreprint
Wei, B., Wang, M., Zhou, H., Lin, J., Xie, J., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02041 · Cited by 19
“(Gu et al., 2017), (Lee et al., 2018) and (Kaiser et al., 2018) respectively. imitate-NAT is our proposed NAT with imitation learning.”
BackgroundMethodInfluentialIncorporating a Local Translation Mechanism into Non-autoregressive Translation
Kong, X., Zhang, Z., & Hovy, E. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.79 · Cited by 21
Uncertainty-Aware Image Captioning
Fei, Z., Fan, M., Zhu, L., Huang, J., Wei, X., & Wei, X. K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25137 · Cited by 26
Glancing Transformer for Non-Autoregressive Neural Machine TranslationPreprint
Qian, L., Zhou, H., Bao, Y., Wang, M., Qiu, L., Zhang, W., Yu, Y. H., & Li, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07905 · Cited by 18
“As for IWSLT16 DE-EN, we adopt linear annealing (from 3 e − 4 to 1 e − 5 ) as in Lee et al. (2018).”
MethodInfluentialParallel Tacotron: Non-Autoregressive and Controllable TTSPreprint
Elias, I., Zen, H., Shen, J., Zhang, Y., Ye, J., Weiss, R. J., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11439 · Cited by 18
“The fine-grained phoneme-level VAE model was trained differ-ently.”
Learning Structural Edits via Incremental Tree TransformationsPreprint
Yao, Z., Xu, F. F., Yin, P., Sun, H., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12087 · Cited by 24
Shao, C., Wu, X., & Feng, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.277 · Cited by 28
Multi-information interaction graph neural network for joint entity and relation extraction
Zhang, Y., Zhang, Y., Wang, Z., Peng, H., Yang, Y., & Li, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121211 · Cited by 17
Liu, Y., Wan, Y., Zhang, J., Zhao, W., & Yu, P. S. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.105 · Cited by 24
Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization
Liu, P., Huang, C., & Mou, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.545 · Cited by 23
Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution
Liu, Q., Yin, X., Yuille, A., Brown, A., & Singh, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00263 · Cited by 51
“Various tasks can be framed as cross-modal/intra-modal mapping problems, including image captioning [20, 24, 38, 39, 54, 80, 82], depth estimation [5, 15, 35, 40, 41, 64, 79], and image super-resolution [19, 67].”
Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems
Helcl, J., Haddow, B., & Birch, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.129 · Cited by 22
Set Generation Networks for End-to-End Knowledge Base Population
Sui, D., Wang, C., Chen, Y., Liu, K., Zhao, J., & Bi, W. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.760 · Cited by 20
“Nowadays, non-autoregressive models are widely explored in natural language and speech processing tasks such as neural machine translation (Lee et al., 2018; Ma et al., 2019) and automatic speech recognition (Chen et al., 2019; Tian et al., 2020; Bai et al., 2020).”
BackgroundLegoNN: Building Modular Encoder-Decoder Models
Dalmia, S., Okhonko, D., Lewis, M., Edunov, S., Watanabe, S., Metze, F., Zettlemoyer, L., & Mohamed, A. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3296019 · Cited by 23
ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation
Li, J., Tang, T., Zhao, W. X., Nie, J., & Wen, J.-R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.68 · Cited by 22
Sohrab, M. G., Asada, M., Rikters, M., & Miwa, M. (2023) · IEEE Access · doi:10.1109/access.2023.3346952 · Cited by 17
“Several S2S-based NAR methods [7], [8] have been proposed to approach the performance of AR methods.”
MethodSubword Language Model for Query Auto-Completion
Kim, G. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1507 · Cited by 21
“In this sense, non-autoregressive neural machine translation (Gu et al., 2017; Lee et al., 2018) is related to our work.”
Joint Phoneme-Grapheme Model for End-To-End Speech Recognition
Kubo, Y., & Bacchiani, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054557 · Cited by 13
“It should also be noted that our iterative refinement model here uses auto-regressive recurrent neural net (RNN) decoders whereas the original work on iterative refinement [7] aimed at using non-auto-regressive models.”
MethodFastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire
Liu, J., Ren, Y., Zhao, Z., Zhang, C., Huai, B., & Yuan, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413740 · Cited by 16
Mask and Predict: Multi-step Reasoning for Scene Graph Generation
Tian, H., Xu, N., Liu, A.-A., Yan, C. C., Mao, Z., Zhang, Q., & Zhang, Y. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475545 · Cited by 13
“The strategy of multi-step reasoning has been embodied in many areas, not just visual tasks like object detection [6] [5] and visual question answering (VQA) [49] [27], but some text tasks such as machine translation [8] [16] and machine reading comprehension [23] [34].”
BackgroundAutoregressive Text Generation Beyond Feedback Loops
Schmidt, F., Mandt, S., & Hofmann, T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1338 · Cited by 19
“Rush, 2019) and conditional (Lee et al., 2018) generation.”
BackgroundGuiding Teacher Forcing with Seer Forcing for Neural Machine Translation
Feng, Y., Gu, S., Guo, D., Yang, Z., & Shao, C. (2021) · doi:10.18653/v1/2021.acl-long.223 · Cited by 24
“English→Romanian We used the preprocessed version of WMT16 En-Ro dataset released by Lee et al. (2018) which includes 0.6M sentence pairs.”
MethodConsistency Training with Virtual Adversarial Discrete Perturbation
Park, J., Kim, G., & Kang, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.414 · Cited by 18
Xu, W., Ma, S., Zhang, D., & Carpuat, M. J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.385 · Cited by 23
Minimum Word Error Training For Non-Autoregressive Transformer-Based Code-Switching ASR
Peng, Y., Zhang, J., Xu, H., Huang, H., & Chng, E. S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746830 · Cited by 14
“In this paper, we take the iterative inference approach, following what is proposed in [17, 20].”
MethodNon-Autoregressive Sequence Generation
Gu, J., & Tan, X. (2022) · doi:10.18653/v1/2022.acl-tutorials.4 · Cited by 16
“For text generation , we cover several tasks: (1) neural machine translation (Gu et al., 2018; Lee et al., 2018; Wang et al., 2018; Kong et al., 2020b; Gu and Kong, 2021); (2) text summarization (Gu et al., 2019; Qi et al., 2021; Jiang et al., 2021); (3) text error correction (Awasthi et al., 2019;…”
MethodOpen-ended Long Text Generation via Masked Language Modeling
Liang, X., Tang, Z., Li, J., & Zhang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.13 · Cited by 33
“Furthermore, the iterative-based NAR model (Lee et al., 2018; Gu et al., 2019; Chi et al., 2021) can achieve comparable performance with the AR model.”
ResultToken Drop mechanism for Neural Machine Translation
Zhang, H., Qiu, S., Duan, X., & Zhang, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.379 · Cited by 13
Neural Networks for Modeling Source Code EditsPreprint
Zhao, R. T., Bieber, D., Swersky, K., & Tarlow, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02818 · Cited by 14
“The structured objects could be images (Gregor et al., 2015; Denton et al., 2015; Karras et al., 2018), translations (Li et al., 2017; Novak et al., 2016; Niehues et al., 2016; Lee et al., 2018), or source code (Gupta et al., 2017; Shin et al., 2018).”
Background(2021) · doi:10.18653/v1/2021.findings-acl.427 · Cited by 10
Mixup Decoding for Diverse Machine Translation
Li, J., Gao, P., Wu, X., Feng, Y., He, Z., Wu, H. R., & Wang, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.29 · Cited by 10
Zhu, M., Wang, J., & Yan, C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.45 · Cited by 10
LenM: Improving Low-Resource Neural Machine Translation Using Target Length Modeling
Mahsuli, M. M., Khadivi, S., & Homayounpour, M. M. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11208-1 · Cited by 15
A Survey of Non-Autoregressive Neural Machine Translation
Li, F., Chen, J., & Zhang, X. (2023) · Electronics · doi:10.3390/electronics12132980 · Cited by 13
On the Discrepancy between Density Estimation and Sequence Generation
Lee, J., Tran, D., Fırat, O., & Cho, K. (2020) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2020.spnlp-1.10 · Cited by 11
Self-citationNon-Autoregressive Transformer with Unified Bidirectional Decoder for Automatic Speech Recognition
Zhang, C., Liu, Y., Zhang, T., Chen, S.-L., Feng, C., & Yin, X.-C. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746903 · Cited by 13
Boundary and Context Aware Training for CIF-Based Non-Autoregressive End-to-End ASR
Yu, F., Luo, H., Guo, P., Liang, Y., Yao, Z., Xie, L., Gao, Y., Hou, L., & Zhang, S. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688238 · Cited by 16
Neural Machine Translation: A ReviewPreprint
Stahlberg, F. (2019) · Journal of Artificial Intelligence Research · doi:10.48550/arxiv.1912.02047 · Cited by 459
“• Non- or partially autoregressive architectures (Wang et al., 2018a; Gu et al., 2017a; Guo et al., 2018; Wang et al., 2019; Libovický & Helcl, 2018; Lee et al., 2018; Akoury et al., 2019)”
BackgroundScaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-1380 · Cited by 10
Non-Autoregressive Neural Machine Translation: A Call for Clarity
Schmidt, R. M., Pires, T., Peitz, S., & Lööf, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.179 · Cited by 11
Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation
Shao, C., Ma, Z., & Feng, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.322 · Cited by 13
Translating Images to Road Network: A Non-Autoregressive Sequence-to-Sequence Approach
Lu, J., Li, H., Peng, R., Wen, F., Cai, X., Zhang, W., Xu, H., & Zhang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00009 · Cited by 19
“Going beyond SAR-RNTR, we employ a masked training technique on SAR-RNTR to mimic the remaining auto-regressive dependency through iterative prediction [32].”
MethodRare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking
Schick, T., & Schütze, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6403 · Cited by 83
Voita, E., Sennrich, R., & Titov, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.667 · Cited by 33
“This is called sequence-level knowledge distillation (Kim and Rush, 2016), and it is currently one of the de-facto standard parts of the NAT training pipelines (Gu et al. (2018); Lee et al. (2018); Ghazvininejad et al. (2019) to name a few).”
BackgroundData-to-text Generation by Splicing Together Nearest Neighbors
Wiseman, S., Bačkurs, A., & Stratos, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.352 · Cited by 15
“…work in non-left-to-right text generation, including tree-based (Welleck et al., 2019; Akoury et al., 2019), non-autoregressive (Gu et al., 2018a; Lee et al., 2018, inter alia ), masked language model-based (Ghazvininejad et al., 2019, inter alia ), and, most closely, insertion-based (Stern et…”
AMOM: Adaptive Masking over Masking for Conditional Masked Language Model
Xiao, Y., Xu, R., Wu, L., Li, J., Qin, T., Liu, T., & Zhang, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26615 · Cited by 16
LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary AttentionPreprint
Li, X., Meng, Y., Yuan, A., Wu, F., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03084 · Cited by 10
“36 x iNAT (Lee et al., 2018) 21.”
BackgroundFaltings, F., Galley, M., Brantley, K., Peng, B., Cai, W., Zhang, Y., Gao, J., & Dolan, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.270 · Cited by 24
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
Xie, P., Cui, Z. M., Chen, X., Hu, X., Cui, J., & Wang, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.2 · Cited by 11
TranSFormer: Slow-Fast Transformer for Machine Translation
Li, B., Yi, J., Tan, X., Xing, Z., Xiao, T., & Zhu, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.430 · Cited by 14
Decoding Language Spatial Relations to 2D Spatial Arrangements
Radevski, G., Collell, G., Moens, M., & Tuytelaars, T. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.408 · Cited by 10
Non-Autoregressive Neural Dialogue GenerationPreprint
Han, Q., Meng, Y., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04250 · Cited by 14
“Non-autoregressive (non-AR) generation (Gu et al., 2018; Ma et al., 2019; Lee et al., 2018) provides a natural way to address”
BackgroundMethodInfluentialAn Empirical Study of Generation Order for Machine Translation
Chan, W., Stern, M., Kiros, J., & Uszkoreit, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.464 · Cited by 7
Nearest Neighbor Non-autoregressive Text Generation
Niwa, A., Takase, S., & Okazaki, N. (2023) · Journal of Information Processing · doi:10.2197/ipsjjip.31.344 · Cited by 10
Yang, W., Li, Z., Peng, D., Jin, L., He, M., & Yao, C. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612499 · Cited by 7
“Thus, some works have taken an iterative refinement strategy [5, 12, 27, 36], but this strategy is not flexible enough to deal with the uncertainty of long sequence lengths like multi-line mathematical expressions.”
BackgroundInference Strategies for Machine Translation with Conditional Masking
Kreutzer, J., Foster, G. F., & Cherry, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.465 · Cited by 8
Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation
Liu, G., Yang, Z., Tao, T., Liang, X., Bao, J., Li, Z., He, X., Cui, S. R., & Hu, Z. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.150 · Cited by 22
Parallel and High-Fidelity Text-to-Lip Generation
Liu, J., Zhu, Z., Ren, Y., Huang, W., Huai, B., Yuan, N. J., & Zhao, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20066 · Cited by 12
“…generates all the target tokens in parallel, which has already pervaded multiple research fields such as neural machine translation (Gu et al. 2018; Lee, Mansimov, and Cho 2018; Ghazvinine-jad et al. 2019; Ma et al. 2019), speech recognition (Chen et al. 2019; Higuchi et al. 2020), speech…”
BackgroundMethodRenewNAT: Renewing Potential Translation for Non-autoregressive Transformer
Guo, P., Xiao, Y., Li, J., & Zhang, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26511 · Cited by 10
Wang, X., Zheng, Z., & Huang, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.371 · Cited by 7
Improving Autoregressive NMT with Non-Autoregressive Model
Zhou, L., Zhang, J., & Zong, C. (2020) · doi:10.18653/v1/2020.autosimtrans-1.4 · Cited by 6
“To speed up the inference of machine translation, non-autoregressive translation (NAT) models have been proposed, which generate all target tokens independently and simultaneously (Gu et al., 2017; Lee et al., 2018; Kaiser et al., 2018; Libovick´y and Helcl, 2018).”
MethodTrain Once, and Decode As You Like
Tian, C., Wang, Y., Cheng, H., Lian, Y., & Zhang, Z. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.25 · Cited by 7
“We compare the performances of our model with three refinement-based non-autoregressive models: NAT-IR (Lee et al., 2018), Latent-Variable NAR (Shu et al., 2019), CMLM (Ghazvininejad et al., 2019) and three other popular non-autoregressive models: NAT (Gu et al., 2017), FlowSeq (Ma et al., 2019),…”
BackgroundMethodInfluentialHint-Based Training for Non-Autoregressive Machine TranslationPreprint
Li, Z., Lin, Z. J., He, D., Tian, F., Qin, T., Wang, L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06708 · Cited by 7
“Comparing to the more than 15 × speed up in Gu et al. (2017), which uses a relatively simpler design choice of z , the speedup of Kaiser et al. (2018) is reduced to about 5 × , and the speedup of Lee et al. (2018) is reduced to about 2 × .”
BackgroundMethodResultInfluentialFine-Grained Image Captioning by Ranking Diffusion Transformer
Wan, J., Gan, M., Zhang, L., Zhou, J., Liu, J., Du, B., & Chen, C. L. P. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3641303 · Cited by 13
Metadata
- Authors
- Lee, J., Mansimov, E., & Cho, K.
- Affiliations
- New York University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1173-1182
- OpenAlex
- W2963434219
- Cited by
- 386
- References
- 46
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 24 · 2023: 55 · 2022: 57 · 2021: 88 · 2020: 86 · 2019: 43 · 2018: 5
- FWCI
- 22.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Samsung Advanced Institute of Technology
- Tencent
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- iterative refinement, deterministic decoding, latent variable models, denoising autoencoder, machine translation, image captioning, decoding speed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.