Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Open accessThe DOI matches its Crossref record; 41 of the 46 references it cites were found in a registry or an index.
Abstract
We propose a conditional non-autoregressive neural sequence model based on iterative refinement.The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task.We extensively evaluate the proposed model on machine translation (En$De and En$Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“It reduces memory costs by avoiding auxiliary models but relies on fixed-size drafting blocks. Verification is strict, and confidence awareness is minimal or implicit.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While recent advances in speculative decoding and non-autoregressive generation have made substantial strides toward accelerating sequence generation, most existing approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5×
“Each image is pre-transformed into a set of 49 512-dimensional feature vectors, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al.”
MethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“Training and Inference We use Adam (Kingma and Ba, 2014) and use L = 3 in Eq.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We extract 49 512-dimensional feature vectors for each image, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009).”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 6×
“…compare the proposed non-autoregressive model against the autoregressive counterpart both in terms of generation quality, measured in terms of BLEU (Papineni et al., 2002), and generation efficiency, measured in terms of (source) tokens and images per second for translation and image captioning,…”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodSelf-citationDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2×
“…Gu et al. (2017), in the context of machine translation, and Oord et al. (2017), in the context of speech generation, have recently discovered that it is important to use knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016) to successfully train a non-autoregressive sequence model.”
Background
Research works printed but not linked: 3
- Andrej Karpathy and Fei-Fei Li. 2015. Deep visualsemantic alignments for generating image descriptions. In CVPR .
- Yoon Kim and Alexander M Rush. 2016. Sequencelevel knowledge distillation. arXiv preprint arXiv:1606.07947 . · arXiv
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL .
Cited by
Items 201–300 of 376
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation
Huang, F., Ke, P., & Huang, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00582 · Cited by 17
“To alleviate the problem, previous studies propose methods including (1) iterative refinement (Lee et al., 2018; Gu et al., 2019; Ghazvininejad et al., 2019; Guo et al., 2020a; Huang et al., 2022d); (2) knowledge distillation (Kim and Rush, 2016; Ding et al.”
MethodMasumura, R., Ihori, M., Tanaka, T., Saito, I., Nishida, K., & Oba, T. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003857 · Cited by 4
“In addition, our proposed method introduces iterative processing as in utterance-level automatic post-editing [28].”
MethodHow Length Prediction Influence the Performance of Non-Autoregressive Translation?
Wang, M., Guo, J., Wang, Y., Chen, Y., Su, C., Shang, H., Zhang, M., Tao, S., & Yang, H. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.14 · Cited by 6
“Another pre-determines the target side length N and generates tokens in a fixed length space conditioned on well-predicted tokens and optionally copied source embedding within N iterations (Gu et al., 2018; Lee et al., 2018; Ghazvininejad et al., 2019).”
BackgroundMethodInfluentialAnticipation-Free Training for Simultaneous Machine Translation
Chang, C., Chuang, S.-P., & Lee, H.-Y. (2022) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2022.iwslt-1.5 · Cited by 7
“The causal encoder models follow a similar training process to non-autoregressive translation (NAT) (Gu et al., 2018; Libovický and Helcl, 2018; Lee et al., 2018; Zhou et al., 2020).”
MethodBilingual Synchronization: Restoring Translational Relationships with Editing Operations
Xu, J., Crego, J., & Yvon, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.548 · Cited by 6
Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation
Liu, M., Bao, Y., Zhao, C., & Huang, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26555 · Cited by 7
Parallel Synthesis for Autoregressive Speech Generation
Hsu, P.-C., Liu, D.-R., Liu, A. T., & Lee, H.-Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3301212 · Cited by 6
“A similar tendency has been observed in natural language processing (NLP), where studies have indicated the effectiveness of autoregressive models in capturing sequential dependencies and contextual information [61], [62], [63], [64].”
BackgroundXLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation
Wang, Y., He, S., Chen, G., Chen, Y., & Jiang, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.466 · Cited by 7
“We find that the training objective in pre-trained MLM can align well with the one in non-autoregressive generation, which is an emergent generation paradigm due to its excellent inference speed-up (Gu et al., 2018; Lee et al., 2018).”
BackgroundMethodInfluentialHypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine Translation
Li, S., Zhang, P., Gan, G., Lv, X., Wang, B., Wei, V. J., & Jiang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.475 · Cited by 20
“For En-Ro, we follow the setup of Lee et al. (2018), which includes 610K training sentence pairs.”
Syntax-driven Iterative Expansion Language Models for Controllable Text Generation
Casas, N., Fonollosa, J. A. R., & Costa‐jussà, M. R. (2020) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2020.spnlp-1.1 · Cited by 9
Predicting and Using Target Length in Neural Machine Translation
Yang, Z. G., Gao, Y., Wang, W., & Ney, H. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.41 · Cited by 6
“, 2019; Takase and Okazaki, 2019) and nonautoregressive (or partially autoregressive) NMT (Gu et al., 2017; Lee et al., 2018; Stern et al., 2019)”
BackgroundInfluentialCTC-based Non-autoregressive Speech Translation
Xu, C., Liu, X., Liu, X., Sun, Q., Zhang, Y., Yang, M., Dong, Q., Ko, T., Wang, M., Xiao, T., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.744 · Cited by 9
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
Ge, T., Xia, H., Sun, X., Chen, S.-Q., & Wei, F. (2022) · arXiv · doi:10.48550/arxiv.2205.10350 · Cited by 6
Non-autoregressive personalized bundle generation
Yang, W., Yang, C. H., Li, J., Tan, Y., Lu, X. B., & Shi, C. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103814 · Cited by 7
Introducing bidirectional attention for autoregressive models in abstractive summarization
Zhao, J., Sun, X., & Feng, C. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121497 · Cited by 5
TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-agent Sports
Capellera, G., Ferraz, L., Rubio, A., Agudo, A., & Moreno-Noguer, F. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0901-7_20 · Cited by 21
Zheng, H., Zhu, W., Wang, P., & Wang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.318 · Cited by 6
Composing Ci with Reinforced Non-autoregressive Text Generation
Song, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.486 · Cited by 6
“, 2018), image captioning (Lee et al., 2018), and summarization (Qi et al.”
BackgroundVCC-DiffNet: Visual Conditional Control Diffusion Network for Remote Sensing Image Captioning
Cheng, Q., Xu, Y., & Huang, Z. (2024) · Remote Sensing · doi:10.3390/rs16162961 · Cited by 7
Non-autoregressive Deliberation-Attention based End-to-End ASR
Gao, C., Cheng, G., Zhou, J., Zhang, P., & Yan, Y. (2021) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp49672.2021.9362115 · Cited by 5
“In order to improve the decoding efficiency of attentionbased models, non-autoregressive attention-based models are attracting increasingly more attention in many fields [9, 10, 11, 12].”
BackgroundMethodInfluentialHow does momentum benefit deep neural networks architecture design? A few case studies
Wang, B., Xia, H., Nguyen, T., & Osher, S. (2022) · Research in the Mathematical Sciences · doi:10.1007/s40687-022-00352-0 · Cited by 15
“We follow the setting in [50].”
MethodLiang, X., Wu, L., Li, J., & Zhang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.550 · Cited by 5
“Meanwhile, the non-autoregressive (NAR) (Gu et al., 2018; Lee et al., 2018; Libovick`y and Helcl, 2018; Su et al., 2021) generation methods show great potential to reduce the inference latency by introducing parallel decoding.”
BackgroundModeling Coverage for Non-Autoregressive Neural Machine Translation
Shan, Y., Feng, Y., & Shao, C. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533529 · Cited by 8
“We closely follow the settings of [6], [15].”
Improving Neural Machine Translation with Pre-trained RepresentationPreprint
Weng, R., Yu, H., Huang, S., Luo, W., & Chen, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07688 · Cited by 4
“For example, the exposure bias (Lee et al., 2018; Wu et al., 2018) leads to that the groundtruth sentence representation which is generated by the reference is not available in the inference stage.”
BackgroundParallel Iterative Edit Models for Local Sequence TransductionPreprint
Awasthi, A., Sarawagi, S., Goyal, R., Ghosh, S., & Piratla, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02893 · Cited by 27
“We create special compounds edits that merge token inserts with preceding edits that yield higher accuracy than earlier methods of independently predicting inserts (Ribeiro et al., 2018; Lee et al., 2018).”
BackgroundMethodInfluentialImproving Fluency of Non-Autoregressive Machine TranslationPreprint
Kasner, Z., Libovický, J., & Helcl, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03227 · Cited by 5
Improved Mask-CTC for Non-Autoregressive End-to-End ASRPreprint
Higuchi, Y., Inaguma, H., Watanabe, S., Ogawa, T., & Kobayashi, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13270 · Cited by 5
“NAR models in neural machine translation have achieved competitive performance to AR models [18–25].”
BackgroundScreening through a broad pool: Towards better diversity for lexically constrained text generation
Yuan, C., Huang, H., Cao, Y., & Cao, Q. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103602 · Cited by 5
Non-Autoregressive Fully Parallel Deep Convolutional Neural Speech Synthesis
Lee, M., Lee, J., & Chang, J. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3156797 · Cited by 5
Self-citationGEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding
Yakovlev, K., Podolskiy, A., Bout, A., Nikolenko, S., & Piontkovskaya, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.86 · Cited by 4
“Since non-autoregressive models are not capable of modeling target side dependencies, several approaches have been proposed to alleviate this issue: knowledge distillation (Gu et al., 2017; Lee et al., 2018), iterative decoding (Ghazvininejad et al.”
BackgroundMethodDePA: Improving Non-autoregressive Translation with Dependency-Aware Decoder
Zhan, J., Chen, Q., Chen, B., Wang, W., Bai, Y., & Gao, Y. (2023) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2023.iwslt-1.47 · Cited by 10
“We hypothesize that target-side embedding compression may also alleviate over-fitting on small datasets and confirm this hypothesis in Section 4.3. et al., 2018).”
BackgroundMethodInfluentialSelf-Improvement of Non-autoregressive Model via Sequence-Level Distillation
Liao, Y., Jiang, S., Li, Y., Wang, Y., & Wang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.878 · Cited by 4
“For WMT16 EN ↔ RO, we use the pre-processed data provided by Lee et al. (2018).”
MethodFast Interleaved Bidirectional Sequence Generation
Zhang, B., Titov, I., & Sennrich, R. (2020) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2010.14481 · Cited by 5
“This motivates researchers to discover the optimal balance by resorting to semi-autoregressive modeling (Wang et al., 2018; Stern et al., 2018), iterative refinement (Lee et al., 2018; Stern et al., 2019; Ghazvininejad et al., 2019) or in-between (Kaiser et al., 2018; Akoury et al., 2019).”
Rule-based Morphological Inflection Improves Neural Terminology Translation
Xu, W., & Carpuat, M. J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.477 · Cited by 7
“…Constraints Instead of generating the output sequence incrementally from left to right, non-autoregressive NMT generates tokens in parallel (Gu et al., 2018; van den Oord et al., 2018; Ma et al., 2019) or by iteratively editing an initial sequence (Lee et al., 2018; Ghazvininejad et al., 2019).”
Exploring Non-Autoregressive Text Style Transfer
Ma, Y., & Li, Q. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.730 · Cited by 7
“…is proposed, which seeks a tradeoff between AR and NAR decoding by adding AR layers on NAR models (Wang et al., 2018; Sun et al., 2019; Akoury et al., 2019), iterative refinement (Lee et al., 2018; Ghazvininejad et al., 2019), or insertion-based decoding (Stern et al., 2019; Gu et al., 2019).”
BackgroundAmortized Noisy Channel Neural Machine Translation
Pang, R. Y., He, H., & Cho, K. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.11 · Cited by 5
Self-citationNon-autoregressive Neural Machine Translation with Distortion Model
Zhou, L., Zhang, J., Zhao, Y., & Zong, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_32 · Cited by 4
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative FlowPreprint
Ma, X., Zhou, C., Li, X., Neubig, G., & Hovy, E. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02480 · Cited by 48
“We use the data provided in Lee et al. (2018)”
MethodSyntax-driven Iterative Expansion Language Models for Controllable Text GenerationPreprint
Casas, N., Fonollosa, J. A. R., & Costa‐jussà, M. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02211 · Cited by 4
“Lee et al. (2018) propose a latent variable nonautoregressive machine translation model where first the target length is predicted by the model, and then, the decoder is iteratively applied to its own output to refine it.”
BackgroundRegularizing Dialogue Generation by Imitating Implicit ScenariosPreprint
Feng, S., Ren, X., Chen, H.-S., Sun, B., Li, K., & Sun, X. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.01893 · Cited by 22
“…of future conversations, the distilled knowledge (Hinton et al., 2015) is also a less noisy and more “deterministic” supervised signal in comparison to real-world responses (Lee et al., 2018; Guo et al., 2019), which provides the student with smoother sequence trajectories that are easier to fit.”
BackgroundFaster Re-translation Using Non-Autoregressive Model For Simultaneous Neural Machine TranslationPreprint
Han, H., Indurthi, S. R., Zaidi, M. A., Lakumarapu, N. K., Lee, B., Kim, S., Kim, C., & Hwang, I.-C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14681 · Cited by 5
“Gu et al. (2017) propose a non-autoregressive framework which uses discrete latent variables, and it is later adopted in Lee, Mansimov, and Cho (2018) as an iterative refinement process.”
MethodLiu, Y., Wan, Y., Zhang, J., Zhao, W., & Yu, P. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08942 · Cited by 4
“For example, several techniques in nonautoregressive models are investigated to mitigate the trade-off between speedup and performance, including iterative refinement (Lee et al., 2018), insertion-based models (Chan et al.”
MethodInfluentialNon-Autoregressive Translation by Learning Target Categorical CodesPreprint
Bao, Y., Huang, S., Xiao, T., Wang, D., Dai, X., & Chen, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11405 · Cited by 4
“Lee et al. (2018) propose a method of iterative refinement based on the previous outputs.”
BackgroundMethodInfluentialProgressive Generation of Long Text with Pretrained Language ModelsPreprint
Tan, B., Yang, Z., Al-Shedivat, M., Xing, E. P., & Hu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15720 · Cited by 4
Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive RescoringPreprint
Inaguma, H., Higuchi, Y., Duh, K., Kawahara, T., & Watanabe, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04411 · Cited by 12
Monotonic Simultaneous Translation with Chunk-wise Reordering and RefinementPreprint
Han, H., Ahn, S., Choi, Y., Chung, I., Kim, S., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09646 · Cited by 4
“Also, there exists repetition problem in NAT (Lee et al., 2018; Gu and Kong, 2021) which is generated in the process of multiple chunk-wise iterative refinement.”
Self-citationFine-Grained Length Controllable Video Captioning With Ordinal Embeddings
Nitta, T., Fukuzawa, T., & Tamaki, T. (2024) · IEEE Access · doi:10.1109/access.2024.3506751 · Cited by 5
“Starting with a sentence filled with MASK tokens of a specified length, LaBERT [13] and LaNAR-BERT [16] iteratively refine the generated sentence [21, 40], while Gibbs-BERT [76] predicts words iteratively using Gibbs sampling.”
MethodA non-autoregressive Chinese-Braille translation approach with CTC loss optimization
Yu, H., Su, W., Yang, Y., Liu, L., Yuan, Y., Xie, Y., & Huang, T. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126356 · Cited by 6
Text Diffusion with Reinforced Conditioning
Liu, Y., Yang, T., Huang, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., & Zhang, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29316 · Cited by 8
Answering Ambiguous Questions via Iterative Prompting
Sun, W., Cai, H., Chen, H., Ren, P., Chen, Z., de Rijke, M., & Ren, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.424 · Cited by 22
Plan-then-Seam: Towards Efficient Table-to-Text Generation
Li, L., Geng, R., Fang, C., Li, B., Ma, C., Li, B., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.15 · Cited by 6
Leveraging Diverse Modeling Contexts With Collaborating Learning for Neural Machine Translation
Liao, Y., Wang, Y., & Wang, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3379884 · Cited by 4
“As a result, the DCMCL HYB can be optimized by the objectives: For the WMT16 EN-RO dataset, we use the pre-processed data provided by [41].”
MethodSun, S., Dong, X., Sha, Y., Sun, Q. L., Chen, Q., Zhou, J. P., & Zhuo, C. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3571314 · Cited by 3
“This eliminates the need for the model to predict sequence lengths dynamically, which is often a challenge in NART-based models [47], [48].”
Improving Non-autoregressive Machine Translation with Soft-Masking
Wang, S., Shi, S., & Huang, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_12 · Cited by 3
Continuous Language Generative Flow
Tang, Z., Zhang, S., Kim, H., & Bansal, M. (2021) · doi:10.18653/v1/2021.acl-long.355 · Cited by 4
“We first replicate Lee et al. (2018)’s transformer autoregressive model baseline, and then we add our flow architecture on top of it.”
MethodImproving Bidirectional Decoding with Dynamic Target Semantics in Neural Machine TranslationPreprint
Shan, Y., Feng, Y., Zhang, J., Meng, F., & Zhang, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01597 · Cited by 3
“For En⇒Ro, we use the preprocessed version of WMT’16 dataset released by Lee et al. (2018) that comprises 0.6M sentence pairs.”
MethodTensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language ModelingPreprint
Zhang, S., Zhang, P., Ma, X., Wei, V. J., Wang, N., & Liu, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01547 · Cited by 3
“8 Iterative Refinement NAT [10] 30.”
Injection of linguistic knowledge into neural text generation models
Manzanares, N. C. (2020) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-341598 · Cited by 3
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive TranslationPreprint
Ding, L., Wang, L., Liu, X., Wong, D. F., Tao, D., & Tu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00903 · Cited by 3
“Some researchers proposed new model architectures (Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019; Kasai et al., 2020), aided with additional signals (Wang et al., 2019; Ran et al., 2019; Ding et al., 2020), introduced sequential information (Wei et al., 2019; Shao et al., 2019; Guo…”
BackgroundResultLiu, Y., Yu, H., Li, H., Han, X., Zhang, X., & Liu, H. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3579587 · Cited by 2
“The compared models include Up-Down [2], GCN-LSTM [23], SGAE [32], ViTCAP [33], COS-Net [34], ExpansionNet v2 [35], AIC-XE, AIC-SCST; STAIC [36], PNAIC [37], SAIC [38]; Bit Diffusion [17], SCD-Net [39]; MNIC [12], MIR [40]; IBM [11]; FNIC [13], and CMAL [29].”
Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities
Chen, Z., Zhu, B., Wang, J., Shin, H., Nallanathan, A., & Niyato, D. T. (2026) · ACM Computing Surveys · doi:10.1145/3809166 · Cited by 12
“For example, AWQ [95] preserves only 1% of salient weights identified via activation statistics to mitigate quantization error, achieving more than 3× speedup over FP16 implementation on both desktop and mobile GPUs.”
Multi-Modal Soccer Scene Analysis with Masked Pre-Training
Peral, M., Capellera, G., Ferraz, L., Rubio, A., & Agudo, A. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00345 · Cited by 2
“Trajectory forecasting [5, 7, 17, 57, 58, 60] and imputation [4, 8, 16, 32, 35, 38, 53, 54] assume partial knowledge of the target agent’s history, but that reliance on past or future information compromises those models performance for long sequences [19, 27].”
Incorporating history and future into non-autoregressive machine translation
Wang, S., Huang, H., & Shi, S. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2022.101439 · Cited by 3
Bidirectional Sentence Ordering with Interactive Decoding
Bai, G., He, S., Liu, K., & Zhao, J. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3561510 · Cited by 2
“Recently, non-autoregressive sequence generation models such as [20] appear, which is not limit to a directional decoding order.”
BackgroundDiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translation
Norouzi, S., Hosseinzadeh, R., Perez, F., & Volkovs, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.542 · Cited by 4
Extrapolating Multilingual Understanding Models as Multilingual Generators
Wu, B., Yuan, F., Zhao, H., Li, L., & Xu, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1031 · Cited by 2
KAT5: Knowledge-Aware Transfer Learning with a Text-to-Text Transfer Transformer
Sohrab, M. G., & Miwa, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70378-2_10 · Cited by 4
Syu, S.-S., Xie, J., & Lee, H.-Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3451977 · Cited by 5
Non‐Autoregressive Translation Algorithm Based on LLM Knowledge Distillation in English Corpus
Ju, F., & Wang, W. (2024) · Engineering Reports · doi:10.1002/eng2.13077 · Cited by 4
Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer
Tokarchuk, E., Rosendahl, J., Wang, W., Petrushkov, P., Lancewicki, T., Khadivi, S., & Ney, H. (2021) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2021.iwslt-1.32 · Cited by 2
“…decoding for the NAT, meaning that instead of only one decoding pass, the model relies on the multiple passes, and conditional dependence might be used on the consecutive passes to achieve better performance (Ghazvininejad et al., 2019; Gu et al., 2019; Lee et al., 2018; Stern et al., 2019).”
MethodBioNART: A Biomedical Non-AutoRegressive Transformer for Natural Language Generation
Asada, M., & Miwa, M. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.34 · Cited by 2
“To alleviate this problem, several Non-AutoRegressive (NAR) methods (Lee et al., 2018; Qi et al., 2021) that can simultaneously predict all target tokens have been proposed in the general domain.”
MethodMultiple Tasks Integration: Tagging, Syntactic and Semantic Parsing as a Single Task
Bernard, T. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.66 · Cited by 2
CMVCG: Non-autoregressive Conditional Masked Live Video Comments Generation Model
Zeng, Z., Tu, C., Gao, N., Xue, C., Ma, C., & Shan, Y. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533460 · Cited by 2
“Effort [17] generates tokens by iteratively refining the outputs of their decoder.”
BackgroundCon-NAT: Contrastive Non-autoregressive Neural Machine Translation
Cheng, H., & Zhang, Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.463 · Cited by 2
“…et al., 2019), enhancing the decoder output (Wang et al., 2019; Sun et al., 2019), learning or transforming from autoregressive model (Li et al., 2019; Guo et al., 2020a; Sun and Yang, 2020; Tu et al., 2020; Liu et al., 2020), latent variable-based model (Lee et al., 2018, 2020; Shu et al., 2020).”
MethodNon-Autoregressive Sentence Ordering
Bin, Y., Shi, W., Ji, B., Zhang, J., Ding, Y., & Yang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.277 · Cited by 4
Distilling Translations with Visual AwarenessPreprint
Ive, J., Madhyastha, P., & Specia, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07701 · Cited by 13
“Iterative refinement approaches have also been proposed that start with a draft translation and then predict discrete substitutions based on an attention mechanism (Novak et al., 2016), or using non-autoregressive methods with a focus on speeding up decoding (Lee et al., 2018).”
MethodFastLR: Non-Autoregressive Lipreading Model with Integrate-and-FirePreprint
Liu, J., Ren, Y. C., Zhao, Z., Zhang, C., Huai, B., & Yuan, N. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02516 · Cited by 2
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review MechanismPreprint
Xie, P., Cui, Z., Chen, X., Hu, X., Cui, J., & Wang, B. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2010.09194 · Cited by 10
“Against them, a promising research line is to perform refinement on the generated result within several iterations (Lee et al., 2018; Ghazvininejad et al., 2019).”
BackgroundMethodInfluentialA Study on the Autoregressive and non-Autoregressive Multi-label LearningPreprint
Barezi, E. J., Calixto, I., Cho, K., & Fung, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01711 · Cited by 2
“In this work, we follow a recent trend in seq2seq learning (Gu et al. 2017; Lee, Mansimov, and Cho 2018; Kaiser et al. 2018) and propose a non-autoregressive latent variable model for extreme classification.”
MethodInfluentialSelf-citationLearning to Generate Code Sketches.Preprint
Guo, D., Svyatkovskiy, A., Yin, J., Duan, N., Brockschmidt, M., & Allamanis, M. (2021) · arXiv (Cornell University) · Cited by 3
Proceedings of the First Workshop on Automatic Simultaneous Translation
(2020) · doi:10.18653/v1/2020.autosimtrans-1 · Cited by 2
“…log P (ỹ(z)nad|x(z), θenc, θnad)} (8) where ỹnad is the reference of NAT, which can be obtained from standard NMT model via sequencelevel knowledge distillation (Gu et al., 2017; Lee et al., 2018; Wang et al., 2019), and λ is a hyperparameter used to balance the preference between the two terms.”
InfluentialA Pyramid Semi-Autoregressive Transformer with Rich Semantics for Sign Language Production
Cui, Z., Chen, Z., Li, Z., & Wang, Z. (2022) · Sensors · doi:10.3390/s22249606 · Cited by 2
“Therefore, non-autoregressive (NAT) transformers [26,27] were proposed, which remove the AT connection directly and generate all target words in parallel.”
BackgroundNAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders
Soares, L. B., Gillick, D., Cole, J. R., & Kwiatkowski, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.156 · Cited by 5
BaSFormer: A Balanced Sparsity Regularized Attention Network for Transformer
Jiang, S., Chen, Q., Xiang, Y., Pan, Y., & Wu, X. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3374062 · Cited by 2
“The WMT2014 English-German (EN-DE) preprocessed dataset dataset [6] and WMT2016 English-Romanian (EN-RO) preprocessed dataset [78] were chosen as the benchmarks for comparing the proposed BaSFormer with other SOTA baselines.”
MethodImprove Shallow Decoder Based Transformer with Structured Expert Prediction
Wang, Z., & Han, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72350-6_15 · Cited by 1
Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization
Chen, X., Duan, S., & Liu, G. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9437-9_19 · Cited by 2
Chen, X., Duan, S., & Liu, G. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128828 · Cited by 1
Widening the bottleneck of lexical choice for non-autoregressive translation
Ding, L., Wang, L., Liu, S., Luo, W., & Zhang, K. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2024.101765 · Cited by 2
Liu, Y., Yu, H. Z., Li, H., Zhang, X., Han, X., & Liu, H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890521 · Cited by 1
“The compared models include AIC(bw=1/3); UAIC [24]; DeeCap [13]; PNAIC [14], STAIC [15], SAIC [16]; MNIC [12], MIR [25], IBM [26], FNIC [17], EENAIC [23] and CMAL [18].”
Wang, T., Xie, X., Jin, Z., Geng, M., Deng, J., Li, Z., Hu, S., Hu, S., Li, G., Cui, M., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3633090 · Cited by 1
Iterative Latent Refinement for Robust Non-Autoregressive Sign Language Production
Kiziltepe, T., Taşyürek, S. M., & Keleş, H. Y. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00512 · Cited by 1
“This framework has demonstrated notable success across three principal domains: (1) sequence generation through iterative decoding architectures [5, 17, 18], (2) image synthesis via stochastic differential equation-based diffusion processes [1, 8, 9, 31, 36], and (3) instance segmentation employing…”
Retrieval-Augmented Diffusion Language Model for Generative Commonsense Reasoning
FANG, Y., Yu, H.-T., Joho, H., & Fujita, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0369-7_22 · Cited by 3
Learning and Analyzing Generation Order for Undirected Sequence Models
Jiang, Y., & Bansal, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.298 · Cited by 1
“Lee et al. (2018) proposed a model that replaces the tokens at all positions and keeps refining the previous outputs for multiple iteration.”
Uncertainty-Aware Sequence Labeling
Ye, J., Zhou, X., Zheng, X., Gui, T., & Zhang, Q. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3138680 · Cited by 1
“Refinement approaches have given accuracy improvement on many NLP tasks including machine translation [42], [43], [44], [45], parsing [46], [47], reading comprehension [48], question answering [49], story generation [50] and named entity recognition [51], [52], [16].”
BackgroundJapanese Writing Support System with Fast Grammatical Error Correction
Homma, H., & Komachi, M. (2021) · Transactions of the Japanese Society for Artificial Intelligence · doi:10.1527/tjsai.37-1_b-l22 · Cited by 1
Lim, Y., Park, E.-J., Song, H.-J., & Park, S.-B. (2022) · IEEE Access · doi:10.1109/access.2022.3169419 · Cited by 4
“There have also been some approaches to use the output of an encoder [4], [6], [7].”
BackgroundMethodInfluentialImproving Non-Autoregressive Machine Translation Using Sentence-Level Semantic Agreement
Wang, S., Huang, H., & Shi, S. (2022) · Applied Sciences · doi:10.3390/app12105003 · Cited by 2
“From modeling the input by latent variables [4,22,28] in the early stage to capturing the target dependency [7,15,24,25], the performance of the NAT model has been greatly improved.”
BackgroundMethodInfluential
Metadata
- Authors
- Lee, J., Mansimov, E., & Cho, K.
- Affiliations
- New York University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1173-1182
- OpenAlex
- W2963434219
- Cited by
- 386
- References
- 46
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 24 · 2023: 55 · 2022: 57 · 2021: 88 · 2020: 86 · 2019: 43 · 2018: 5
- FWCI
- 22.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Samsung Advanced Institute of Technology
- Tencent
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- iterative refinement, deterministic decoding, latent variable models, denoising autoencoder, machine translation, image captioning, decoding speed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.