Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Open accessThe DOI matches its Crossref record; 41 of the 46 references it cites were found in a registry or an index.
Abstract
We propose a conditional non-autoregressive neural sequence model based on iterative refinement.The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task.We extensively evaluate the proposed model on machine translation (En$De and En$Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“It reduces memory costs by avoiding auxiliary models but relies on fixed-size drafting blocks. Verification is strict, and confidence awareness is minimal or implicit.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“While recent advances in speculative decoding and non-autoregressive generation have made substantial strides toward accelerating sequence generation, most existing approaches are limited by static drafting policies, rigid verification strategies, or architectural dependencies that do not generalize well across tasks or model configurations.”
From Confidence-Modulated Speculative Decoding for Large Language Models (2025) · cites this paper as [13]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5×
“Each image is pre-transformed into a set of 49 512-dimensional feature vectors, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al.”
MethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“Training and Inference We use Adam (Kingma and Ba, 2014) and use L = 3 in Eq.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We extract 49 512-dimensional feature vectors for each image, using a ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009).”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 6×
“…compare the proposed non-autoregressive model against the autoregressive counterpart both in terms of generation quality, measured in terms of BLEU (Papineni et al., 2002), and generation efficiency, measured in terms of (source) tokens and images per second for translation and image captioning,…”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“When this is augmented with an extra conditioning variable X , it becomes conditional sequence modeling log p(Y |X) which serves as a basis on which many recent advances in, for instance, machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Kalchbrenner & Blunsom, 2013) and speech recognition (Chorowski…”
BackgroundMethodSelf-citationDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2×
“…Gu et al. (2017), in the context of machine translation, and Oord et al. (2017), in the context of speech generation, have recently discovered that it is important to use knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016) to successfully train a non-autoregressive sequence model.”
Background
Research works printed but not linked: 3
- Andrej Karpathy and Fei-Fei Li. 2015. Deep visualsemantic alignments for generating image descriptions. In CVPR .
- Yoon Kim and Alexander M Rush. 2016. Sequencelevel knowledge distillation. arXiv preprint arXiv:1606.07947 . · arXiv
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL .
Cited by
Items 301–376 of 376
Synchronous Refinement for Neural Machine Translation
Chen, K., Utiyama, M., Sumita, E., Wang, R., & Zhang, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.235 · Cited by 1
“Lee et al. (2018) designed an iterative inference strategy to minimize the generation latency.”
BackgroundA Context-Enhanced Transformer with Abbr-Recover Policy for Chinese Abbreviation Prediction
Cao, K., Yang, D., Liu, J., Liang, J., Xiao, Y., Feng, W., Wu, B., & Lu, Q. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557074 · Cited by 2
“And[16]proposed toaddaniterativerefinementstageinthedecodertosignificantly speedupdecoding.”
Non-Autoregressive Speech Recognition with Error Correction Module
Qian, Y., Zhuang, X., Zhang, Z., Zhou, L., Lin, X., & Wang, M. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980031 · Cited by 1
“Non-autoregressive models were first proposed in text translation [5].”
MethodBetter Localness for Non-Autoregressive Transformer
Wang, S., Huang, H., & Shi, S. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3587266 · Cited by 1
“Since the proposal of the NAT model, a series of works have been proposed to improve the performance of the NAT model [9, 10, 16].”
MethodImproving Autoregressive Grammatical Error Correction with Non-autoregressive Models
Cao, H., Cao, Z., Hu, C., Hou, B., Xiao, T., & Zhu, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.760 · Cited by 2
“Therefore, a natural idea is to improve the model performance by exploiting the global information, which can be captured by the non-autoregressive (NAR) (Gu et al., 2018; Lee et al., 2018) model.”
BackgroundHybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation
Wang, Q., Hu, X., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.367 · Cited by 1
Twist Decoding: Diverse Generators Guide Each Other
Kasai, J., Sakaguchi, K., Bras, R. J. L., Peng, H., Lu, X., Radev, D., Choi, Y., & Smith, N. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.326 · Cited by 4
Non-Autoregressive Models for Fast Sequence Generation
Feng, Y., & Shao, C. (2022) · doi:10.18653/v1/2022.emnlp-tutorials.6 · Cited by 1
“…more expressive NAR models (Wang et al., 2017; Libovický and Helcl, 2018; Sun et al., 2019; Huang et al., 2022), improved decoding approaches (Lee et al., 2018; Ghazvinine-jad et al., 2019; Gu et al., 2019; Ran et al., 2020; Saharia et al., 2020; Deng and Rush, 2020; Geng et al., 2021; Stern…”
BackgroundLin, Y. (2023) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa60987.2023.10302580 · Cited by 1
“While effective at capturing interdependencies in the output space [12, 13], AR models face issues such as training-inference discrepancies [14, 15], error accumulation [16], and high inference latency [17, 18].”
BackgroundTreepiece: Faster Semantic Parsing via Tree Tokenization
Wang, S., Shrivastava, A., & Livshits, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.740 · Cited by 2
Alleviating repetitive tokens in non-autoregressive machine translation with unlikelihood training
Wang, S., Shi, S., & Huang, H. (2024) · Soft Computing · doi:10.1007/s00500-023-09490-1 · Cited by 1
“NAT-IR (Lee et al. 2018): NAT-IR is the first iterative refinement non-autoregressive model.”
InfluentialLV-CTC: Non-Autoregressive ASR With CTC and Latent Variable Models
Fujita, Y., Watanabe, S., Chang, X., & Maekaku, T. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389698 · Cited by 2
“None of the existing NAR E2E ASR using CTC [10, 12–15, 17, 25, 32] assumes such a dependency of CTC alignment in this latent space.”
MethodTransformer-based Double-token Bidirectional Autoregressive Decoding in Neural Machine Translation
Imamura, K., & Sumita, E. (2020) · Workshop on Asian Translation (WAT) · doi:10.18653/v1/2020.wat-1.3 · Cited by 1
“Non-autoregressive decoding generates all to-kens simultaneously, utilizing the parallelism of Transformer ( Gu et al., 2017; Lee et al., 2018; Ghazvininejad et al., 2019).”
MethodInfluentialHI-CMLM: Improve CMLM with Hybrid Decoder Input
Wang, M., Guo, J., Wang, Y., Chen, Y., Su, C., Wei, D., Zhang, M., Tao, S., & Yang, H. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.16 · Cited by 1
“…have witnessed impressive advances in non-autoregressive models, such as fully-NAT and its variants (Gu et al., 2018; Guo et al., 2019; Wang et al., 2019), insertion-based models (Stern et al., 2019; Gu et al., 2019) and iterative refinement models (Lee et al., 2018; Ghazvininejad et al., 2019).”
MethodBetter Intermediates Improve CTC Inference
Komatsu, T., Fujita, Y., Lee, J., Lee, L. J., Watanabe, S., & Kida, Y. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11276 · Cited by 9
Self-citationUncertainty-aware non-autoregressive neural machine translation
Liu, C., & Yu, J. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2022.101444 · Cited by 3
Non-Autoregressive Document-Level Machine Translation
Bao, G., Teng, Z., Zhou, H., Yan, J., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.986 · Cited by 4
“…models achieve significant acceleration on the inference, with even better translation performance, when compared to auto-regressive translation (AT) models on sentence-level MT (Gu et al., 2018, 2019; Stern et al., 2019; Ma et al., 2019; Lee et al., 2020; Huang et al., 2022; Shao and Feng, 2022).”
MethodProbing Contextualized Sentence Representations with Visual AwarenessPreprint
Zhang, Z., Wang, R., Chen, K., Utiyama, M., Sumita, E., & Zhao, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02971 · Cited by 3
“(Lee et al., 2018) 32”
Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoderPreprint
Inaguma, H., Higuchi, Y., Duh, K., Kawahara, T., & Watanabe, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13047 · Cited by 1
“However, iterative refinement methods generally achieve better quality than pure NAR methods at the cost of speed because of multiple forward passes [20, 23, 39].”
MethodNon-Autoregressive Grammatical Error Correction Toward a Writing Support System
Homma, H., & Komachi, M. (2020) · Workshop on Natural Language Processing Techniques for Educational Applications (NLP-TEA) · doi:10.18653/v1/2020.nlptea-1.1 · Cited by 1
“To solve this problem, in recent studies, iteratively refining the output (Lee et al., 2018; Gu et al., 2019) and partially autoregressively outputting the sentence divided into segments (Ran et al.”
MethodGuiding Teacher Forcing with Seer Forcing for Neural Machine TranslationPreprint
Feng, Y., Gu, S., Guo, D., Yang, Z., & Shao, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06751 · Cited by 1
“English→Romanian We used the preprocessed version of WMT16 En-Ro dataset released by Lee et al. (2018) which includes 0.6M sentence pairs.”
MethodGoal-driven text descriptions for imagesPreprint
Luo, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12575 · Cited by 4
Confidence-Modulated Speculative Decoding for Large Language Models
Sen, J., Dasgupta, S., & Waghela, H. (2025) · 2025 IEEE 6th India Council International Subsections Conference (INDISCON) · doi:10.1109/indiscon66021.2025.11254640
Semantic Role Labeling with Iterative Structure RefinementPreprint
Lyu, C., Cohen, S. B., & Titov, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03285
“Refinement strategies have been successful in machine translation (Lee et al., 2018; Novak et al., 2017; Xia et al., 2017; Hassan et al., 2018), but their effectiveness in other NLP tasks is yet to be demonstrated.”
BackgroundMethodInfluentialCapturing Argument Interaction in Semantic Role Labeling with Capsule NetworksPreprint
Chen, X., Lyu, C., & Titov, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03136
“Unlike standard refinement methods (Belanger et al., 2017; Lee et al., 2018) which sum losses across all refinement iterations, our loss is only based on the prediction made at the last iteration.”
MethodNeural Machine Translation Based on Improved Actor-Critic Method
Guo, Z., Hou, H., Wu, N., & Sun, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61616-8_28
Word-Level Error Correction in Non-autoregressive Neural Machine Translation
Guo, Z., Hou, H., Wu, N., & Sun, S. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-63820-7_83
Fast Interleaved Bidirectional Sequence Generation
Zhang, B., Titov, I., & Sennrich, R. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-193900
Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic SurveyPreprint
Chandu, K. R., & Black, A. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07279 · Cited by 5
“Following this work, Lee et al. (2018) use iterative refinement by denoising the latent variables at each of the refinement steps.”
MethodYang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01893
“English-to-Romanian We used the preprocessed version of WMT’16 4 Romanian-English dataset released by Lee et al. (2018) which includes 0.6M sentence pairs. news-dev2016 and news-test2016 are used for development and testing respectively, both of which contain 1999 sentences.”
MethodStructured Refinement for Sequential Labeling
Wang, Y., Shindo, H., Matsumoto, Y., & Watanabe, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.164
“In existing work, this method was successfully applied to various tasks, e.g., text classification (Yu et al., 2017), sequential labeling (Cui and Zhang, 2019; Lyu et al., 2019), machine translation (Lee et al., 2018), and question answering (Nema et al., 2019).”
MethodHigh-Speed and High-Quality Text-to-Lip Generation.Preprint
Liu, J., Zhu, Z., Ren, Y., & Zhao, Z. (2021) · arXiv (Cornell University)
Towards controllable neural generation of arguments
Hua, X. (2021) · Digital Repository Service (Northeastern University) · doi:10.17760/d20412319
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video CaptioningPreprint
Liu, F., Ren, X., Wu, X., Yang, B., Ge, S., Zou, Y., & Sun, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02359 · Cited by 11
“Our approach is adapted from the non-autoregressive decoding models (Lee et al., 2018; Ghazvininejad et al., 2019), which is based on the Transformer decoder (TFM) (Vaswani et al., 2017).”
BackgroundMethodInfluentialLearning Energy-Based Approximate Inference Networks for Structured Applications in NLPPreprint
Tu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12522 · Cited by 1
“Iterative Refinement [Lee et al., 2018] - 25.73 † NAT with Fertility [Gu et al., 2018] - 29.06 † CTC [Libovický and Helcl, 2018] - 24.71 † FlowSeq [Ma et al., 2019] 27.55 † 30.44 † CMLM [Ghazvininejad et al., 2019] 28.25 28.20 † Bag-of-ngrams-based loss [Shao et al., 2020] - 29.29 † AXE CMLM [Ghazvininejad et al., 2020…”
BackgroundInfluentialSpan Pointer Networks for Non-Autoregressive Task-Oriented Semantic ParsingPreprint
Shrivastava, A., Chuang, P., Babu, A., Desai, S., Arora, A., Zotov, A., & Aly, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07275 · Cited by 5
“advantages and disadvantages, primarily tradingoff latency and quality: while non-autoregressive models are optimized for inference and suitable in production settings, they are often qualitatively worse than autoregressive models due to strong independence assumptions during generation (Gu et al., 2018; Lee et al., 2…”
BackgroundSequence-to-Lattice Models for Fast Translation
Deng, Y., & Rush, A. M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.318
“Gu et al. (2017) proposed the task of nonautoregressive machine translation, and since then there have been many followup works (Lee et al., 2018, inter alia).”
BackgroundMethodEnhanced encoder for non-autoregressive machine translation
Wang, S., Shi, S., & Huang, H. (2021) · Machine Translation · doi:10.1007/s10590-021-09285-x · Cited by 1
“Lee et al. (2018) introduced an iterative decoding method for NAT models, which significantly improved the performance of NAT models.”
BackgroundMethodResultInfluentialFastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech RecognitionPreprint
Leng, Y., Tan, X., Wang, R., Zhu, L., Xu, J., Liu, W., Liu, L., Qin, T., Li, X., Lin, E. C., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14420 · Cited by 14
“Taking the recognized sentence from the ASR system as source and the ground-truth sentence as target, ASR correction can be formulated as a sequence-to-sequence problem and modeled with autoregressive (Mani et al., 2020; Liao et al., 2020) or non-autoregressive (Leng et al., 2021) generation.”
MethodInfluentialJoint source–target encoding with pervasive attention
Elbayad, M., Besacier, L., & Verbeek, J. J. (2021) · Machine Translation · doi:10.1007/s10590-021-09289-7
“We use the data provided by Lee et al. (2018) where we train on 608K pairs, develop on newsdev-2016 (1999 pairs) and report test results on newstest-2016 (1999 pairs).”
MethodMR-P: A Parallel Decoding Algorithm for Iterative Refinement Non-Autoregressive Translation
Cheng, H., & Zhang, Z. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.25
“An approach for alleviating this problem is to iteratively refine the model output and make a trade-off between inference speed and model performance (Lee et al., 2018; Ghazvininejad et al., 2019; Kasai et al., 2020).”
MethodA Survey of Machine Translation Technologies for Machine Translation on Patent Documents
Imamura, K., Echizen‐ya, H., Ehara, T., Goto, I., Sudoh, K., Sonoh, S., Tsunakawa, T., Nakazawa, T., Ninomiya, T., & Wang, X. (2022) · Journal of Natural Language Processing · doi:10.5715/jnlp.29.925 · Cited by 1
Analysis on the number of layers in the transformer-based model for neural machine translation
Li, D., & Luo, Z. (2022) · International Conference on Computer Science and Communication Technology (ICCSCT) · doi:10.1117/12.2661982
Non-Autoregressive Machine Translation with a Novel Masked Language Model
Li, K., Li, J., & Wangjun, W. (2022) · International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · doi:10.1109/iccwamtip56608.2022.10016511
SUNMASK: Mask Enhanced Control in Step Unrolled Denoising Autoencoders
Kastner, K., Cooijmans, T., Wu, Y., & Courville, A. C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-29956-8_10
Deep Equilibrium Non-Autoregressive Sequence Learning
Zheng, Z., Zhou, Y., & Zhou, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.747 · Cited by 4
“As for NAR models that conduct explicit iterative refinement on discrete tokens (Lee et al., 2018; Ghazvininejad et al., 2019), the state z t is defined as a sequence of one-hot vectors corresponding to each of the intermediate predicted tokens.”
BackgroundMethodInfluentialDifference-Aware Iterative Reasoning Network for Key Relation Detection
Zhao, B., Chen, W., Hu, B., Xie, H., & Mao, Z. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00055 · Cited by 5
Attentive Multi-Layer Perceptron for Non-autoregressive Generation
Jiang, S., Zhang, J., Feng, J., Lin, Z., & Kong, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43415-0_36 · Cited by 1
“To mitigate the quality degradation, many researchers devote to improve the model performance with iterative decoding [29,16,18,20,22], curriculum learning [19,34,42,43,4], latent variable modeling [38,45,3,4], imitation learning [31,56] and learning objec-tive [48,15,32,12].”
BackgroundControlling keywords and their positions in text generation
Sasazawa, Y., Morishita, T., Ozaki, H., Imaichi, O., & Sogawa, Y. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.29 · Cited by 7
Isotropy-Enhanced Conditional Masked Language Models
Guo, P., Xiao, Y., Li, J., Ji, Y., & Zhang, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.555
Improving Iteration-based Non-Autoregressive Language Model With Time Step Awareness
Jiang, H., Cai, G., & Tang, X. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads60453.2023.00385
“As a compromise, iteration-based non-autoregressive (INAR) models [4], [5] are proposed to find a better trade-off between generation speed and quality.”
MethodFilter-GLAT: Filter Glanced Decoder Output for Non-autoregressive Transformer
Wang, Z., Zheng, H., & Wang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7232-2_5
LayerGLAT: A Flexible Non-autoregressive Transformer for Single-Pass and Multi-pass Prediction
Li, S., Unanue, I. J., & Piccardi, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70344-7_14
Learning to Deliberate: Multi-pass Decoding for Document-Grounded Conversations
Qiu, J., & Yang, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650055
“[8] proposed to progressively improve the generated translation results through multiple iterations.”
BackgroundSummarizing Like Human: Edit-Based Text Summarization with Keywords
Liang, Y., Guo, J., Zhu, Y., & Xu, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72350-6_23 · Cited by 1
Correcting translation for non-autoregressive transformer
Wang, S., Huang, H., Shi, S., Li, D., & Guo, D. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112488
Hypformer: A Fast Hypothesis-Driven Rescoring Speech Recognition Framework
Zhuang, X., Qian, Y., & Wang, M. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3516700
Dual Decoder for Fast Inference in Natural Language Generation
Wang, W., Wang, H., Wang, Z., Li, S., & Yu, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889713
CPS-Net: Segmentation Network Based on Feature Fusion and Siamese
Shi, K., Xu, J., & Mao, J. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913030
“Then, linear transformation is performed on the input through the above formula to obtain matrices A, K, and V [6, 7, 8].”
Improving Fully Non-Autoregressive Translation with Pre-Trained Language Models
Fu, Z., & Wang, S. (2024) · doi:10.1145/3722237.3722267
“There are several researchers working to improve the performance of NAT model, including latent variables models [ 4 - 7 ] , iterative refinement models [ 8 - 9 ] , loss functions [ 10 - 16 ] , and training methods [ 17 - 19 ] .”
InfluentialYao, W., & Zhou, W. (2025) · Engineering Reports · doi:10.1002/eng2.70287
“The WMT’16 English-Romanian (En-Ro) [31] dataset employs a preprocessing framework jointly developed by [32, 33], implementing a cross-lingual Byte Pair Encoding (BPE) fusion strategy to build a vocabulary containing 40,000 shared subword units, where newsdev2016 and newstest2016 are designated for…”
Predicting Token Length for Large Language Models with Multi-Branch Headers
Li, C., Liu, G., Liao, R., Xuan, Y., Yu, L., Zhou, L., Ding, N., & Zhang, J. (2025) · International Conference on Computer, Control and Robotics (ICCCR) · doi:10.1109/icccr65461.2025.11072648
“Most previous token length prediction models such as [8], [9], and [10], are built on non-auto-regressive (NAR) models for machine translation and image caption tasks, in which actual sequence length is short and the predictions are largely based on the patterns of the high-quality datasets.”
Translating Images to Road Network: A Sequence-to-Sequence Perspective
Lu, J., Nie, M., Zhang, B., Peng, R., Cai, X., Xu, H., Wen, F., Zhang, W., & Zhang, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3612940 · Cited by 3
“Going beyond SAR-RNTR, we employ a masked training technique on SAR-RNTR to mimic the remaining auto-regressive dependency through iterative prediction [23].”
Sun, S., & Zhuo, C. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3621408 · Cited by 1
“Building upon the strengths of MDM, NAR decoders leverage this technique to capture complex output dependencies while enabling efficient parallel decoding for fast inference [41], [42].”
Shared Latent Space by Both Languages for Non-Autoregressive Neural Machine Translation
Heo, D., & Choi, H. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228007
“However, the increased translation speed often comes at the cost of translation quality due to the lack of dependencies between target words, leading to the ‘mul-timodality problem’ [3].”
Non–autoregressive Machine Translation with Target Language Components in English Corpus
Huang, B., Mao, M., & Wang, Y. (2025) · Journal of Information & Knowledge Management · doi:10.1142/s0219649225501205
Li, R., Xu, S., Lin, P., Feng, F., & Ma, Z. (2026) · Multimedia Systems · doi:10.1007/s00530-025-02179-5
Adaptive Masking Over Masking for Sequence Generation
Xiao, Y., You, W., Wu, L., Li, J., & Zhang, M. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3633048
“Thus, iterative non-autoregressive (NAR) models [14], [20], [39],[40],[70]areproposedtoachieveabettertrade-offbetween theinferencespeedupandgenerationquality.”
Rethinking the Design of Sequence-to-Sequence Models for Efficient Machine Translation
Elbayad, M. (2020) · theses.fr (ABES) · doi:10.70675/97c82c7ez86c5z4bdbza801z12232614c776 · Cited by 2
Deep learning based trajectory prediction for autonomous vehicles
Messaoud, K. (2021) · theses.fr (ABES) · doi:10.70675/938e399az41f4z447czb17ezca64c38bcb7b · Cited by 3
Robustness Analysis of Classifiers Against Out-of-Distribution and Adversarial Inputs
Chali, S. (2024) · theses.fr (ABES) · doi:10.70675/045f4849z9f1ez4973z8c70z8d4cedc8a05d
Sun, Y., Cheng, H., Zheng, S., Yu, H., & Zou, H. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00754-y
“NAR models were first introduced in neural machine translation by Lee et al. (2018), who proposed an iterative refinement mechanism to mitigate quality degradation caused by parallel prediction.”
Dynamic token halting for efficient abstractive summarization with importance-aware regularization
Huang, H., Bai, Y., Gao, Y., & Liao, M. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116105
Improvement of Speech-Paraformer Large ASR for Industrial Voice Control in High-Noise Environments
Mariya, V., Sun, H., & Jiang, Y. (2026) · International Journal of Computer Science and Information Technology · doi:10.62051/ijcsit.v8n4.07
“Non-Autoregressive Decoding: Unlike conventional autoregressive models that predict tokens sequentially (each step depending on previous outputs), Paraformer employs parallel decoding [16].”
Semantic Flow Refinement for Recognizing Low-Legibility Handwritten Answers
Chandola, S., Ravikiran, M., & Saluja, R. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29788-4_9
Metadata
- Authors
- Lee, J., Mansimov, E., & Cho, K.
- Affiliations
- New York University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1173-1182
- OpenAlex
- W2963434219
- Cited by
- 386
- References
- 46
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 24 · 2023: 55 · 2022: 57 · 2021: 88 · 2020: 86 · 2019: 43 · 2018: 5
- FWCI
- 22.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Samsung Advanced Institute of Technology
- Tencent
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- iterative refinement, deterministic decoding, latent variable models, denoising autoencoder, machine translation, image captioning, decoding speed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.