The DOI matches its Crossref record; 4 of the 5 references it cites were found in a registry or an index, and 1 are not checked yet.
DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-off
Open accessAbstract
This paper introduces DrDiff, a novel framework for long-text generation that overcomes the efficiency-quality trade-off through three core technologies.First, we design a dynamic expert scheduling mechanism that intelligently allocates computational resources during the diffusion process based on text complexity, enabling more efficient handling of text generation tasks of varying difficulty.Second, we introduce a Hierarchical Sparse Attention (HSA) mechanism that adaptively adjusts attention patterns according to a variety of input lengths, reducing computational complexity from O(n 2 ) to O(n) while maintaining model performance.Finally, we propose a Semantic Anchor States (SAS) module that combines with DPM-solver++ to reduce diffusion steps, significantly improving generation speed.Comprehensive experiments on various long-text generation benchmarks demonstrate the superiority of our DrDiff over the existing SOTA methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Processing longer inputs with LLMs results in rapidly increasing API costs, especially when pricing models are expensive.
“At the same time, processing longer inputs with LLMs results in rapidly increasing API costs, especially when pricing models are expensive [17], [18].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [17]
The Introduction says long sequences bring quadratically increasing computational complexity, supporting the general point that longer inputs require more resources. It does not discuss API charges or expensive pricing models; searches for API costs and token pricing found no such discussion. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 47
- Saleh Ashkboos, Iman Mirzadeh, Keivan Alizadeh, Mo- hammad Hossein Sekhavat, Moin Nabi, Mehrdad Farajtabar, and Fartash Faghri. 2024. Computational bottlenecks of training small-scale large language models. Preprint, arXiv:2410.19456. · arXiv
- Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024b. Longwriter: Unleashing 10,000+ word generation from long context llms. Preprint, arXiv:2408.07055. · arXiv
- Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024c. Longwriter: Unleashing 10,000+ word generation from long context llms. Preprint, arXiv:2408.07055. · arXiv
- Philipp Becker, Abhinav Mehrotra, Ruchika Chavhan, Malcolm Chadwick, Luca Morreale, Mehdi Noroozi, Alberto Gil Ramos, and Sourav Bhattacharya. 2025. Edit: Efficient diffusion transformers with linear com- pressed attention. Preprint, arXiv:2503.16726. · arXiv
- Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. Preprint, arXiv:2004.05150. · arXiv
- Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley. 2023. Unlimiformer: long- range transformers with unlimited length input. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA.
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b. Language models are few-shot learn- ers. Preprint, arXiv:2005.14165. · arXiv
- Jon Chamberlain, Massimo Poesio, and Udo Kr- uschwitz. 2008. Addressing the resource bottleneck to create large-scale annotated texts. In Semantics in Text Processing. STEP 2008 Conference Proceedings, pages 375-380. College Publications.
- Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. 2023. Walking down the mem- ory maze: Beyond context limit through interactive reading. Preprint, arXiv:2310.05029. · arXiv
- Yafeng Chen, Zhihao Lan, Jensen Li, and Jie Zhu. 2021. Topologically protected second harmonic genera- tion via doubly resonant high-order photonic modes. Physical Review B, 104(15). · DOI
- Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019b. Generating long sequences with sparse transformers. Preprint, arXiv:1904.10509. · arXiv
- Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019c. Generating long sequences with sparse transformers. Preprint, arXiv:1904.10509. · arXiv
- Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. Preprint, arXiv:2003.10555. · arXiv
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. Preprint, arXiv:1901.02860. · arXiv
- DeepSeek-AI. 2025. Deepseek-r1: Incentivizing rea- soning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. · arXiv
- Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Hayden Kwok-Hay So, Ting Cao, Fan Yang, and Mao Yang. 2024. Seerattention: Learning intrin- sic sparse attention in your llms. arXiv preprint arXiv:2410.13276. · arXiv
- Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. Diffuseq: Sequence to sequence text generation with diffusion models. Preprint, arXiv:2210.08933. · arXiv
- Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wen- biao Ding, and Minlie Huang. 2021. Long text gener- ation by modeling sentence-level and discourse-level coherence. Preprint, arXiv:2105.08963. · arXiv
- Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. Preprint, arXiv:2006.11239. · arXiv
- Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In ACL.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Inter- national Conference on Learning Representations.
- Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. Preprint, arXiv:2301.13298. · arXiv
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. 2020. Albert: A lite bert for self-supervised learning of language representations. Preprint, arXiv:1909.11942. · arXiv
- Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. 2023. Text generation with diffusion language mod- els: A pre-training approach with continuous para- graph denoise. Preprint, arXiv:2212.11685. · arXiv
- Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024. Longgenbench: Long-context genera- tion benchmark. Preprint, arXiv:2410.04199. · arXiv
- LLAMA2. 2023. Llama 2: Open foundation and fine- tuned chat models. Preprint, arXiv:2307.09288. · arXiv
- LLAMA3. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
- Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongx- uan Li, and Jun Zhu. 2025. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, pages 1-22.
- Mohaddeseh Mirbeygi and Hamid Beigy. 2025. Prompt guided diffusion for controllable text generation. In Proceedings of the Tenth Workshop on Noisy and User-generated Text, pages 78-84, Albuquerque, New Mexico, USA. Association for Computational Linguistics.
- Alex Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. Preprint, arXiv:2102.09672. · arXiv
- OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. · arXiv
- Qwen. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text trans- former. Preprint, arXiv:1910.10683. · arXiv
- Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. Computing Research Repository, arXiv:1503.06733. Version · arXiv
- Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer. Preprint, arXiv:1701.06538. · arXiv
- Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: query- aware sparsity for efficient long-context llm inference. ICML'24. JMLR.org.
- Zecheng Tang, Pinzheng Wang, Keyan Zhou, Juntao Li, Ziqiang Cao, and Min Zhang. 2023. Can diffu- sion model achieve better performance in text gener- ation? bridging the gap between training and infer- ence! Preprint, arXiv:2305.04465. · arXiv
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022a. Efficient transformers: A survey. Preprint, arXiv:2009.06732. · arXiv
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022b. Efficient transformers: A survey. Preprint, arXiv:2009.06732. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762. · arXiv
- Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics.
- Tong Wu, Junzhe Shen, Zixia Jia, Yuxuan Wang, and Zilong Zheng. 2025. From hours to minutes: Loss- less acceleration of ultra long sequence generation up to 100k tokens. Preprint, arXiv:2502.18890. · arXiv
- Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. 2025. Energy-based diffusion language mod- els for text generation. Preprint, arXiv:2410.21357. · arXiv
- Zhiyu Yang, Wenhan Xiong, Mo Yu, and William Yang Wang. 2018. Hotpotqa: A dataset for diverse, ex- plainable multi-hop question answering. In EMNLP.
- Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2023. Seqdiffuseq: Text dif- fusion with encoder-decoder transformers. Preprint, arXiv:2212.10325. · arXiv
- Jusheng Zhang, Yijia Fan, Kaitong Cai, and Keze Wang. 2025a. Kolmogorov-arnold fourier networks. Preprint, arXiv:2502.06018. · arXiv
- Jusheng Zhang, Zimeng Huang, Yijia Fan, Ningyuan Liu, Mingyan Li, Zhuojie Yang, Jiawei Yao, Jian Wang, and Keze Wang. 2025b. Kabb: Knowledge-aware bayesian bandits for dynamic ex- pert coordination in multi-agent systems. Preprint, arXiv:2502.07350. · arXiv
Cited by
Sorted by citations
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7
Metadata
- Authors
- Zhang, J., Fan, Y., Cai, K., Huang, Z., Sun, X., Wang, J., Tang, C., & Wang, K.
- Affiliations
- Snap (United States) (US)
- Alibaba Group (China) (CN)
- Venue
- Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
- Published
- 2025
- Pages
- 9331-9351
- OpenAlex
- W4416036775
- Cited by
- 1
- References
- 0
- Citations by year
- 2025: 1
- FWCI
- 0.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Natural Science Foundation of China (62276283)
- Sun Yat-sen University
- China Meteorological Administration
- Fundamental Research Funds for the Central Universities
- Basic and Applied Basic Research Foundation of Guangdong Province (2023A1515012985)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- long-form text generation, diffusion models, dynamic routing, text complexity, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
