Reinforcing Pretrained Models for Generating Attractive Text Advertisements
Abstract
We study how pretrained language models can be enhanced by using deep reinforcement learning to generate attractive text advertisements that reach the high quality standard of real-world advertiser mediums. To improve ad attractiveness without hampering user experience, we propose a model-based reinforcement learning framework for text ad generation, which constructs a model for the environment dynamics and avoids large sample complexity. Based on the framework, we develop Masked-Sequence Policy Gradient, a reinforcement learning algorithm that integrates efficiently with pretrained models and explores the action space effectively. Our method has been deployed to production in Microsoft Bing. Automatic offline experiments, human evaluation, and online experiments demonstrate the superior performance of our method.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More recent data-driven methods incorporate user feedback as rewards in a reinforcement learning framework to guide the generation process.
“More recent data-driven methods have emerged, which incorporate user feedback as rewards in the reinforcement learning framework to guide the generation process [259-262].”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [260]
Reinforcement Learning from Human Feedback (RLHF) strategy can be applied to fine-tune models to better capture the user intent information, similar to existing RL-based framework for text ad generation.
“Secondly, Reinforcement Learning from Human Feedback (RLHF) strategy can be applied to fine-tune models to better capture the user intent information, similar to existing RL-based framework [260] for text ad generation.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [260]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · see citing works
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225 · see citing works
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182 · see citing works
MASS: Masked Sequence to Sequence Pre-training for Language GenerationPreprint
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02450 · Cited by 834 · see citing works
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2016) · International Conference on Learning Representations · Cited by 572 · see citing works
Model Ensemble for Click Prediction in Bing Search Ads
Ling, X., Deng, W., Gu, C., Zhou, H., Li, C., & Sun, F. R. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3054192 · Cited by 138 · see citing works
Generative Adversarial User Model for Reinforcement Learning Based Recommendation SystemPreprint
Chen, X., Li, S., Li, H., Jiang, S., Qi, Y., & Song, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.10613 · Cited by 134 · see citing works
Weighted importance sampling for off-policy learning with linear function approximation
Mahmood, A. R., van Hasselt, H. P., & Sutton, R. S. (2014) · neural information processing systems · Cited by 93 · see citing works
Towards a Deep and Unified Understanding of Deep Neural Models in NLP
Guan, C., Wang, X., Zhang, Q., Chen, R., He, D., & Xie, X. (2019) · International Conference on Machine Learning · Cited by 81 · see citing works
Self-citationGuo, H., Yu, J., Liu, Q., Tang, R., & Zhang, Y. (2019) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3298689.3347033 · Cited by 62 · see citing works
Automatic generation of bid phrases for online advertising
Ravi, S., Broder, A., Gabrilovich, E., Josifovski, V., Pandey, S., & Pang, B. (2010) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1718487.1718530 · Cited by 52 · see citing works
Generating Better Search Engine Text Advertisements with Deep Reinforcement Learning
Hughes, J. W., Chang, K.-H., & Zhang, R. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330754 · Cited by 43 · see citing works
Examining Searcher Perceptions of and Interactions with Sponsored Results
Jansen, B. J. J., & Resnick, M. L. (2005) · Cited by 39 · see citing works
Rare Query Expansion Through Generative Adversarial Networks in Search Advertising
Lee, M.-C., Gao, B., & Zhang, R. (2018) · ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) · doi:10.1145/3219819.3219850 · Cited by 34 · see citing works
Automated snippet generation for online advertising
Thomaidou, S., Lourentzou, I., Katsivelis-Perakis, P., & Vazirgiannis, M. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2507876 · Cited by 30 · see citing works
Selection and Generation: Learning towards Multi-Product Advertisement Post Generation
Chan, Z., Zhang, Y., Chen, X., Gao, S., Zhang, Z., Zhao, D., & Yan, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.313 · Cited by 26 · see citing works
Personalized Ad Delivery When Ads Fatigue: An Approximation Algorithm
Abrams, Z., & Vee, E. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-77105-0_57 · Cited by 26 · see citing works
Natural language generation for sponsored-search advertisements
Bartz, K., Barr, C., & Aijaz, A. (2008) · Proceedings of the 9th ACM conference on Electronic commerce · doi:10.1145/1386790.1386792 · Cited by 21 · see citing works
Bid generation for advanced match in sponsored search
Broder, A., Gabrilovich, E., Josifovski, V., Mavromatis, G., & Smola, A. J. (2011) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1935826.1935901 · Cited by 19 · see citing works
Automatic generation of listing ads by reusing promotional texts
Fujita, A., Ikushima, K., Sato, S., Kamite, R., Ishiyama, K., & Tamachi, O. (2010) · Proceedings of the 12th International Conference on Electronic Commerce: Roadmap for the Future of Electronic Business · doi:10.1145/2389376.2389401 · Cited by 15 · see citing works
Flagged first, then by citations
3 of the 30 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367 · see citing works
Self-citationNeural Retrievers are Biased Towards LLM-Generated Content
Dai, S., Zhou, Y., Pang, L., Liu, W., Hu, X., Liu, Y., Zhang, X., Wang, G., & Xu, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671882 · Cited by 35 · see citing works
Wei, P., Yang, X., Liu, S., Wang, L., & Zheng, B. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track · doi:10.18653/v1/2022.naacl-industry.2 · Cited by 9 · see citing works
Creating Ad Campaigns Using Generative AI
Bulut, A., & Arslan, B. (2024) · Springer eBooks · doi:10.1007/978-3-031-46238-2_2 · Cited by 8 · see citing works
Generating ad creatives using deep learning for search advertising
ÇOĞALMIŞ, K. N., & Bulut, A. (2022) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.3911 · Cited by 8 · see citing works
Generating Campaign Ads & Keywords for Programmatic Advertising
Bulut, A., & Mahmoud, A.-R. K. A.-R. (2023) · IEEE Access · doi:10.1109/access.2023.3269505 · Cited by 8 · see citing works
Personalized Chit-Chat Generation for Recommendation Using External Chat Corpora
Chen, C., Wang, X., Yi, X., Wu, F., Xie, X., & Yan, R. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539215 · Cited by 5 · see citing works
Self-citationPLATO-Ad: A Unified Advertisement Text Generation Framework with Multi-Task Prompt Learning
Lei, Z., Zhang, C., Xu, X., Wu, W., Niu, Z.-Y., Wu, H., Wang, H., Yang, Y., & Li, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/v1/2022.emnlp-industry.52 · Cited by 5 · see citing works
Put Your Voice on Stage: Personalized Headline Generation for News Articles
Ao, X., Luo, L., Wang, X., Yang, Z., Chen, J.-H., Qiao, Y., He, Q., & Xie, X. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3629168 · Cited by 5 · see citing works
Self-citationPiscopo, A., Inel, O., Vrijenhoek, S., Millecamp, M., & Balog, K. (2022) · ACM SIGIR Forum · doi:10.1145/3582900.3582915 · Cited by 5 · see citing works
SwiftPruner: Reinforced Evolutionary Pruning for Efficient Ad Relevance
Zhang, L. L., Homma, Y., Wang, Y., Wu, M., Yang, M., Zhang, R., Cao, T., & Shen, W. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557139 · Cited by 3 · see citing works
Fact-Preserved Personalized News Headline Generation
Zhao, Y., Lian, J., & Ao, X. (2023) · Proceedings · doi:10.1109/icdm58522.2023.00197 · Cited by 3 · see citing works
Sahlin, J., Sundell, H., Mbiydzenyuy, G., & Holgersson, J. (2023) · doi:10.1142/9789811269264_0049 · Cited by 2 · see citing works
DeepGen: Diverse Search Ad Generation and Real-Time Customization
Golobokov, K., Chai, J., Dong, V. Y., Gu, M., Chi, B., Cao, J., Yan, Y., & Liu, Y. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.19 · Cited by 2 · see citing works
Self-citationWeb Personalization with Large Language Models: Challenges and Future Trends
Bansal, N., Bala, M., & Sharma, K. (2025) · Algorithms for intelligent systems · doi:10.1007/978-981-97-8669-5_21 · Cited by 2 · see citing works
Neural Recommendation Reasoning with Logic Rules
Yao, J., Wang, X., Lian, J., Yi, X., & Xie, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3742856 · Cited by 3 · see citing works
Self-citationSelf-Supervised Augmentation and Generation for Multi-lingual Text Advertisements at Bing
Kou, X., Zhao, T., Zhang, F., Li, S., & Zhang, Q. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539091
From Clicks to Journeys: Empowering Advertising Text Generation With Social Simulator Guidance
Xie, Y., Wang, L., Nie, W., Li, W., Jin, G., & Liu, A.-A. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3717481
RELATE: A Reinforcement-Learning-Enhanced LLM Framework for Advertising Text Generation
Wang, J., Liu, J., Wu, J., Luo, Z., Chen, Z., Li, C., Han, B., Deng, T., Li, Y., Li, S., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818469
Peng, Y., Liu, Y., Wang, Q., Zhang, L., Xie, C., Zhang, Y., & Mao, Z. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3837071
- Authors
- Wang, X., Gu, X., Cao, J., Zhao, Z., Yan, Y., Middha, B., & Xie, X.
- Affiliations
- Microsoft Research Asia (China) (CN)
- Microsoft (United States) (US)
- Microsoft Research Asia, Shanghai, China
- Venue
- Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2021-08-14
- Pages
- 3697-3707
- OpenAlex
- W3166731436
- Cited by
- 20
- References
- 30
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 3 · 2023: 4 · 2022: 8
- FWCI
- 1.73
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- deep reinforcement learning, pre-trained language models, model-based reinforcement learning, user experience, search engine advertising
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
