A contextual-bandit approach to personalized news article recommendation
Open accessAbstract
Personalized web services strive to adapt their services (advertisements, news articles, etc.) to individual users by making use of both content and user information. Despite a few recent advances, this problem remains challenging for at least two reasons. First, web service is featured with dynamically changing pools of content, rendering traditional collaborative filtering methods inapplicable. Second, the scale of most web services of practical interest calls for solutions that are both fast in learning and computation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The problem of parameter selection has recently received considerable attention in the machine learning community in the context of multi-armed bandit problems.
“The problem of parameter selection has recently received considerable attention in the machine learning community in the context of multi-armed bandit problems [76], [77], [78].”
From Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2017) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Social Choice and Individual Values
Schnepp, G. J., & Arrow, K. (1951) · Economica · doi:10.2307/2549613 · Cited by 10,000 · see citing works
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033 · cited 4× · see citing works
“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”
BackgroundMethodInfluentialWhat Can Economists Learn from Happiness Research?
Frey, B. S., & Stutzer, A. (2001) · Social Science Research Network · doi:10.1257/002205102320161320 · Cited by 4,139 · see citing works
van der Lubbe, J. C. A. (1987) · Nature · doi:10.1007/978-0-387-35973-1_638 · Cited by 3,620
Usher, D., & Sen, A. (1987) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.2307/135247 · Cited by 3,280 · see citing works
Will raising the incomes of all increase the happiness of all
Easterlin, R. (1995) · Journal of Economic Behavior & Organization · doi:10.1016/0167-2681(95)00003-b · Cited by 3,202 · see citing works
Income and Happiness: Towards a Unified Theory
Easterlin, R., & Johnson, S. (2001) · The Economic Journal · doi:10.1111/1468-0297.00646 · Cited by 2,782 · see citing works
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712 · see citing works
Multiple Criteria Decision Analysis
Belton, V., & Stewart, T. J. (2002) · doi:10.1007/978-1-4615-1495-4 · Cited by 2,659 · see citing works
Asymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406 · cited 2× · see citing works
“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”
BackgroundMethod(2002) · doi:10.1137/s0097539701398375 · Cited by 2,275 · cited 2× · see citing works
“The EXP4 algorithm [8] uses the exponential weighting technique to achieve añ O( √ T ) regret, 2 but the computational complexity may be exponential in the number of fe atures.”
BackgroundMethodSelf-citationSome aspects of the sequential design of experiments
Robbins, H. (1952) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1952-09620-8 · Cited by 2,184 · see citing works
Multicriteria Methodology for Decision Aiding
French, S. (1996) · doi:10.1057/palgrave.jors.2600981 · Cited by 2,001
Environmental Accounting: Emergy and Environmental Decision Making
Odum, H. (1995) · Choice Reviews Online · doi:10.5860/choice.34-0412 · Cited by 1,997 · see citing works
Google news personalization: scalable online collaborative filtering
Das, A., Datar, M., Garg, A., & Rajaram, S. (2007) · The Web Conference · doi:10.1145/1242572.1242610 · Cited by 1,833 · cited 1× · see citing works
“. service vendors acquire and maintain a large amount of content in their repository, for instance, for filtering news articles [14] or for the display of advertisements [5].”
BackgroundThe Adaptive Web, Methods and Strategies of Web Personalization
Brusilovsky, P., Kobsa, A., & Nejdl, W. (2007) · The Adaptive Web · doi:10.1007/978-3-540-72079-9 · Cited by 1,776 · cited 1× · see citing works
“Recently, personalized recommendation has become a desirable feature for websites to improve user satisfaction by tailoring content presentation to suit individual users’ needs [10].”
BackgroundBandit processes and dynamic allocation indices
Gittins, J. C. (1979) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1979.tb01068.x · Cited by 1,487 · see citing works
Semiorders and a Theory of Utility Discrimination
Luce, R. (1956) · Econometrica · doi:10.2307/1905751 · Cited by 1,126 · see citing works
Utilitarianism and beyond: Contents
Sen, A., & Williams, B. (1982) · Cambridge University Press eBooks · doi:10.1017/cbo9780511611964 · Cited by 880 · see citing works
Information Theory, Inference, and Learning Algorithms
(2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.834752 · Cited by 866 · cited 1× · see citing works
“Furthermore, in information theory [19], the diff erential entropy ofp(θa) is defined as− 1 2 ln((2π) detAa).”
BackgroundWeak Comparability of Values as a Foundation for Ecological Economics
Martínez-Alier, J., Munda, G., & O'Neill, J. (1998) · Ecological Economics · doi:10.1016/s0921-8009(97)00120-1 · Cited by 794 · see citing works
The Foundations of Welfare Economics
Hicks, J. (1939) · The Economic Journal · doi:10.2307/2225023 · Cited by 787 · see citing works
Social multi-criteria evaluation: Methodological foundations and operational consequences
Munda, G. (2004) · European Journal of Operational Research · doi:10.1016/s0377-2217(03)00369-2 · Cited by 783 · see citing works
Sample mean based index policies by O(log n) regret for the multi-armed bandit problem
Agrawal, R. (1995) · Advances in Applied Probability · doi:10.2307/1427934 · Cited by 714 · cited 1× · see citing works
“In contrast to the unguided exploration strategy adopted by greedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”
Personal Utilities and Public Judgements: Or What's Wrong with Welfare Economics?
Sen, A. (1979) · The Economic Journal · doi:10.2307/2231867 · Cited by 665 · see citing works
Quah, E., & Mishan, E. (1972) · doi:10.4324/9780203695678 · Cited by 655
Sustainability and the measurement of wealth
Arrow, K., Dasgupta, P., Goulder, L., Mumford, K. J., & Oleson, K. (2012) · Environment and Development Economics · doi:10.1017/s1355770x12000137 · Cited by 637 · see citing works
Choices without Prices without Apologies
Vatn, A., & Bromley, D. (1994) · Journal of Environmental Economics and Management · doi:10.1006/jeem.1994.1008 · Cited by 573 · see citing works
Robust Multidimensional Poverty Comparisons
Duclos, J., Sahn, D., & Younger, S. D. (2006) · The Economic Journal · doi:10.1111/j.1468-0297.2006.01118.x · Cited by 522 · see citing works
Social Choice and Multicriterion Decision-Making
Bonner, J., Arrow, K., & Raynaud, H. (1986) · The Economic Journal · doi:10.2307/2232938 · Cited by 414 · see citing works
Maximization and the Act of Choice
Sen, A. (1997) · doi:10.2307/2171939 · Cited by 406
Arnold, F., & Poincaré, H. (1905) · The Journal of Philosophy Psychology and Scientific Methods · doi:10.2307/2012085 · Cited by 386 · see citing works
Michalos, A. (1980) · Social indicators research series · doi:10.1007/1-4020-3742-2_10 · Cited by 371 · see citing works
Bandit Problems: Sequential Allocation of Experiments.
Bather, J., Berry, D., & Fristedt, B. E. (1986) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981558 · Cited by 336 · cited 1× · see citing works
“The context-free K-armed bandit problem has been studied by statisticians for a long time [9, 24, 26].”
BackgroundText-learning and related intelligent agents: a survey
Mladenić, D. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.784084 · Cited by 320
Representing People, Representing Nature, Representing the World
O'Neill, J. (2001) · Environment and Planning C Government and Policy · doi:10.1068/c12s · Cited by 319 · see citing works
Incentives and political economy
Laffont, J. (2000) · doi:10.1093/0199248680.001.0001 · Cited by 305 · see citing works
Personalized recommendation on dynamic content using predictive bilinear models
Chu, W., & Park, S.-T. (2009) · The Web Conference · doi:10.1145/1526709.1526802 · Cited by 288 · cited 4× · see citing works
“We re-trained the bili near logistic regression model studied in [12] on Sept 2008 random t raffic data, using features zt,a constructed above.”
MethodInfluentialSelf-citationGarmendia, E., & Stagl, S. (2010) · Ecological Economics · doi:10.1016/j.ecolecon.2010.03.027 · Cited by 270 · see citing works
Efficient bandit algorithms for online multiclass prediction
Kakade, S., Shalev-Shwartz, S., & Tewari, A. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390212 · Cited by 218 · see citing works
Consequential evaluation and practical reason
Sen, A. (2000) · doi:10.2307/2678488 · Cited by 214
Graham, G., & Raz, J. (1987) · doi:10.2307/2219586 · Cited by 206
PHILOSOPHICAL PROBLEMS IN COST–BENEFIT ANALYSIS
Hansson, S. (2007) · Economics and Philosophy · doi:10.1017/s0266267107001356 · Cited by 174
Deliberative Monetary Valuation and the Evidence for a New Value Theory
Spash, C. (2008) · Land Economics · doi:10.3368/le.84.3.469 · Cited by 173
Cost-benefit analysis in integrated environmental assessment: some methodological issues
Munda, G. (1996) · Ecological Economics · doi:10.1016/0921-8009(96)00048-1 · Cited by 135 · see citing works
Reinforcement Learning with Immediate Rewards and Linear Hypotheses
Abe, N., Biermann, A., & Long, P. M. (2003) · Algorithmica · doi:10.1007/s00453-003-1038-1 · Cited by 133 · cited 1× · see citing works
“Assumin g the expected payoff of an arm is linear in its features, Auer [6] d escribes theLinRel algorithm that is essentially a UCB-type approach and shows that one of its variants has a regret of Õ( √ T ), a significant improvement over earlier algorithms [1].”
MethodKim, K. H. (1981) · Mathematical Social Sciences · doi:10.1016/0165-4896(81)90010-x · Cited by 132 · see citing works
Explore/Exploit Schemes for Web Content Optimization
Agarwal, D., Chen, B.-C., & Elango, P. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.52 · Cited by 130 · cited 1×
“free algorithms [2] in real bucket tests.”
Exploring compact reinforcement-learning representations with linear regressionPreprint
Walsh, T. J., Szita, I., Diuk, C., & Littman, M. L. (2009) · Conference on Uncertainty in Artificial Intelligence · doi:10.48550/arxiv.1205.2606 · Cited by 122 · cited 1× · see citing works
“When components in ca are independent conditioned on corresponding rows in Da, it can be shown [27] that, with probability at least 1− δ, ̨̨̨ x t,aθ̂a −E[rt,a|xt,a] ̨̨̨ ≤ α q x t,a(D a Da + Id)−1xt,a (4)”
COMBINING SOCIAL, ECONOMIC AND ENVIRONMENTAL INDICATORS TO MEASURE SUSTAINABLE HUMAN WELL-BEING
Michalos, A. (1997) · Social indicators research series · doi:10.1007/978-94-017-0389-5_2 · Cited by 110 · see citing works
Environment and Development: An Economic Approach
Bojö, J., Mäler, K., & Unemo, L. (1990) · doi:10.1007/978-94-015-8008-3 · Cited by 108
Some Problems with Environmental Economics
Sagoff, M. (1988) · doi:10.5840/enviroethics198810128 · Cited by 98
Valuing Objects and Evaluating Policies in Imperfect Economies
Dasgupta, P. (2001) · The Economic Journal · doi:10.1111/1468-0297.00617 · Cited by 90 · see citing works
A case study of behavior-driven conjoint analysis on Yahoo!: front page today module
Chu, W., Park, S.-T., Beaupre, T., Motgi, N., Phadke, A., Chakraborty, S., & Zachariah, J. (2009) · Knowledge Discovery and Data Mining · doi:10.1145/1557019.1557138 · Cited by 84 · cited 2× · see citing works
“Here, we intentionally used five users (and articles) groups , which has been shown to be representative in segmentation an alysis [13].”
MethodSelf-citationJust-in-time contextual advertising
Anagnostopoulos, A., Broder, A., Gabrilovich, E., Josifovski, V., & Riedel, L. (2007) · International Conference on Information and Knowledge Management · doi:10.1145/1321440.1321488 · Cited by 82 · cited 1×
“service vendors acquire and maintain a large amount of content in their repository, for instance, for filtering news articles [14] or for the display of advertisements [5].”
BackgroundOn the economic theory of interest groups: Towards a group frame of reference in political economics
van Winden, F. (1999) · doi:10.1023/a:1018347607116 · Cited by 80
Social multi-criteria evaluation of different development scenarios of the Aysén region, Chile
Gamboa, G. (2006) · Ecological Economics · doi:10.1016/j.ecolecon.2005.10.014 · Cited by 71 · see citing works
Hybrid Systems for Personalized Recommendations
Burke, R. (2003) · ITWP · doi:10.1007/11577935_7 · Cited by 61 · cited 1×
“Hybrid approaches [11] have been developed by combining two or more recommendation techniques; for example, the inabil ity of collaborative filtering to recommend new items is commonly a lleviated by combining it with content-based filtering.”
MethodEconomists Favour the Price System--Who Else Does?
Frey, B. (1986) · Kyklos · doi:10.1111/j.1467-6435.1986.tb00677.x · Cited by 60 · see citing works
New perspectives on public finance: recent achievements and future challenges
Stiglitz, J. (2002) · doi:10.1016/s0047-2727(01)00193-1 · Cited by 50
Straton, A., Jackson, S., Marinoni, O., Proctor, W., & Woodward, E. (2011) · doi:10.1007/s11269-010-9691-z · Cited by 50
Scolobig, A., Broto, V., & Zabala, A. (2008) · doi:10.1068/c0765s · Cited by 49
Monterroso, I., Binimelis, R., & Rodríguez-Labajos, B. (2011) · Journal of Environmental Management · doi:10.1016/j.jenvman.2010.09.017 · Cited by 46
Cerreta, M., & Toro, P. D. (2010) · doi:10.1504/ijsd.2010.035096 · Cited by 42
Local democracy implications for coastal zone management—A case study in southern Norway
Soma, K., & Vatn, A. (2009) · doi:10.1016/j.landusepol.2008.10.002 · Cited by 36
Social Multicriteria Evaluation of Conflict over Rural Electrification and Solar Energy in Spain
Munda, G., & Russi, D. (2008) · Environment and Planning C Government and Policy · doi:10.1068/c06105s · Cited by 34 · see citing works
Zendehdel, K., Rademaker, M., de Baets, B., & Huylenbroeck, G. V. (2010) · doi:10.1016/j.jaridenv.2009.09.011 · Cited by 30
Özkaynak, B. (2008) · Progress in Planning · doi:10.1016/j.progress.2008.04.002 · Cited by 27 · see citing works
Public economics yesterday, today and tomorrow
Laffont, J. (2002) · doi:10.1016/s0047-2727(01)00188-8 · Cited by 27
Simulation Studies of Multi-armed Bandits with Covariates (Invited Paper)
Pavlidis, N. G., Tasoulis, D., & Hand, D. (2008) · International Conference on Modelling and Simulation · doi:10.1109/uksim.2008.86 · Cited by 24 · cited 1×
“Algorithm 1 (or the similar algorithm in [22]) computes the inverse of the matrix, D a Da + Id (or D a Da), where Da is again the design matrix with rows corresponding to features in the training data.”
BackgroundEthics and Climate Change Cost-Benefit Analysis: Stern and After
Aldred, J. (2009) · New Political Economy · doi:10.1080/13563460903288221 · Cited by 23 · see citing works
The justice and rationale of cost-benefit analysis
Copp, D. (1987) · doi:10.1007/bf00127337 · Cited by 21
Mismeasuring the value of statistical life
Grüne-Yanoff, T. (2009) · Journal of Economic Methodology · doi:10.1080/13501780902940703 · Cited by 16 · see citing works
The Stern Review and its critics: economics at work in an interdisciplinary setting
Hansen, F. (2011) · doi:10.1080/1350178x.2011.611026 · Cited by 9
Flagged first, then by citations
16 of the 90 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
Karpatne, A., Atluri, G., Faghmous, J. H., Steinbach, M., Banerjee, A., Ganguly, A., Shekhar, S., Samatova, N., & Kumar, V. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2720168 · Cited by 1,752 · see citing works
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342 · see citing works
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 368 · see citing works
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Kohavi, R., Tang, D., & Xu, Y. (2020) · doi:10.1017/9781108653985 · Cited by 297 · see citing works
Survey on reinforcement learning for language processing
Uc-Cetina, V., Navarro-Guerrero, N., Martin-Gonzalez, A., Weber, C., & Wermter, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10205-5 · Cited by 177 · see citing works
Interactive collaborative filtering
Zhao, X., Zhang, W., & Wang, J. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505690 · Cited by 165 · see citing works
A Primer on Reinforcement Learning in Medicine for Clinicians
Jayaraman, P., Desman, J., Sabounchi, M., Nadkarni, G., & Sakhuja, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01316-0 · Cited by 130 · see citing works
“Additionally, RL has also been used for generating personalized recommendations 21 for subscription-based viewers, content moderation on social media 22 platforms, and smart grid systems 23 .”
An Efficient Bandit Algorithm for Realtime Multivariate Optimization
Hill, D. N., Nassif, H., Liu, Y., Iyer, A., & Vishwanathan, S. V. N. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098184 · Cited by 112 · see citing works
Beyond Preferences in AI Alignment
Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 85 · see citing works
“…of recommender systems (Thorburn et al., 2022), with many papers modeling recommendation as the problem of showing items to users that they are most likely to engage with, which is presumed to be the item they find the most rewarding (Li et al., 2010; Hill et al., 2017; McInerney et al., 2018).”
BackgroundMixLLM: Dynamic Routing in Mixed Large Language ModelsPreprint
Wang, X., Liu, Y.-C., Cheng, W., Zhao, X.-J., Chen, Z., Yu, W.-C., Fu, Y.-J., & Chen, H.-F. (2025) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.18482 · Cited by 80 · see citing works
“…s trade n,l to control the priority of quality over cost, leading to different budgets accordingly: To handle prediction errors, we introduce an uncertainty measurement ( s unc n,l ) to enhance robustness (Li et al., 2010): where A − 1 l represents the inverse covariance matrix for the l -th LLM.”
Beyond Ranking: Optimizing Whole-Page Presentation
Wang, Y., Yin, D., Jie, L., Wang, P., Yamada, M., Chang, Y., & Mei, Q. (2016) · Proceedings of the Ninth ACM International Conference on Web Search and Data Mining · doi:10.1145/2835776.2835824 · Cited by 66 · see citing works
Li, D., Raymond, L., & Bergman, P. (2020) · doi:10.3386/w27736 · Cited by 64 · see citing works
Huang, J., Oosterhuis, H., de Rijke, M., & van Hoof, H. (2020) · Fourteenth ACM Conference on Recommender Systems · doi:10.1145/3383313.3412252 · Cited by 64 · see citing works
Panel Experiments and Dynamic Causal Effects: A Finite Population Perspective
Bojinov, I., Rambachan, A., & Shephard, N. (2021) · Quantitative Economics · doi:10.3982/qe1744 · Cited by 60 · see citing works
“Such experiments are widely used by technology companies to quickly discern user preferences in recommendation algorithms (Li et al. (2010), Li, Karatzoglou, and Gentile (2016)) and by academics interested in improving their power against a particular hypothesis (van der Laan Mark (2008)).”
MethodMetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs
Nguyen, Q. H., Dao, T., Hoang, D. C., Decugis, J., Manchanda, S., Chawla, N., & Doan, K. D. (2024) · arXiv.org · doi:10.48550/arxiv.2407.10834 · Cited by 55
“In this section, we provide empirical results of MetaLLM on popular APIs and benchmark datasets.”
KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems
Zhang, J., Huang, Z.-M., Fan, Y.-J., Liu, N.-Y., Li, M., Yang, Z.-J., Yao, J.-W., Wang, J., & Wang, K.-Z. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2502.07350 · Cited by 49
“Classical algorithms like UCB and Thompson Sampling excel in recommendation and resource allocation, while Contextual Bandits and adaptive methods refine decision-making in dynamic settings (Li et al., 2010).”
Wang, X.-B., Griffith, J., Adler, D. A., Castillo, J., Choudhury, T., & Wang, F. (2025) · International Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713852 · Cited by 45 · see citing works
Wu, Q., Han, J., Yan, Y.-M., Kuo, Y.-H., & Shen, Z.-J. M. (2025) · Health Care Management Science · doi:10.1007/s10729-025-09699-6 · Cited by 43 · see citing works
Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level ApproachesPreprint
Tao, Y.-C., Li, Y.-T., Qin, Y., & Liu, Y. (2025) · arXiv.org · doi:10.48550/arxiv.2510.04905 · Cited by 41 · see citing works
“LinUCB [103] for online adaptation of retrieval strategies, while FT2Ra [39] exploits ∆ logits as retrieval signals, simulating fine-tuning effects via adaptive learning rates and multi-round retrieval.”
Practice Makes Perfect: Planning to Learn Skill Parameter Policies
Kumar, N., Silver, T., McClinton, W., Zhao, L.-F., Proulx, S., Lozano-Pérez, T., Kaelbling, L., & Barry, J. L. (2024) · arXiv.org · doi:10.48550/arxiv.2402.15025 · Cited by 41
“We can view this parameter selection problem as a contextual bandit with infinite arms [6, 37].”
MethodVideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step
Wang, H.-Y., Liu, F.-F., Chi, J.-W., & Duan, Y.-Q. (2025) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.01536 · Cited by 40 · see citing works
“For instance, algorithms like LinUCB [20] and Thompson Sampling for linear models [2] assume that the expected reward is a linear function of the context.”
A Survey on Diffusion Models for Recommender Systems
Lin, J., Liu, J.-Q., Zhu, J.-C., Xi, Y.-J., Liu, C.-K., Zhang, Y.-T., Yu, Y., & Zhang, W.-N. (2024) · arXiv.org · doi:10.48550/arxiv.2409.05033 · Cited by 38
“Finally, the authors use LinUCB [15, 64] contextual multi-armed bandit algorithm to select the prompt with the highest predicted CTR from a pre-defined pool of prompts associated with the product category in the given context (defined by user, item, and location features), which is designed to…”
MethodEVOLvE: Evaluating and Optimizing LLMs For Exploration
Nie, A., Su, Y., Chang, B., Lee, J. N., Chi, E. H., Le, Q. V., & Chen, M.-M. (2024) · arXiv.org · doi:10.48550/arxiv.2410.06238 · Cited by 37
“Numerous algorithms have been developed to enable efficient exploration in both MAB (Auer, 2002) and CB (Langford & Zhang, 2007; Li et al., 2010) settings.”
InfluentialAdaptive Operator Selection for Meta-Heuristics: A Survey
Pei, J., Mei, Y., Liu, J.-L., Zhang, M.-J., & Yao, X. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3545792 · Cited by 36 · see citing works
“As operators’ reward distributions vary, the most related MAB variants to AOS include nonstationary MAB [150] and contextual MAB [151].”
A Unified Confidence Sequence for Generalized Linear Models, with Applications to Bandits
Lee, J., Yun, S., & Jun, K.-S. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2407.13977 · Cited by 36
“This encompasses a wide range of distributions, which in turn makes it ubiquitous in various real-world applications, such as news recommendations (Bernoulli; Li et al. (2010, 2012)), social network influence maximization (Poisson; Gisselbrecht et al. (2015); Lage et al. (2013)), and more.”
Algorithmic and Ethical Aspects of Recommender Systems in e-Commerce
Paraschakis, D. (2018) · doi:10.24834/2043/24268 · Cited by 35 · see citing works
Stepwise Alignment for Constrained Language Model Policy Optimization
Wachi, A., Tran, T. Q., Sato, R., Tanabe, T., & Akimoto, Y. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2404.11049 · Cited by 35
“See Li et al. [28].”
Abacus: A Cost-Based Optimizer for Semantic Operator Systems
Russo, M., Liu, C.-W., Sudhir, S., Vitagliano, G., Cafarella, M. J., Kraska, T., & Madden, S. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3796195.3796215 · Cited by 34
“Furthermore, if Abacus has access to learned embeddings for each operator, then it can model the search as a contextual MAB [21, 39] which, under some theoretical assumptions, should perform better than its current context-free approach.”
Austin, D., Korikov, A., Toroghi, A., & Sanner, S. (2024) · ACM Conference on Recommender Systems · doi:10.1145/3640457.3688142 · Cited by 34 · see citing works
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
Moslem, Y., & Kelleher, J. D. (2026) · arXiv.org · doi:10.48550/arxiv.2603.04445 · Cited by 32
“It extracts a lightweight context vector per query, encoding task type, semantic cluster, and textual complexity, and then employs LinUCB (Li et al., 2010) to learn routing policies online across a pool of 16 open-access LLMs.”
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
Dai, R., Song, L.-F., Liu, H., Liang, Z.-W., Yu, D., Mi, H.-T., Tu, Z.-P., Liu, R., Zheng, T., Zhu, H., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2509.09675 · Cited by 32
“Algorithms such as UCB (Lai, 1987) for multi-armed bandits, LinUCB (Li et al., 2010), LSVI-UCB (Jin et al., 2020) for linear bandits/MDPs achieve near-optimal exploration guarantees across a variety of settings.”
The Digital Transformation in Health: How AI Can Improve the Performance of Health Systems
Periáñez, Á., del Río, A. F., Nazarov, I., Jané, E., Hassan, M., Rastogi, A., & Tang, D. (2024) · Health systems and reform · doi:10.1080/23288604.2024.2387138 · Cited by 32 · see citing works
Contextual Client Selection for Efficient Federated Learning Over Edge Devices
Pan, Q., Cao, H., Zhu, Y.-F., Liu, J.-C., & Li, B. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3323645 · Cited by 32 · see citing works
“Comparison approaches: We compare our approach with four other solutions: random selection [13], Oort [20], k-LinUCB [45], and ClusterFL [17].”
MethodThe Generalization Gap in Offline Reinforcement Learning
Mediratta, I., You, Q., Jiang, M., & Raileanu, R. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2312.05742 · Cited by 31
An interactive food recommendation system using reinforcement learning
Liu, L.-L., Guan, Y., Wang, Z., Shen, R., Zheng, G.-W., Fu, X., Yu, X., & Jiang, J.-C. (2024) · Expert systems with applications · doi:10.1016/j.eswa.2024.124313 · Cited by 30 · see citing works
Efficient Frameworks for Generalized Low-Rank Matrix Bandit Problems
Kang, Y., Hsieh, C.-J., & Lee, T. C. M. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2401.07298 · Cited by 30
“The contextual bandit has proven to be a powerful framework for sequential decision-making problems, with great applications to clinical trials [35], recommendation system [21], and personalized medicine [4].”
BackgroundIn-Context Reinforcement Learning for Variable Action Spaces
Sinii, V., Nikulin, A., Kurenkov, V., Zisman, I., & Kolesnikov, S. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2312.13327 · Cited by 29
“Results and Discussion: As shown in Figure 6, Headless-AD’s performance is on par with LinUCB across varied arm counts.”
MethodInfluentialGlobal and Local Differential Privacy for Collaborative Bandits
Wang, H., Zhao, Q., Wu, Q., Chopra, S., Khaitan, A., & Wang, H. (2020) · Fourteenth ACM Conference on Recommender Systems · doi:10.1145/3383313.3412254 · Cited by 28 · see citing works
LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments
Ye, Z., Yoganarasimhan, H., & Zheng, Y.-F. (2024) · arXiv.org · doi:10.48550/arxiv.2406.02611 · Cited by 28
“Yahoo was one of the earliest platforms to use the linear contextual bandit for news article recommendation (Li et al., 2010).”
BackgroundMethodFrom Data to Decisions: The Power of Machine Learning in Business Recommendations
Gangadharan, K., Purandaran, A., Malathi, K., Subramanian, B., Jeyaraj, R., & Jung, S. K. (2024) · IEEE Access · doi:10.1109/access.2025.3532697 · Cited by 28 · see citing works
“Implement user access control [211] to restrict access to sensitive data and ensure that each stakeholder has access only to the relevant information.”
DeepServeCB: SLO-Aware and Cost-Aware Elastic Scheduling for Serverless Multi-Tenant LLM Inference
Li, X.-X., Liu, X.-Y., Wang, Z.-L., Xiao, Z.-J., Hsieh, C.-Y., & Liu, Y.-X. (2026) · 2026 International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519206 · Cited by 26 · see citing works
“…(1) We formulate the joint SLO--cost scheduling problem as a contextual bandit optimization---an online learning framework where the agent observes a context (per-request features and system states) and selects an action to maximize cumulative reward without requiring a full environment model [9].”
Optimal Baseline Corrections for Off-Policy Contextual Bandits
Gupta, S., Jeunen, O., Oosterhuis, H., & de Rijke, M. (2024) · ACM Conference on Recommender Systems · doi:10.1145/3640457.3688105 · Cited by 26 · see citing works
“While this line of research is not inherently new [35, 52], methods based on contextual ban-dits (or reinforcement learning by extension) have now become widespread in the recommendation field [2, 5, 24, 39, 40, 56, 64].”
Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution
Poon, M.-H., Dai, X.-X., Liu, X., Kong, F.-Y., Lui, J. C. S., & Zuo, J.-H. (2025) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2506.17670 · Cited by 25
“Our work is built upon the theory of contextual bandits (Agrawal and Devanur 2016; Li et al. 2010), a classic approach for online learning in dynamic environments like recommendation (Luo et al. 2018; Zeng et al. 2025; Wang et al. 2025c) or pricing (Xiong et al. 2019).”
Feel-Good Thompson Sampling for Contextual Dueling Bandits
Li, X.-H., Zhao, H.-Y., & Gu, Q.-Q. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2404.06013 · Cited by 25
“The effectiveness of FGTS is theoretically justified: when applied to linear contextual bandits, FGTS can achieve the minimax-optimal regret bound as UCB-based algorithms like LinUCB (Li et al., 2010) or OFUL (Abbasi-Yadkori et al., 2011).”
BackgroundAutomatic Engineering of Long Prompts
Hsieh, C.-J., Si, S., Yu, F., & Dhillon, I. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.10117 · Cited by 25
“We model sentence selection as a contextual ban-dit problem (Langford and Zhang, 2007; Li et al., 2010).”
MethodA Multi-Objective Recommender System for Enhanced Consumer Behavior Prediction in E-Commerce
Sun, Y.-P., Xiang, Y., Zou, D.-X., Li, N., & Chen, H.-T. (2024) · 2024 IEEE 6th International Conference on Power, Intelligent Computing and Systems (ICPICS) · doi:10.1109/icpics62053.2024.10796677 · Cited by 24
Generalized Linear Bandits with Limited Adaptivity
Sawarni, A., Das, N., Barman, S., & Sinha, G. (2024) · Neural Information Processing Systems · doi:10.52202/079017-0267 · Cited by 24
“Similarly, in online advertising [SBF17] and recommendations [LCLS10], updating the policy after every iteration during deployment is often infeasible due to infrastructural constraints.”
Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offsPreprint
Wei, W., Yang, T., Chen, H.-J., Zhao, Y., Dernoncourt, F., Rossi, R. A., & Eldardiry, H. (2025) · arXiv.org · doi:10.48550/arxiv.2510.07429 · Cited by 23 · see citing works
“Classical bandit algorithms include LinUCB (Li et al., 2010), which uses optimism in linear reward models; Thompson Sampling (Agrawal & Goyal, 2014), which maintains a posterior over reward parameters; and ϵ -greedy strategies, which trade off exploration and exploitation through randomization.”
IC-Cache: Efficient Large Language Model Serving via In-context Caching
Yu, Y.-F., Gan, Y., Sarda, N., Tsai, L., Shen, J., Zhou, Y.-Q., Krishnamurthy, A., Lai, F., Levy, H. M., & Culler, D. E. (2025) · Symposium on Operating Systems Principles · doi:10.1145/3731569.3764829 · Cited by 23
“Its lightweight nature and minimal needs for online feedback make it especially well-suited for real-time serving [46].”
Optimal Design for Human Preference Elicitation
Mukherjee, S., Lalitha, A., Kalantari, K., Deshmukh, A., Liu, G., Ma, Y., & Kveton, B. (2024) · Neural Information Processing Systems · doi:10.52202/079017-2861 · Cited by 23
“Front Page Today Module User Click Log Dataset [50, 51, 103, 7].”
Jump Starting Bandits with LLM-Generated Prior Knowledge
Alamdari, P. A., Cao, Y., & Wilson, K. H. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2406.19317 · Cited by 22
“When users encounter content that caters to their needs and preferences, they are more inclined to engage, e.g., watch a movie [3] or click on a news article [31].”
BackgroundMethodAutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation
Fu, J., Qin, X., Yang, F., Wang, L., Zhang, J., Lin, Q.-W., Chen, Y.-B., Zhang, D.-M., Rajmohan, S., & Zhang, Q. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2406.19251 · Cited by 22
“Specifically, we frame hyper-parameter selection as an online multi-armed bandit (MAB) problem (Lai and Rob-bins, 1985; Vermorel and Mohri, 2005; Li et al., 2010) and explore several variants in MAB.”
MethodOffline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
Jiang, N., & Xie, T. (2025) · Statistical Science · doi:10.48550/arxiv.2510.04088 · Cited by 21
DAT: Dynamic Alpha Tuning for Hybrid Retrieval in Retrieval-Augmented Generation
Hsu, H.-L., Wang, J., & Tzeng, J.-N. (2025) · arXiv.org · doi:10.48550/arxiv.2503.23013 · Cited by 21
Agricultural land use modeling and climate change adaptation: A reinforcement learning approach
Stetter, C., Huber, R., & Finger, R. (2024) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13448 · Cited by 21 · see citing works
PageRank Bandits for Link Prediction
Ban, Y., Zou, J., Li, Z., Qi, Y.-Z., Fu, D.-Q., Kang, J., Tong, H., & He, J.-R. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2411.01410 · Cited by 20
“The first line of works studies the linear reward assumption, typically calculated using ridge regression [39, 8, 1, 60, 21, 53].”
A Theoretical Framework for AI-driven data quality monitoring in high-volume data environments
Bangad, N., Jayaram, V., Krishnappa, M., Banarse, A. R., Bidkar, D. M., Nagpal, A., & Parlapalli, V. (2024) · arXiv.org · doi:10.48550/arxiv.2410.08576 · Cited by 20
“To address the challenge of alert fatigue, the component uses multi-armed bandit algorithms for intelligent alert prioritization [47].”
MethodEfficient Sequential Decision Making with Large Language Models
Chen, D., Zhang, Q., & Zhu, Y. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2406.12125 · Cited by 20
“This approach has been widely applied in real-world scenarios, including recommendation systems (Li et al., 2010; Agarwal et al., 2016), healthcare (Tewari and Murphy, 2017; Svensson, 2023), and dialogue systems (Li et al., 2016).”
MethodNeural Contextual Bandits for Personalized Recommendation
Ban, Y., Qi, Y.-Z., & He, J.-R. (2023) · The Web Conference · doi:10.1145/3589335.3641241 · Cited by 20
“– Linear Upper Confidence Bound (UCB) [1] – Linear Thompson Sampling (TS) [2] – Linear Personalized Recommendation [24] • Part II: Neural Contextual Bandits Duration: 40 minutes, Presenter: Yunzhe Qi and Yikun Ban This is the core part.”
BackgroundFeedback-based self-learning in large-scale conversational AI agents
Ponnusamy, P., Ghias, A., Yi, Y., Yao, B., Guo, C., & Sarikaya, R. (2022) · AI Magazine · doi:10.1609/aimag.v42i4.15102 · Cited by 19 · see citing works
Lin, Z.-H., Ma, W., Zhou, M., Zhao, Y.-J., Wang, H., Liu, Y., Wang, J., & Li, L. (2024) · arXiv.org · doi:10.48550/arxiv.2409.14177 · Cited by 19
“Harmful questions are chosen randomly, while jailbreak templates are selected using the UCB strategy (Li et al., 2010).”
MethodSelf-citationReinforcing Long-Term Performance in Recommender Systems with User-Oriented Exploration Policy
Zhang, C.-S., Chen, S., Zhang, X., Dai, S.-H., Yu, W.-J., & Xu, J. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657714 · Cited by 19
“Contextual Bandit algorithms [26, 27, 54, 61] typically involve calculating the upper confidence bound of each arm or sampling actions based on the posterior distribution of observed context.”
RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning
Yuan, M.-Q., Castanyer, R. C., Li, B., Jin, X., Zeng, W.-J., & Berseth, G. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2405.19548 · Cited by 19
“We aim to implement this logic in the future in our RLeXplore codebase.”
Unified Dual-Intent Translation for Joint Modeling of Search and Recommendation
Zhang, Y., Wu, Y.-Q., Han, R.-D., Sun, Y., Zhu, Y., Li, X., Lin, W., Zhuang, F.-Z., An, Z.-L., & Xu, Y.-J. (2024) · Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671519 · Cited by 18 · see citing works
“Aiming to help users discover items of interest from a vast array of options, recommendation systems have become an essential component of various online platforms, such as e-commerce [26, 48, 49] and digital news services [7, 19, 35].”
Incentivized Learning in Principal-Agent Bandit Games
Scheid, A., Tiapkin, D., Boursier, E., Capitaine, A., Mhamdi, E. E., Moulines, É., Jordan, M. I., & Durmus, A. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2403.03811 · Cited by 18
“Decision-making under uncertainty is a ubiquitous feature of real-world applications of machine learning, arising in domains as diverse as recommendation systems (Li et al., 2010), healthcare (Yu et al., 2021), and agriculture (Evans et al., 2017).”
BackgroundMitigating Exposure Bias in Recommender Systems—A Comparative Analysis of Discrete Choice Models
Krause, T., Deriyeva, A., Beinke, J. H., Bartels, G. Y., & Thomas, O. (2024) · Trans. Recomm. Syst. · doi:10.1145/3641291 · Cited by 18 · see citing works
“In this study, we built upon findings from previous research that discrete choice–based matrix factorization models reduce exposure bias on synthesized data.”
Letard, A., Gutowski, N., Camp, O., & Amghar, T. (2024) · Expert systems with applications · doi:10.1016/j.eswa.2024.123151 · Cited by 18 · see citing works
PolicySim: An LLM-Based Agent Social Simulation Sandbox for Proactive Policy Optimization
Huang, R.-H., Tang, N., Xu, J.-R., Cao, Y., Tu, Q., Guo, S., Zheng, B., Liu, H., & Yang, Y. (2026) · The Web Conference · doi:10.1145/3774904.3792555 · Cited by 17 · see citing works
Generalized Linear Bandits: Almost Optimal Regret with One-Pass Update
Zhang, Y.-J., Xu, S., Zhao, P., & Sugiyama, M. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2507.11847 · Cited by 17
“This flexibility enables the modeling of various real-world tasks, such as recommendation systems [Li et al., 2010] and personalized medicine [Tewari and Murphy, 2017], where the feedback is binary (Bernoulli) or count-based (Poisson) and inherently non-linear.”
Tang, X.-Q., Li, J., Du, N., & Xie, S. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2412.07618 · Cited by 17
“LinUCB (Li et al. 2010) under-performs due to its inability to handle the high-dimensional, complex nat-ural language embeddings.”
InfluentialConversational Recommendation With Online Learning and Clustering on Misspecified Users
Dai, X.-X., Wang, Z., Xie, J., Liu, X., & Lui, J. C. S. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3423442 · Cited by 17 · see citing works
“Our comparison includes the following baselines: LinUCB: A method uses a single estimated preference vector shared across all users [5].”
Contextual Bandits for Unbounded Context Distributions
Zhao, P.-N., Fan, R.-F., Wang, S.-W., Shen, L., Zhang, Q.-X., Ke, Z., & Zheng, T.-H. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2408.09655 · Cited by 17
“Many research assume linear rewards (Abbasi-Yadkori et al., 2011; Bastani & Bayati, 2020; Bastani et al., 2021; Qian et al., 2023; Langford & Zhang, 2007; Dudik et al., 2011; Chu et al., 2011; Li et al., 2010), which is restrictive and may not fit well into practical scenarios.”
EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems
Yu, Y., Gao, C., Chen, J., Tang, H., Sun, Y., Chen, Q., Ma, W.-Z., & Zhang, M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657868 · Cited by 17 · see citing works
POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition
Saito, Y., Yao, J.-H., & Joachims, T. (2024) · arXiv.org · doi:10.48550/arxiv.2402.06151 · Cited by 17
“…framework has emerged as a favored approach for online learning and decision-making under uncertainty (Lattimore & Szepesv´ari, 2020), spurring the development of numerous efficient algorithms for navigating (potentially vast or infinite) action spaces (Agrawal & Goyal, 2013; Li et al., 2010).”
BackgroundSample-based Dynamic Hierarchical Transformer with Layer and Head Flexibility via Contextual Bandit
Meng, F., Zhang, L., Chen, Y., & Wang, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2312.03038 · Cited by 17
“…configurations of layers and heads for each sample in a real-time fashion, we integrate Uniform Confidence Bound multi-arm contextual bandits (UCB) (Li et al., 2010) and Thompson Sampling semi-bandits (TSP) (Wang et al., 2017) into our DHT model, where the UCBs are responsible for determining the…”
MethodResultvan Niekerk, J. V., le Roux, J. L., & Craig, I. (2025) · Control Engineering Practice · doi:10.1016/j.conengprac.2025.106522 · Cited by 16 · see citing works
Multi-Armed Bandits With Machine Learning-Generated Surrogate Rewards
Ji, W., Pan, Y.-H., Zhu, R., & Lei, L. (2025) · arXiv.org · doi:10.48550/arxiv.2506.16658 · Cited by 16
“Similarly, online news vendors deploy MAB algorithms to continuously refine content recommendations and enhance user engagement in real time [24, 14, 31].”
Gu, J., Fu, Y., & Hung, K. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3360959 · Cited by 16 · see citing works
“The utilization of context information draws inspiration from contextual bandit algorithms [30].”
MethodErfanian, M., Jagadish, H. V., & Asudeh, A. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3682014 · Cited by 16 · see citing works
Dynamic Product Image Generation and Recommendation at Scale for Personalized E-commerce
Czapp, Á. T., Jani, M., Domián, B., & Hidasi, B. (2024) · ACM Conference on Recommender Systems · doi:10.1145/3640457.3688045 · Cited by 15 · see citing works
“We deployed a solution similar to [1] and use the LinUCB [2, 4] contextual bandit algorithm to select the prompt that has the best estimated CTR in the given context – defined by user, item and placement features – from the predefined prompt pool belonging to the product’s category.”
MethodOptimizing Novelty of Top-k Recommendations using Large Language Models and Reinforcement Learning
Sharma, A., Li, H., Li, X., & Jiao, J. (2024) · Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671618 · Cited by 15 · see citing works
Amebleh, J., & Igba, E. (2024) · International Journal of Scientific Research and Modern Technology · doi:10.38124/ijsrmt.v3i5.819 · Cited by 15 · see citing works
Treatment Effect Estimation for User Interest Exploration on Recommender Systems
Chen, J.-J., Wang, W.-J., Gao, C.-M., Wu, P., Wei, J.-X., & Hua, Q.-S. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657736 · Cited by 15 · see citing works
“• Bandit algorithms [22, 23, 38] consider the potential rewards for enhancing the exploration-exploitation trade-off.”
BackgroundMethodInfluentialTowards Safe Policy Learning under Partial Identifiability: A Causal Approach
Joshi, S., Zhang, J., & Bareinboim, E. (2024) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29198 · Cited by 15
“This class of environmental models is also referred to as contextual bandit (Langford and Zhang 2008) and is widely applied in reinforcement learning literature (Li et al. 2010).”
BackgroundPearl: A Production-ready Reinforcement Learning Agent
Zhu, Z., Braz, R. D. S., Bhandari, J., Jiang, D. R., Wan, Y., Efroni, Y., Wang, L.-Y., Xu, R., Guo, H., Nikulkov, A., et al. (2023) · Journal of machine learning research · doi:10.48550/arxiv.2312.03814 · Cited by 15
“We tested neural implementations of the LinUCB, Thompson Sampling (TS), and SquareCB (Li et al., 2010; Agrawal & Goyal, 2013; Foster & Rakhlin, 2020).”
BackgroundMethodBalancing exploration and exploitation in episodic reinforcement learning
Chen, Q., Zhang, Q.-W., & Liu, Y.-L. (2023) · Expert systems with applications · doi:10.1016/j.eswa.2023.120801 · Cited by 15 · see citing works
Hidden Incentives for Auto-Induced Distributional Shift
Krueger, D., Maharaj, T., & Leike, J. (2020) · arXiv · doi:10.48550/arxiv.2009.09153 · Cited by 14 · see citing works
“Non-i.i.d bandits: Contextual bandits (Wang et al., 2005; Langford & Zhang, 2008) are frequently discussed as an approach to content recommendation (Li et al., 2010).”
BackgroundAgent-as-a-Router: Agentic Model Routing for Coding Tasks
Zhou, P., Tang, Z.-W., Ma, Y.-X., Tang, J.-S., Han, Y., Wan, Z.-L., Meng, F.-Q., Wang, W., Zhuang, B.-H., Zhao, W.-B., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.22902 · Cited by 14
“This loop relates to a contextual multi-armed bandit [22], so cumulative regret (the running gap to the per-task oracle) becomes the natural streaming metric.”
InfluentialActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
Lin, X.-Q., Verma, A., Dai, Z.-X., Rus, D., Ng, S.-K., & Low, K. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2505.19241 · Cited by 14
“Furthermore, recent studies (Lin et al., 2024) have demonstrated that sufficiently large transformer models, when pre-trained on offline interaction sequences, can approximate near-optimal online reinforcement learning algorithms such as LinUCB (Li et al., 2010) and Thompson Sampling in multi-armed bandits (Agrawal an…”
No-Regret is not enough! Bandits with General Constraints through Adaptive Regret Minimization
Bernasconi, M., Castiglioni, M., & Celli, A. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2405.06575 · Cited by 14
“The first regression-based approach for constrained BwK was proposed by [3] by exploiting the optimistic approach for linear CB [35, 22, 1].”
MethodNeural Active Learning Beyond Bandits
Ban, Y., Agarwal, I., Wu, Z.-W., Zhu, Y.-D., Weldemariam, K., Tong, H., & He, J.-R. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2404.12522 · Cited by 14
“In contextual bandits [38; 68], the learner is presented with K arms (context vectors) and required to select one arm in each round.”
BackgroundBayesian Off-Policy Evaluation and Learning for Large Action Spaces
Aouali, I., Brunel, V.-E., Rohde, D., & Korba, A. (2024) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.2402.14664 · Cited by 14
“Online learning under uncertainty is often modeled within the framework of contextual bandits (Lattimore & Szepesvari, 2019; Li et al., 2010; Chu et al., 2011).”
BackgroundExploring on role of location in intelligent news recommendation from data analysis perspective
Lv, P.-T., Zhang, Q., Shi, L., Guan, Z.-H., Fan, Y.-F., Li, J., Zhong, K., & Deveci, M. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120213 · Cited by 14 · see citing works
Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits
Taufiq, M. F., Doucet, A., Cornish, R., & Ton, J.-F. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2312.01457 · Cited by 14
“This paradigm is prevalent in many real-world applications such as healthcare, personalized recommendation systems, or online advertising [1–3].”
BackgroundLi, Z., Liu, M.-L., Dai, X.-X., & Lui, J. C. S. (2025) · The Web Conference · doi:10.1145/3696410.3714773 · Cited by 13
“• LinUCB [2, 23]”
Demystifying Inference After Adaptive Experiments
Bibaut, A. F., & Kallus, N. (2025) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040522-015431 · Cited by 13
Online Learning and Detecting Corrupted Users for Conversational Recommendation Systems
Dai, X.-X., Wang, Z., Xie, J., Yu, T., & Lui, J. C. S. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3448250 · Cited by 13 · see citing works
“(cid:2) LinUCB [41]: Use a single non-robust estimated vector to represent the preference of each user.”
BackgroundHu, X., Leung, H.-F., & Farnia, F. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2410.13287 · Cited by 13
“On the other hand, we note that the existing arm-specific bandit algorithms in (Li et al., 2010; Xu et al., 2020) are designed considering a linear pay-off structure, which is generally different from the kernel-based formulation in our work.”
Influential
- Authors
- Li, L., Chu, W., Langford, J., & Schapire, R. E.
- Affiliations
- Yahooo! Labs, Santa Clara, CA, USA
- Yahoo (United States) (US)
- Princeton University (US)
- Venue
- Proceedings of the 19th international conference on World wide web
- Publisher
- ACM
- Published
- 2010-04-26
- Pages
- 661-670
- OpenAlex
- W2112420033
- Cited by
- 2,562
- References
- 90
- Citations by year
- 2026: 197 · 2025: 164 · 2024: 171 · 2023: 212 · 2022: 211 · 2021: 317 · 2020: 312 · 2019: 260 · 2018: 177 · 2017: 119 · 2016: 86 · 2015: 112 · 2014: 83 · 2013: 59 · 2012: 41
- FWCI
- 31.45
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International World Wide Web Conference (WWW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- personalized news recommendation, contextual bandits, personalized web services, collaborative filtering limitations, scalable learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
