A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“Cascades, orchestration, bandit-style, multi-round, and online-serving routers (Ding et al., 2024; Zhang et al., 2023; 2025b; Woisetschläger et al., 2025; Li et al., 2010; Zhang et al., 2025a; Wu & Silwal, 2025) sit at the edge of the single-shot setting and serve as reference points.”

“…decision-making under bandit feedback (e.g., see Cesa-Bianchi and Lugosi [2006], Hazan [2016], Lattimore and Szepesv ´ ari [2020]) is a fundamental problem in machine learning, with applications ranging from recommendation systems [Li et al., 2010] to experimental design [Srinivas et al., 2010].”

“The regret incurred during Phase 1 satisfies:”

“Among RL methods, contextual multiarmed bandits, particularly the LinUCB algorithm [3], are well-suited for resource-constrained devices since they combine low computational complexity with the ability to exploit feature information.”

“We adopt the Linear Upper Confidence Bound (Lin-UCB) algorithm [45] as the learning engine, which models the expected reward of each arm k as a linear function of the context vector: where θ k ∈ R 4 is the unknown parameter vector for arm k. Lin-UCB maintains, for each arm k, a matrix A k ∈ R 4 × 4…”

“At the algorithmic level, our online branch builds on the optimism-based foundations of contextual and linear bandits: LinUCB and SupLinUCB [Li et al., 2010, Chu et al., 2011], the OFUL analysis [Abbasi-Yadkori et al., 2011], and linear Thompson-style approaches [Agrawal and Goyal, 2013].”

“Consequently, the learner only receives the reward corresponding to the chosen arm/MVHS [34].”

“In standard online learning settings, e.g., basic multi-armed bandits [7] or linear bandits [8], the feedback is usually a scalar stochastic reward or loss, which is only related to the action (probe) taken by the algorithm.”

“However, when CBs have yet to gather any user data, they perform essentially randomly and thus exhibit linear regret (Li et al., 2010; Auer et al., 2002).”

“The second group learns from partial feedback and naturally fits the contextual bandit setting Li et al. [2010], Zhou [2015].”

“CF approaches [1, 11, 42] infer preferences from behavioral similarity, while CB approaches [14, 25, 26, 32] leverage textual content, categories, and metadata to model relevance.”

“The Linear Upper Confidence Bound (LinUCB) algorithm (Li et al., 2010; Abbasi-Yadkori et al., 2011) employs the principle of optimism in the face of uncertainty .”

“…framework has proven remarkably versatile across a wide range of applications, including clinical trials (Gittins, 1979), recommendation systems (Li et al., 2010), financial optimization (Brochu et al., 2010), and modern incentivized learning methods (Frazier et al., 2014; Wang and Huang, 2018;…”

“Thus, following most online bandit and LTR models (Li et al., 2010; Joachims et al., 2017), we suppose that we observe the binary match indicators immediately after the recommendation for simplicity.”

“This approach has been well-developed for the contextual bandit setting, with a variety of algorithms and theoretical results on optimality (Li et al., 2010; Abbasi-Yadkori et al., 2011).”

“Given the wide applicability of contextual bandits in settings involving sensitive personal data—such as news recommendation [9], healthcare [10], and education [11]— protecting both context and reward information from adversaries is paramount.”

“This lack of exploration often reinforces feedback loops—models repeatedly recommend items similar to those already popular, leading to “rich get richer” dynamics—and ultimately results in suboptimal long-term performance [19, 33, 35].”

“Finally, Section 7 concludes the paper by summarizing the findings and outlining several promising directions for future research, including real-time adaptation, multi-agent learning, and deployment in physical wireless systems.”

“We instantiate this with linear contextual bandits (LinUCB) [18] and add (a) non-stationary variants (discounted/sliding-window) and (b) safe exploration via a conservative fallback policy.”

“More recently, multi-armed bandits have been applied to a wide range of selection problems, including recommendation (Li et al. 2010) systems, online advertising (Chapelle and Li 2011) and dose-finding clinical trials (Aziz et al. 2021; Jin et al. 2023; Kojima 2023).”

“We simulate the algorithm Lin-UCB in the setting of contextual bandits under three different parameter configurations [22].”

“To accommodate the state information, a modified approach called the linear disjoint UCB (LinUCB) algorithm is proposed in [10].”

“In the value-based setting, we assume the context-conditional expected loss function belongs to a given function class, whether linear [2, 6, 30, 40, 56], non-parametric [33, 48, 52], or a generic class with a regression oracle [22, 24, 26, 47, 58].”

“2 The matching results under single platform setting and multi-platform cooperation settings. achieve different objectives [9, 12, 22, 23].”

“Multi-armed Bandit (MAB) [1] is a foundational framework for sequential decision-making under uncertainty, with widespread applications in clinical studies [2], online recommendation [3], and hyperparameter tuning [4].”

“Many works also make use of some features of the arms, such as their similarities, to inform the choices [29].”

“However, many real-world applications—such as dynamic pricing [14], evolving user preferences [21], and environmental shifts [16]—are inherently non-stationary, with reward distributions that change over time.”

“Linear Upper Confidence Bound (LinUCB) is a generalized contextual bandit algorithm that assumes the expected reward of an arm is linear to the features Li et al. [2010].”

“In these settings, regression models such as linear (Abbasi-Yadkori et al., 2011), generalized linear models (GLMs) (Li et al., 2010; Filippi et al., 2010), and more flexible nonparametric approaches, including kernels, nearest neighbors, and neural networks (Yang and Zhu, 2002; Rigollet and Zeevi,…”

“…for personalization and adaptive decision-making, contextual bandits have been widely adopted in various domains, including recommender systems [1], [2], online advertising [3], and healthcare [4], where decisions must be made based on contextual information to maximize user engagement,…”

“To answer these questions, we formulate a contextual bandit [17] problem, modeled as a single-step Markov Decision Process (MDP).”

“In standard online learning settings, e.g., basic multi-armed bandits [Auer, 2002] or linear bandits [Li et al., 2010], the feedback is usually a scalar stochastic reward or loss, which is only related to the action (probe) taken by the algorithm.”

“• Linear Upper Confidence Bound (LinUCB) [32]: LinUCB is a learning-based algorithm that leverages context information to determine where to place requests given the entire topology.”

“…∪ E ∪ U as (12) 12: end for 13:14: // Select the best decomposition 15: w t := argmax w ∈ W ′ UCB t ( θ t,r , ψ r ( s t , w )) x v t,x · LCB t ( θ t,x , ψ x ( s t , w )) To address this issue, we adapt ideas from linear bandit algorithms, particularly LinUCB [18], and embed them into PrimalDualBwK.”

“…by introducing modeling flexibility in both decision and feedback processes: Combinatorial bandits extend the decision process to allow multiple selection for a subset of arms [5]; Contextual bandits provide the feedback process with more fine-grained modeling of rewards based on the context [6].”

“Applications of the user-centric selection problem are vast and include personalized content delivery [19] and resource allocation in ridesharing platforms [7].”

“One branch of CB algorithms, which will be considered as a reference for comparison in this paper, is the LinUCB [18].”

“Telecommuting days are days [2 , 4] in days [1 , 2 , ..., 7] .”

“• Linear Contextual Bandit (LUCB) [19], [20]: The user implements conventional linear contextual bandits (as opposed to neural bandits) using the same hierarchical structure and contexts from HNUCB .”

“…This framework has gained significant attention in the literature as it yields analytic tools that can be applied to several concrete applications, such as online advertising (Abe et al., 2003), recommendation systems (Li et al., 2010; Korkut and Li, 2021), and dynamic pricing (Cohen et al., 2020).”

“The arm with the highest UCB is selected, thereby encouraging exploration of arms whose rewards are uncertain but potentially high [3, 4, 13].”

“The arm selection and parameter update rules are as follows: Select arm : Update : • Linear UCB (LinUCB): LinUCB (Li et al., 2010; Abbasi-yadkori et al., 2011) extends UCB to contextual bandits, assuming that rewards follow a linear function of the context/feature vector x t ∈ R d as explained…”

“Contextual bandits are a standard framework for modeling these problems, especially in personalized decision-making, where side information helps tailor actions to individuals (Tewari & Murphy, 2017; Li et al., 2010).”

“Many real-world decision-making systems, such as recommendation systems (Li et al., 2010; Swaminathan et al., 2017; Saito & Joachims, 2021), ad-placement systems (Bot-tou et al., 2013), healthcare (Qian & Murphy, 2011; Liao et al., 2021), and many others, interact with their users through the…”

“The architecture employs a LinUCB-based ( Li et al., 2010 ) caching optimization algorithm to address the identi fi ed issues.”

“Conventional content-based approaches [18, 19] can utilize news term frequency features, such as TF-IDF, along-side the user profiles derived from historically clicked news to recommend news with a higher similarity to the user’s profile.”

“While these methods show promise, challenges such as computational intensity and the need for extensive hyperparameter tuning remain significant obstacles for real-time applications[8].”

“Bayesian RL builds on this by using Bayesian inference to guide the agent’s exploration [8].”

“Obviously, our setting relates to contextual bandits (Wang et al., 2005; Li et al., 2010; Abbasi-Yadkori et al., 2011; Hao et al., 2020) and, indeed, our results hold for the contextual setting.”

“In Linear Upper Confidence Bound (LINUCB) [19]–[21] and in Contextual Thompson Sampling (CTS) [8], the authors assume a linear dependency between the expected reward of an action and its context, where the representation space was modeled using a set of linear predictors.”

“The “monitor” arm generates stochastic We propose to make monitoring decisions using the Lin-UCB algorithm, with the variables listed above as input covariates [35].”

“…Zhou, 2016) is a versatile and general framework for sequential decision-makings and has been widely deployed in various practical domains, such as personalized recommendations (e.g. Li et al., 2010), clinical trials (e.g. Ameko et al., 2020), and portfolio management (e.g. Cannelli et al., 2023).”

“Therefore, LinUCB (Li et al., 2010) is adopted on new M ′ tasks respectively.”

“Corollary 1 (Uncertainty region) The essence of LNUCB-TA revolves around the concept of ”optimism in the face of uncertainty” (Liu et al., 2024a; Kamiura and Sano, 2017; Lykouris et al., 2021; Li et al., 2010; Russo and Roy, 2013).”

“This problem has broad applications in online recommendation systems [16], adaptive clinical trials [2, 30], and dynamic pricing [4], among others.”

“Bandit models underpin a wide range of engineering systems, from recommendation and ads to dynamic pricing and real-time bidding (Li et al., 2010; Schwartz et al., 2017; Sertan et al., 2012; Tajik et al., 2024; Flajolet & Jaillet, 2017).”

“In these scenarios, assumptions based on low-order and moderately sized features [12], [13] are no longer applicable, necessitating a shift in research focus towards high-order, high-dimensional problems.”

“For example, UCB applied to linear bandits from Li et al. (2010) requires the solution of a nonconvex optimization problem at each iteration that becomes computationally challenging in high dimensions.”

“Inspired by this, we apply the simulator grounded on Yahoo! news dataset to evaluate our proposed bandit policy with certain customizations [3].”

“Contextual bandit model [38], different than the traditional bandit problems, assumes that the expected payoff of an action is a linear combination of the contextual information and an arm-specific bandit parameter that is initially unknown and learned through sequential experimentation.”

“Off-policy Off-policy policy evaluation (OPE) is commonly applied to evaluate the utility of learned policies [10, 14, 18, 22].”

“Upper Confidence Bound (UCB) [14], [15] based algorithms that use optimism in the face of uncertainty have been developed in works such as [1], [2], and [16].”

“Canonical examples are UCB (Upper Confidence Bound) [20], and Thompson sampling [21], with numerous applications across various domains such as online advertising [22], personalized news feeds [23], customer support [24], and e-commerce recommendations [25, 26].”

“Peekaboo employs the LinUCB [25] algorithm to determine whether to transmit over an available path or wait a more advantageous one, while OLAPS uses the UCB [19] algorithm to decide whether to redundantly send packets.”

“In linear bandits, the singularity problem is alleviated by e.g. adding a regularisation term in the regression loss function [32, 22] or applying a dimensionality reduction technique [38, 10].”

“In this paper, we report from a case study in an F2P massively multiplayer online role-playing game (MMORPG), which aims to start exploring the application of bandit meth-ods for the personalization of mobile game starter packs - specifically, of off-the-shelf contextual bandit methods [17].”

“These bandit models have been applied across various domains, such as advertisements [Li et al. , 2010; Li et al. , 2011], dynamic procurement [Badanidiyuru et al. , 2012; Badani-diyuru et al. , 2013] and crowdsourcing platforms [Slivkins and Vaughan, 2014].”

“Indeed, existing work has examined MAB systems under a multitude of setups and constraints such as stochastic bandits [5]–[7], adversarial bandits [8]–[10], or contextual bandits [10], [11]; and over a number of distributed setups as well [12]– [15].”

“To bridge the gap, our study investigates contextual bandit algorithms in the presence of approximate inference , which are widely used in various real-world applications, such as personalized healthcare platforms [32] and recommendation systems [33].”

“Denoting the contexts from the source domain S as { x S t,a } i ∈ [ N ] ,a ∈ [ K ] , the associated ground-truth action as { a ∗ i } Ni =1 , and the contexts from the target domain T as { x T t,a } i ∈ [ N ] ,a ∈ [ K ] , the upper bound for our target regret R T is where R S , ϵ S ( h ) , and…”

“( This dataset is often used for testing contextual bandit algorithms [Li et al., 2010, Semenov et al., 2022].)”

“Among those literatures, the pioneer work of linear bandit algorithm described in Li et al. (2010) assumes a linear relationship between contextual features and the expected rewards.”

“Copyright 2026 by the author(s). as advertising, consumer goods, music, videos, news, job markets, and travel routes (Koren et al., 2009; Li et al., 2010; Covington et al., 2016; Wang et al., 2017; Zheng et al., 2018; McInerney et al., 2018; Naumov et al., 2019; Lewis et al., 2020; Bao et al.,…”

“The proposed FCOM algorithm is compared against various benchmark MAB algorithms, including LinUCB [24], Sync-LinUCB [19], and CLUCB [2].”

“• LinUCB [17], which applies the contextual multi-armed bandit algorithm LinUCB to solve VBR-encoded video streaming directly.”

“Today Module by Li et al. (2010).”

“LinUCB [Li et al., 2010]: It is a generic contextual bandit algorithm used by [Chen et al., 2019a].”

“Several algorithms such as Upper Confidence Bound and e -greedy are used to solve the multi-armed bandit problem which can be applied to the domain of news recommendation as well [48].”

“A more representative study is the LinUCB algorithm proposed by Li et al. [14], which extends the UCB algorithm to a linear parameterization setting.”

“Please refer to [5] for classical derivations of UCB under linear model.”

“SBUCB is a batched version of LinUCB [19], which updates the policy after receiving a batch of feedback data [17].”

“DQN is a reinforcement learning algorithm used to balance the Q values of state pairs in discrete space [5].”

“We implemented three policy evaluation methods: • Rejection Sampling [16] : Reject sampling is an unbiased estimator that estimates the performance of a policy by collecting some data on which actions are chosen at random, a reject sampling scheme is performed later on the policy to be evaluated,…”