A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“The contextual multi-armed bandit (MAB) is a widely used method in real-world applications requiring sequential decision-making under uncertainty, such as recommendation systems, computer networks, among others (Li et al., 2010; Wang et al., 2023; Dai et al., 2024b).”

“The conventional formulation of recommendation follows (contextual) ban-dits (Li et al., 2010) and assumes that viewers and providers are static across timesteps.”

“For example, earlier works have proven contextual bandits can excel in recommending news article links (actions) based on user clicks (reward) (Li et al., 2010; Tang et al., 2014).”

“A variety of engineering decision-making problems can be mapped to a bandit problem instance, e.g., in recommendation systems [3], [4], online advertising [5], [6], [7], dynamic pricing [8] and real-time bidding [9].”

“(2) LinUCB-Ind: LinUCB (Li et al., 2010) with separate preference vectors estimated for each user.”

“Bandits have been successfully applied to problems where uncertainty modeling and adaptation are beneficial, such recommender systems [32, 54, 25, 35] and hyper-parameter optimization [34].”

“The Linear Upper Confidence Bound (LinUCB) algorithm [52] is the most widely used CMAB model [55] and therefore used as a baseline for the developed algorithms.”

“This limits their ability to leverage structures like linear utility functions, which are widely adopted both in preference learning with large option spaces [21, 24, 41, 54, 56] and in cognitive models for human multi-attribute decision-making [26, 64, 76].”

“One of the most used approaches to developing deep neural bandits is stacked neural-linear bandit: a linear bandit [1, 34] is grafted atop a deep feature extractor, [60, 65].”

“In this section, we evaluate NegUCB across the three representative negotiation tasks depicted in Figure 1, comparing it with five representative baselines: ANAC agent 1 , LinUCB, FactorUCB, KernelUCB, and a reinforcement learning-based negotiation method (Cao et al., 2018; Bagga et al., 2020).”

“In order to alleviate the problem of information overload, help users target their reading interests, and propose personalized news recommendation [1–3].”

“…bandit (MAB) problem is one of basic models for sequential decision-making under uncertainty, with diverse applications in areas such as clinical trials [1]– [3], financial portfolio design [4], recommendation systems [5], [6], search engine systems [7], and cognitive radio networks [8].”

“The contextual bandit framework has been widely adopted to model such problems because of its applicability and ele-gance [1], [2], [3].”

“…the machine learning community has conducted extensive research in this area, driven by numerous applications, including website optimization, A/B testing, and the formulation of meta-algorithms for algorithmic procedures (Auer et al., 2002; Li et al., 2010; Komiyama et al., 2015; Li et al., 2017).”

“…2002) vs., identifying the best arm (Audibert et al., 2010; Garivier & Kaufmann, 2016), bandit algorithms have been widely applied in various real-world applications, such as model selection (Maron & Moore, 1993), recommender systems (Li et al., 2010a;b), and clinical trials (Durand et al., 2018).”

“For RelativeConUCB and ArcUCB , the arm-level recommendation (Line 8-10 of Algorithm 2 and Line 9-11 of Algorithm 4) is essentially similar to that of LinUCB (Li et al. 2010).”

“The MAB problem has applications in healthcare, advertising, telecommunications and more (Bouneffouf et al., 2020).”

“For example, Li et al. [16] represent news articles and users, respectively, by using URLs and information such as geographic information and demographics.”

“Synthetic Dataset Generation Following standard procedures in the OPE literature (Chu et al., 2011; Li et al., 2012; Agrawal and Goyal, 2013), we adapt supervised learning datasets into example logged bandit datasets where we know the true underlying generative process.”

“A finite time-varying arm set has many meaningful practical applications such as recommendation system (Li et al., 2010; Chu et al., 2011), and the distributed (federated) nature of the applications naturally falls in the federated linear bandits problem with finite time-varying arms.”

“When it is not known, the problem of learning to optimize it online can be formulated as a contextual bandit (Li et al., 2010; Chu et al., 2011), where the goal is to learn a policy that takes the most rewarding action in each context .”

“While MAB has found broad applications in recommender systems [44], healthcare [63], wireless communications [24], and beyond [8], a systematical connection between MAB and prompt optimization has not been established before to the best of our knowledge, which may spark new research activities (see…”

“Memory utility is state-dependent (Li et al., 2010; Agrawal and Goyal, 2013; Lin et al., 2024a; Ashizawa et al., 2025).”

“MABs capture a fundamental exploration-exploitation tradeoff and have thus become a standard model for applications such as recommendation systems [Li et al., 2010], online experimentation [Villar et al., 2015], and digital advertising [Chapelle and Li, 2011].”

“Besides adversarial poisoning, the preference data is also inherently noisy due to annotation difficulty, limited expert availability, and systematic differences across annotators (Casper et al., 2023; Yan et al., 2024).”

“The left side of the figure demonstrates the multi-armed bandit framework in action: at each step, our PA-LinUCB agent, which is inspired by LinUCB [12] and tailored for the environment of the IC3 algorithm observes the current proof context and selects among different generalization strategies…”

“We opt to employ Neural Thompson Sampling (Zhang et al., 2021), a recent contextual bandit algorithm (Auer, 2002; Filippi et al., 2010; Li et al., 2010; Agrawal and Goyal, 2013), to serve as f ( · | θ ) .”

“The policy π θ ( s t ) is implemented via LinUCB [17], which selects actions by maximizing an upper confidence bound on the expected reward: where ˆ µ a is the estimated mean reward, A a is the design matrix, and α controls exploration.”

“In contextual bandits and related online decision problems, actions are chosen using past observations, inducing dependence between observations collected at different rounds [Auer et al., 2002, Li et al., 2010, Lattimore and Szepesvári, 2020, Sutton and Barto, 2018].”

“The online contextual bandit learning problem, a one-step version of online reinforcement learning (RL), has garnered a lot of attention due to its usage in modern applications such as online advertising, recommendations, mobile health (Li et al., 2010, Tewari and Murphy, 2017).”

“We focus on LinUCB [49] as the representative AI-based approach, since it o ff ers strong online guarantees for this type of experiment.”

“MAB algorithms have been widely applied in domains such as online recommendation systems (Kveton et al., 2015; Chapelle et al., 2014; Li et al., 2010), adaptive clinical trials (Kuleshov & Precup, 2014), and dynamic pricing (Tullii et al., 2024).”

“Instead of treating all queries identically, contextual bandits leverage relationships between input characteristics and reward outcomes [44].”

“Information access has long motivated technologies from search engines [27, 8] to diverse downstream applications [17, 24].”

“In practice, partially observable reinforcement learning is prevalent in numerous practical applications such as robotics [37], autonomous driving [38], game playing [39], recommendation systems [40], and resource allocation [41].”

“They have achieved remarkable success in applications such as recommendation systems [54], adaptive clinical trials [23], online advertising [18], and intelligent tutoring systems [65].”

“√ In terms of regret, [QCZ14] presents a variant of LinUCB in [LCLS10] that achieves (cid:101) O ( d ST ) regret.”

“To remedy this limitation, various extensions have introduced dependencies among arms, including linear bandits [DHK08, AYPS11], ban-dits with graph feedback [ACBDK15, ACBG + 17], ban-dits with interference [JFK24, AAMW24, JSK25], and causal bandits [LLR16, YHS + 18, LB18].”

“The goal is to minimize the cumulative regret [30, 31] assumes a linear reward model E [ r t | z t , a ] = θ ⊤ a z t for each atomic action a , and selects actions using an upper confidence bound where A a is a covariance matrix and α > 0 controls exploration.”

“Inaddition,manyothermeta featuresofnewsarealsotakenintoconsiderationbynews modeling,suchascategories[47],[48],[49],[50],[51],[52], [53], [54], [55], [56], [57], [58], locations [59], [60], [61], [62], publishers [57], [60], [63], time stamp [34], [48], [55], [60], [64], as well as popularity [38],…”

“This exploration-exploitation trade-off is central to many real-world applications, ranging from clinical trials [Villar et al. , 2015] to personalized recommendations [Li et al. , 2010; Li et al. , 2011].”

“A payoff function, or a cost function, is learned for each arm to map contexts to expected rewards from that arm (L. Li et al. 2010).”

“This extension enablesthesurrogatemodeltoadaptitsbehaviorinresponsetochanges in external conditions, with potential applications in personalized recommendations 40 and stock investments 41 .”

“They form the backbone of many interactive applications, including personalized recommendation [1], online advertising [2], clinical trials [3], and adaptive routing [4].”

“Through comprehensive evaluations from multiple perspectives, we demonstrate that VIPO learns accurate models efficiently and consistently outperforms existing approaches.”

“Baselines AdaO2B was compared with several classic on-line to batch conversion algorithms as well as their variants: Best Batch Learner (BBL) chooses the model with the highest cumulative reward (Dekel and Singer 2005) AdaO2B and these baselines are model-agnostic and can be applied to various bandit algorithms: Seque…”

“P ROPOSED S OLUTION : L IN UCB FOR R ATE A DAPTATION (L IN RA) The LinUCB algorithm [11] extends the Upper Confidence Bound algorithm, designed for contextual bandit problems.”

“Linear bandits (Li et al., 2010; Agrawal & Goyal, 2013; Rusmevichientong & Tsitsiklis, 2010) and kernelized contextual bandits (Chowdhury & Gopalan, 2017; Valko et al., 2013; Zhou et al., 2020) is relevant to ours in using the similarity among actions and rewards for improving data efficiency.”

“The version of bandits that can deal with patient-specific effects is known s contextual bandits (Li et al., 2010; Chu et al., 2011), where typically the reward is modelled as an inner-product of some unknown arm-specific vector with the instance covariates.”

“Li et al. (2010) and Bouneffouf et al. (2012) leverage contextual information to enhance mobile and news article recommendation systems.”

“• Our numerical experiments confirm that our algorithm consistently outperforms OLB (Li et al., 2010; Agrawal & Goyal, 2013; Kim & Paik, 2019) and MAB (Latti-more & Szepesvári, 2020) algorithms, validating both its practicality and theoretical guarantees (Section 6).”

“This process, commonly referred to as offline policy evaluation/learning, has diverse applications across various domains, including healthcare (Tang and Wiens 2021), recommendation (Li et al. 2010), and fi-nance (Qin et al. 2022).”

“Bandits learning is a powerful framework for solving a broad spectrum of sequential decision-making problems, such as recommendation systems (Li et al., 2010), clinical 1 Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA USA 2 Electrical & Computer…”

“Another approach to OPE is the direct method (DM) (Li et al., 2010; Beygelzimer and Langford, 2009; van Seijen et al., 2009; Harutyunyan et al., 2016; Le et al., 2019; Voloshin et al., 2021).”

“Many applications of sequential decision-making under uncertainty use bandit learners, from news [4], movie [5], crowd-sourcing recommendation [6], strategic playing [7], software testing [8], to database index selection [9], [10], [11], query optimisation [12], and adaptive online query plan…”

“…could be extending these results to other variants of learning problem with bandit feedback, such as contextual bandits (Langford and Zhang, 2007; Li et al., 2010; Slivkins, 2011a; Agarwal et al., 2014; Agrawal and Devanur, 2016; Krishnamurthy et al., 2020; Foster and Rakhlin, 2020; Foster and…”

“The contextual bandit (CB) approach, an extended form of multiarm bandit (MAB), can meet our requirements [28], [29], [30].”

“The upper confidence bound (UCB) (Li et al., 2010; Chu et al., 2011; Zhou et al., 2020) and Thompson sampling (TS) (Krause & Ong, 2011; Agrawal & Goyal, 2013; Zhang et al., 2021) are widely-used techniques for dealing with the exploration-exploitation trade-off.”

“To record the shape and boundaries of the object of interest, M. Ayle et al. modified the LinUCB [39] method for BAR-CB and took into consideration the Histogram of Oriented Gradients (HOG) for the state.”

“…an e O ( √ dT ) (the e O ( · ) notation omits logarithmic factors in T ) regret upper bound, its empirical performance falls short of both the algorithm in Li et al. (2010) and the Linear Thompson Sampling algorithm (Agrawal & Goyal (2013)) as mentioned in Lattimore & Szepesvári (2020, Chapter 22).”

“It’s a sequence of zero-mean independent sub-Gaussian random variables with a uniform upper bound on the sub-Gaussian constants[14, 50, 51].”

“To this end, the linear or contextual UCB is proposed, which models the relationship of reward and arm side-information ( also known as context) as a linear function with learnable parameters [26].”

“…system applications where personalization is allowed, the platform can also benefit from user-dependent payoffs through contextual learning [Li et al., 2010, Li and Zhang, 2018, Besbes et al., 2021, Cohen et al., 2020, Leme and Schneider, 2018, Liu et al., 2018, Gollapudi et al., 2021].”

“We consider the linear contextual bandit setting ( e.g., Li et al. 2010, Chu et al. 2011, Abbasi-Yadkori et al. 2011) as the general learning problem.”

“Lemma 4.2.1 is the regret bound of SupLinUCB (a varient of LinUCB) which has been proved by Li et al. (2010), and lemma 4.2.2 is the regret bound of oracle FTL. Corollary 4.2.1 compares EWC with both LinUCB and Oracle FTL. =1 l i,t .”

“In recommender systems, privacy constraints might limit access to certain user features, but we can infer a distribution over these features [2].”

“A widely adopted ridge regression approach [35], [46] is applied to derive the closed-form of ˆ θ a,t and ˆ v w,t .”

“The contextual bandit problem is widely studied in fields such as sequential treatment allocation (Tewari & Murphy, 2017), personalized recommendations (Beygelzimer et al., 2011), and online advertising (Li et al., 2010).”

“One of the most popular contextual bandit algorithms is LinUCB, proposed by Li et al. [20].”

“A possible solution adopted in [13] is to apply ridge regression and regularize B a,t by summing an identity matrix, at the expenses of biasing the estimated ˆ Q ( s, a ) [5].”

“Thus, to find a more practical problem setting, we draw parallels with news recommendation systems, where models can only learn from user feedback, such as clicks on a single article (Li et al., 2010; Bouneffouf and Rish, 2019).”

“LIMBO is a gradient-free contextual bandit for inference-time allocation, extending the LinUCB formulation [24] with a two-head parameterization for joint accuracy and cost prediction.”

“This fits a contextual bandit setting, in which the available arms are described by features and can vary across rounds (Li et al., 2010; Chu et al., 2011).”

“However, the MOMAB framework treats each arm independently and thus fails to exploit contextual or structural information often available in real-world applications, such as product categories, prices, or user profiles in recommender systems [Li et al. , 2010].”

“Existing contextual bandit algorithms do not capture this architecture: • LinUCB / LinTS [Li et al., 2010, Agrawal and Goyal, 2013] estimate context weights and arm preferences jointly by a single UCB or posterior update; the supervised signal is never extracted into a separate pipeline, the gate is never modelled.”

“LinUCB is a contextual bandit algorithm [20]: it assumes each arm’s expected reward is a linear function of the current context, and maintains a running estimate of the weights (using ridge-regression) from observed rewards.”

“…a fundamental framework for sequential decision-making under uncertainty [Robbins, 1952, Lai and Robbins, 1985, Auer, 2002], with widespread applications in personalized recommendation [Li et al., 2010], online advertising [Schwartz et al., 2017], and resource allocation [Khansa et al., 2021].”

“Contextual bandits [13] are a specialized form of reinforcement learning designed to map contexts to actions that maximize an immediate reward.”

“2, we employ Contextual Multi-Arm Bandits (CMAB) [55], as formulated by Algorithm 2.”

“While this is a common challenge in both practical recommendation systems (Li et al., 2010; Chapelle and Li, 2011) and theoretical bandits with expert advice (Auer et al., 2002; Beygelzimer et al., 2011), the extremely limited expert observability is unique in the context of model routing: Only an…”

“The standard LinUCB regret bound (Li et al., 2010) gives cumulative regret O ( d √ t log t ) , which translates into a per-round sub-optimality gap of ˜ O ( (cid:112) d/t ) .”

“The study of bandit algorithms has seen a resurgence over the past two decades, driven by applications in online advertising (Li et al., 2010; Mehta et al., 2007), clinical trials (Durand et al., 2018), anomaly detection (Ding et al., 2019; Soemers et al., 2018), and recommendation systems (Mary et…”

“…penalize over-explored” behavior with a lightweight count-based bonus over facet tags , inspired by the Upper Confidence Bound (UCB) family (Auer, 2002; Auer et al., 2002; Li et al., 2010): rarely used facets receive larger bonuses, and the bonus decays naturally as they are chosen more frequently.”

“Non-stationary and contextual bandits adapt to changing rewards and query context (Garivier and Moulines, 2011; Kocsis and Szepesvári, 2006; Russac et al., 2019; Li et al., 2010).”

“LinUCB [28, 13] Linear ˜ O ( d √ T ) [1] ˜ O ( √ dT ) under feature diversity assumption [16] LinTS [7, 2] Linear ˜ O ( d 3 / 2 √ T ) [2] GLM-UCB [17] Generalized linear model ˜ O ( d √ T ) [17] ˜ O ( √ dT ) under feature diversity assumption [29] KernelUCB [38] Lies in RKHS ˜ O ( ˜ dT ) with ˜ d…”

“In tabular applications such as mobile health [15, 2] and recommendation systems [17], linear reward models [1] are standard yet often badly misspecified: human behavior involves interactions, threshold effects, and heterogeneity that linearity cannot capture.”

“…optimization literature [SHW26, CDH21] and captures several practical settings, including policy optimization and reinforcement learning [KL02, SLA + 15], online recommendation and contextual bandits [LCLS10], continual learning [PKP + 19], and federated learning with non-stationary clients [KM21].”