A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“The framework additionally supports two further bandit levels, evaluated in the ablation (Table 3): Planner selection (LinUCB (Li et al., 2010)).”

“Clearly, the partitioning mechanism effectively replaces the linear assumption in traditional contextual multi-armed bandits [34], [35], enabling us to model a non-linear relationship between contextual features and expected rewards.”

“• LASER: The first RM routing method that employs LinUCB to select a single RM per batch (Nguyen et al., 2024).”

“It has found widespread applications in areas such as online advertising, recommender systems, and mobile health (Li et al., 2010; Agarwal et al., 2016; Tewari & Murphy, 2017).”

“We evaluate the performance of the proposed GMORL algorithms with a single policy and compare it with a linear upper confidence bound (LinUCB)-based scheme [32], a multi-policy MORL scheme [24], a simulated annealing(SA)-basedscheme,andarandom-basedscheme.”

“Contextual bandit techniques have been applied in a range of personalization applications, such as healthcare (Tewari and Murphy, 2017; Kallus and Zhou, 2018; Mintz et al., 2020) and recommendation systems (Li et al., 2010; Agrawal et al., 2019).”

“Choosing the confidence radius based on a generalization bound as practiced commonly in bandit research (Li et al., 2010; Srinivas et al., 2010; Kaufmann et al., 2012; Lattimore and Szepesvári, 2020) and increasing the Bayesian modeling hierarchy may make EPPO more robust to hyperparameters.”

“MAB problem has found applications in numerous fields. such as online advertising [49], clinical trials [50], networking [47], [51], and pairwise ranking [52].”

“In information theory, ∥ x i ∥ V − 1 also reflects entropy reduction in the model posterior, and is used to measure the estimator uncertainty improvement contributed by the chosen action x i [22].”

“MABs have found applications in modern recommendation systems [17], clinical trials design [28], and mobile health [25].”

“LinUCB utilises confidence bounds to strike a balance between exploration and exploitation, making it well-suited for scenarios involving structured user feedback [66], [67].”

“While early systems made use of Bayesian models [38], bandits [34] and collaborative filtering [11], in recent years the focus has shifted towards content-based and often deep models.”

“Motivated by this question, we study off-policy evaluation (Li et al., 2010; Bottou et al., 2013; Li et al., 2015; Hof-mann et al., 2016) with human feedback.”

“Linear contextual bandit is a notable extension that incorporates contextual information and assumes an effective linear structure to improve scalability [17]–[19].”

“Note that the linear bandits model we consider is different from the contextual linear bandits in [5], [38] where the parameter is not shared by actions (although assuming linear reward function), and thus, the actions are not correlated through the parameter.”

“First, a multi-armed bandit approach [53], [54], [55], [56], [57] is adopted to balance exploration and exploitation.”

“…trade-off, along with the side information, is formalized by the contextual multi-armed bandit (CMAB) problem (Langford and Zhang, 2007; Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Nourani-Koliji et al., 2022), a notable extension of the multi-armed bandit (MAB)…”

“When the reward function is linear w.r.t. the context, LinUCB [Li et al., 2010, Abbasi-Yadkori et al., 2011] and Linear Thompson Sampling (LinTS) [Agrawal and Goyal, 2013] have demonstrated good performance.”

“These approaches maintain a modified confidence set C t ⊆ d using a ridge regression estimator of the true reward parameter θ ∗ and use Upper Confidence Bound (UCB) (Lattimore & Szepesv´ari, 2020; Li et al., 2010) In the setting of corrupted contexts, the algorithm can only calculate the corrupted ridge regression est…”

“…also been the object of a large body of work aiming at developing new and improving existing recommendation algorithms (see Koren et al. [2009], Li et al. [2010], L¨u et al. [2012], Wang and Zhang [2012], Luo et al. [2014], Covington et al. [2016], He et al. [2017], Yi et al. [2019] for a…”

“This recognition has sparked a growing interest in studying adaptive learning algorithms in the setting of contextual bandits [26, 33, 6] and reinforcement learning (RL) [46].”

“…past decade has seen the rapid development of contextual bandit as a legit framework to model interactive decision-making scenarios, such as personalized recommendation (Li et al. 2010), online advertising (Tang et al. 2013; Avadhanula et al. 2021), and anomaly detection (Ding, Li, and Liu 2019).”

“To conduct off-policy evaluation using our proposed algorithm and the competing variations of the TS algorithm, we outline the IPS estimator for an unbiased estimate of the per-trial expected reward based on what has been studied in Li et al. (2010).”

“In online advertising, as Li et al. (2010) highlight, these algorithms determine which ads to display to maximize click-through rates.”

“We refer the reader to Woodroofe [1979], Li et al. [2010], Agarwal et al. [2016] for additional practical examples.”

“It composes four foundational ingredients: LinUCB (Li et al., 2010) for contextual reward learning, BwK-inspired Lagrangian relaxation (Badanidiyuru et al., 2013; Agrawal et al., 2016) for budget enforcement, geometric discounting (Garivier & Moulines, 2011; Russac et al., 2019) for non-stationarity, and offline-to-on…”

“A widely-studied CB is the linear CB, which assumes that the expected reward of each arm is a linear function of context (Li et al., 2010; Chu et al., 2011).”

“Our setting is also conceptually adjacent to contextual bandits and sequential resource allocation: at each state, the system must choose which candidates to evaluate under a limited budget, us-ing context-dependent uncertainty to decide where evaluation has high marginal value(Li et al., 2010).”

“To provide rigorous theoretical guarantees for the exploration-exploitation trade-off, the field subsequently adopted the Multi-Armed Bandit (MAB) formulation [Lai and Robbins, 1985, Garivier and Moulines, 2008, Li et al., 2010].”

“▷ Generate abstract profile for all agent a i ∈ A in parallel do 8: Send profile p P forms pairs E t = { ( a s , a r , x sr ) } by selecting K p disjoint pairs that greedily maximize the LinUCB score: √ for all pair ( a s , a r , x sr ) ∈ E t in parallel do 14: a r receives knowledge package κ s from a s and integrate…”

“2) MAB-based Allocation adopts the LinUCB algorithm [35] to allocate queries based on historical performance and uncertainty estimates, rather than the feature extraction of a neural network.”

“While their approach relies on a sequential model that tracks changing interests, our solution emphasizes contextual awareness and leverages the effectiveness of contextual bandits [17] to balance exploration and exploitation [16, 18].”

“There are several standard approaches for OPE, including IS [Precup et al., 2000], direct method (DM) [Li et al., 2010, Beygelzimer and Langford, 2009, van Seijen et al., 2009, Harutyunyan et al., 2016, Le et al., 2019, Voloshin et al., 2021], and doubly robust approaches (DR) [Farajtabar et al.,…”

“This paper focuses on the latter by incorporating contextual information (e.g., user and item features), a setting known as the contextual bandit (CB) (Li et al. 2010, Chu et al. 2011).”

“Traditionally, recommendation systems rely on explicit user identifiers and interaction histories to personalize suggestions (Li et al. 2010).”

“Our construction of the confidence ball is analogous to the standard LinUCB (Li et al., 2010; Abbasi-Yadkori et al., 2011) algorithm for linear bandits.”

“These bounds are computed using a LinUCB-based framework [Li et al., 2010].”

“Contextual bandits form the foundation of modern sequential decision-making problems, driving applications such as recommendation systems, advertising, and interactive information retrieval (Li et al., 2010).”

“…(Slivkins et al. 2019) is a fundamental and versatile framework for sequential decision-making problems, with applications in online recommendation (Li et al. 2010), prompt engineering (Lin et al. 2023), clinical trials (Villar, Bowden, and Wason 2015) and hyperparameter tuning (Ding et al. 2022).”

“For linear bandit problems, the worst-case regret has been widely studied (Auer, 2002; Dani et al., 2008; Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011b; Li et al., 2019 √ Subsequently, (Zhou et al., 2021; Zhang et al., 2021; Kim et al., 2021; Zhou and Gu, 2022; Dai et al., 2022;…”

“Agents in this experiment utilize LinUCB (Li et al., 2010).”

“Each day, a new set of articles A τ is released, which the agent recommends to users who arrive throughout the day.”

“TheUCB exploration strategy [2], on the other hand, incorporates a second factor as shown in (2).”

“Similarly, contextual bandits (Lattimore and Szepesv´ari 2020) have been established as the leading model for personalized recommender systems (Li et al. 2010) and have proven essential in the alignment of large language models (Christiano et al. 2017; Rafailov et al. 2024).”

“LinUCB is a contextual MAB algorithm that has been developed with the aim of and commercially employed for personalizing news article recommendations [27].”

“The contextual multiarmed bandit task captures this by assuming that the reward distribution depends not only on the arm but also on a set of features that describe the decision context on that round (L. Li et al., 2010).”

“This direction could also explore more advanced CB methods than the LinUCB to enhance the scalability and applicability of our framework.”

“Thus,instead,weopttostudysimplercon-textualmultiarmedbandit(CMAB)formulation,whichhasbeen widelystudiedinrecommendationsystems,finance,healthcare, andrecently,robotics[13],[14],[15],[16],andallowsusto advantageouslyabstractcertainlowerlevelbehavioralaspects ofthesearchsiteselectionproblem.”

“On the other hand, the techniques of adding quadratic exploration bonuses, e.g., the one proposed in LinUCB (Li et al., 2010) and used in LSVI for linear MDPs, also fail here as Bellman backups of quadratic functions are not necessarily within the linear class F .”

“For these values of λ and γ , the regret analysis of LinUCB ((Li et al., 2010; Abbasi-Yadkori et al., 2011; Lattimore and Szepesv´ari, 2020)), under Assumption 2, leads to a regret guarantee of ˜ O (( d √ In Theorem 3, we present a much stronger guarantee when Assumption 1 is also satisfied.”

“The framework of contextual bandits is a widely adopted model for addressing online learning in uncertain environments [Lattimore and Szepesvari, 2019, Auer et al., 2002, Thompson, 1933, Russo et al., 2018, Li et al., 2010, Chu et al., 2011].”

“It has found application in various domains including healthcare [37] and online recommendation [25].”

“…surged across various fields, including clinical trials and healthcare (Murphy 2003; Bertsimas et al. 2017), online advertising and marketing (Bottou et al. 2013; Bertsimas and Kallus 2020), revenue management (Chen et al. 2022b), and online news recommendation (Li et al. 2010; Ban and Rudin 2019).”

“Based on this data, the coefficient vectors can be estimated online using Ridge regression [13].”

“The contextual bandit problem has been widely studied in fields such as sequential treatment allocation (Tewari & Murphy, 2017), personalized recommendations (Beygelzimer et al., 2011), and online advertising (Li et al., 2010).”

“…learning (RL) (Sutton and Barto, 2018) aims to optimize the interaction process between agents and the environment, and has succeeded in many practical applications, e.g., games (Silver et al., 2016), robotics (Levine et al., 2016; Gu et al., 2017), and recommendation systems (Li et al., 2010).”

“…step, the agent evaluates the predictive posterior distribution as where P ( c t | ˆ c t , γ ∗ ) is as defined in (2) and posterior distribution P ( γ ∗ |H t − 1, c ,ˆ c ) is obtained via Baye’s rule as P ( γ ∗ |H t − 1, c ,ˆ c ) ∝ P ( γ ∗ ) ∏ t − 1 τ = 1 P ( c τ , ˆ c τ | γ ∗ ) using the history…”

“It has a wide range of real-world applications, including recommender systems (Li et al., 2010; Yue et al., 2012; Joachims et al., 2021), online advertising (Tang et al., 2013; Bottou et al., 2013), experimental design (Krause & Ong, 2011), and medical interventions (Lei et al., 2014; Tewari &…”

“We use two classic exploration methods for scoring, Upper Confidence Bound (UCB) (Li et al. 2010) and Thompson Sampling (TS; Russo and Van Roy (2014)), to enable our algorithm to consider the potential of each node and look ahead.”

“Sequential optimization methods [35], including Bayesian optimization [34], contextual ban-dits [20], reinforcement learning [46], and adaptive experimental allocation [33], improve optimization efficiency by balancing exploration and exploitation through iterative feedback.”

“Adaptive Org Routing featurizes each task into an interpretable 7-dimensional vector and uses a LinUCB contextual bandit (Li et al. 2010) to select, per task, the protocol that maximizes a cost-adjusted reward r = quality − λ · cost, updating online via rank-1 Sherman–Morrison updates.”

“Common baselines include ϵ -greedy [35], UCB [2], and LinUCB [24].”

“Linear methods such as LinUCB [32, 15] assume rewards are linear in context features, while KernelUCB [44] and NeuralUCB [56] model nonlinear reward functions.”

“To achieve this, we cast the attack as a contextual bandit problem (Li et al., 2010), which is a natural fit for the core challenge of balancing the search for new biases ( exploration ) with the use of known ones ( exploitation ).”

“These networks share a common contextual-bandit policy [10], which surfaces the most informative precedents as advisory context at inference time and retires entries whose running reward falls below a deprecation threshold.”

“Future work involves refining prompt instructions using zero-and few-shot prompting, supporting additional properties [56, 86], and incorporating additional planners [30, 82].”

“Relaxing the assumption and considering context-dependent reward distribution will be related to a contextual bandit problem (Li et al., 2010) while we leave this as a future work.”

“To evaluate new items and discover user interests, researchers employ the Exploration and Exploitation paradigm ( e.g., Multi-armed Bandit [123], Contextual-Bandit [104]) to collect real-time user feedback on new items via a small amount of exploration traffic.”

“Beyond the multi-armed setting, Fan et al. (2025) study linear bandits and provide inference guarantees for LinUCB (Li et al., 2010; Abbasi-Yadkori et al., 2011), and related stability-based approaches have also been developed in broader contextual or adversarial regimes (Praharaj and Khamaru,…”

“Recent work of Fan et al. (2025) show that the LinUCB (Li et al., 2010; Abbasi-Yadkori et al., 2011) algorithm algorithm also satisfy this stability condition for linear contextual bandit problem.”

“…and Thompson Sampling (Thompson, 1933) offer optimal regret bounds for stationary environments, while recent work has extended these to contextual (Li et al., 2010; Chu et al., 2011), metric (Kleinberg et al., 2008), clustering (Gentile et al., 2014), and hierarchical settings (Bubeck et al.,…”

“It is worth mentioning that the multi-armed bandit approaches in recommendation systems (Li et al., 2010) naturally embody the exploration-exploitation-creation tradeoffs by continuously balancing known user preferences with the discovery of new content types and recommendation strategies.”

“The decision process cyclically executes and intelligently transitions between an Exploration Phase and an Exploitation Phase based on the model’s maturity[38].”

“…on three well-known problems: multi-armed bandits, linear contextual bandits, and reinforcement learning from human feedback (RLHF), which are widely used in online learning and advertising [Li et al., 2010], healthcare [Villar et al., 2015], and large language model alignment [Bai et al., 2022].”

“The celebrated LinUCB has been proposed and analyzed in Li et al. [2010], which consider linear contextual bandits and achieve regret ˜ O ( √ KdN 1 ) for K -armed disjoint linear model.”

“A study demonstrated the utility of contextual bandits in making personalized news article recommendations, which resulted in increased engagement and satisfaction [7].”

“4) Baseline 4: Contextual UCB [26], [55] based NS , where the UCB is introduced to determine whether arms are fully explored, this scheme jointly considered the relationship between historic reward and context, and achieves promising sublinear regret under sub-Gaussian distributed cases.”

“Later on, various extensions of the stochastic bandits have been proposed, such as the linear bandits (Abbasi-Yadkori et al., 2011), contextual bandits (Li et al., 2010), and combinatorial bandits (Chen et al., 2013).”

“Linear contextual bandits (CB) are studied since [Li et al., 2010, Chu et al., 2011, Abbasi-Yadkori et al., 2011].”

“…Press (2009), Magirr et al. (2012), Villar et al. (2015)), pricing experiments (Misra et al. (2019), Calvano et al. (2020), Wang et al. (2021)), portfolio selection (Gagliolo and Schmidhuber (2011), Shen et al. (2015), Huo and Fu (2017)), content recommendation (Li et al. (2010)), and others.”

“In more general RL problems, the reward is a function of transitioning from state x to state x ′ via action a : r ( x , a , x ′ ) [68]; however in Equation (5) we simply have r ( x , a , x ′ ) = r ( x , a ) (i.e., the contextual multi-armed bandit setting [46]).”

“The offline contextual bandit problem has emerged as a critical area of study in sequential decision-making, with significant implications for decision systems for various domains including recommendation (Li et al., 2010) and online advertising (Schwartz et al., 2017).”