A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“Among these, LinUCB served as the baseline, employing standard non-intersecting linear UCB without safety constraints or causal bias removal, representing the upper performance limit of traditional reinforcement learning methods in medical settings [12] .”

“2) UCB-based Exploration (UCB-Ex): This exploration framework is based on Upper Confidence Bound (UCB) [28], which is frequently applied to the recommendation sys-tem [29], [30].”

“Multi-armed bandits are a fundamental model for sequential decision-making under uncertainty [1], [2], with applications in medical trials [3], [4], [5], advertisement placement [6], [7], news recommendation [8], [9], web search [10], and recommendation systems [11], [12].”

“Multi-armed bandit theory [16] provides regret bounds for sequential decisions under uncertainty; the contextual extension [15] conditions action selection on observable system state.”

“Therefore, the MAB framework has been widely applied to online recommendation [1], adaptive routing [2], and dynamic pricing [3].”

“These problems are widely applicable, including dynamic resource allocation [14], online scheduling [15] and recommendation systems [16].”

“First, online prompt selection is naturally related to stochastic and contextual bandits, where a policy trades off exploration against exploitation under delayed or noisy reward [1, 10, 12].”

“We compare CSTS with (i) static global weights, (ii) audience potential max-imisation using a supervised audience rating regressor trained offline on past broadcasts, (iii) vanilla Thompson sampling [21] and (iv) LinUCB contextual bandit [13].”

“We use LinUCB [Li et al., 2010, Chu et al., 2011] to solve the routing task.”

“Advances in machine learning have enabled estimation of heterogeneous treatment effects (Athey and Imbens, 2016) that inform targeting policies in a number of domains like healthcare (Bastani et al., 2021), recommendation systems (Li et al., 2010), and microfinance (Banerjee et al., 2013).”

“There is also a large literature in general on regret bounds for fair contextual bandits (Li et al., 2010; Badanidiyuru et al., 2018; Balseiro et al., 2020; Neu et al., 2022; Blanchet et al., 2024), but this work focuses on algorithmic regret bounds rather than the characterization of assignment…”

“The contextual multi-armed bandit (MAB) framework serves as a cornerstone of sequential decision-making under uncertainty, finding widespread application in recommendation systems, online advertising, and personalized services (Li et al., 2010; Lattimore & Szepesv´ari, 2020).”

“Since mutation selection is a sequential decision-making process, we explore learning-guided mutation with the linear upper confidence bound (LinUCB) contextual bandit algorithm [22], implemented with MABWiser [28].”

“The bandit can use LinUCB, Thompson Sampling, or neural bandits [6] with exploration bonuses.”

“Extensions to the classical MAB problem include the contextual bandit, where additional information (context) is available at each decision point [4], and the adversarial bandit, where rewards are determined by an adversary rather than a fixed distribution [5].”

“First, the acceptance probability model be made more context-aware (route intent, time-of-day, and local supply – demand regime), and the scoring function can be optimized in a multi-objective way using online learning while preserving stability constraints [1], [6], [14].”

“LinUCB [17] introduced linear reward models with upper confidence bounds, and Thompson Sampling [18] provides Bayesian exploration-exploitation approaches.”

“Reinforcement learning provides a general framing for sequential decision-making, while contextual bandits focus on action selection using observable context and near-term reward signals [5], [6].”

“Then we have where δ ∈ (0 , 1) and α = 1 + log(4 /δ ) / 2 [11], [41].”

“2) NeuralUCB-based split-point selection: Contextual ban-dit methods such as LinUCB [13] assume a linear reward model and thus too restrictive for the highly nonlinear cost in heterogeneous SFL, while KernelUCB [14] partially captures nonlinearities but suffers from kernel selection and quadratic…”

“DK acknowledges funding from the Swedish Strategic Research Foundation under the Future Research Leaders program (Grant FFL24-0111), the Swedish Research Council under the Starting Grant program (Grant 2025-05421), and the Swedish Knowledge Foundation (Contract 20210016). impact on the user [1].”

“The individual components have well-studied guarantees: FedAvg-style aggregation [1], [3], LinUCB with O ( √ T log T ) regret [22], and TS with near-optimal regret [35].”

“…an inner product of an unknown parameter θ ∗ (i.e., context vector) and the action vector x t plus noise [Abbasi-Yadkori et al. , 2011], offering a simple yet general approach to model sequential decision-making in recommendation systems and adaptive control [Li et al. , 2010; Chu et al. , 2011] .”

“…underlies fatigue detection, frequency capping, and exploration scheduling, drawing on multi-armed bandits [Auer et al., 2002, Thompson, 1933, Li et al., 2010, Lattimore and Szepesvári, 2020], contextual bandits [Agarwal et al., 2014], and curiosity-driven approaches [Chen et al.,…”

“…estimates, and this need spans a broad set of application domains, including recommender systems [Su et al., 2024, Joachims et al., 2018, Li et al., 2010], content moderation [Avadhanula et al., 2022], clinical decision support [Durand et al., 2018, Mintz et al., 2020, Esteva et al.,…”

“Bandit algorithms offer a principled framework for decision-making under uncertainty, with strong theoretical guarantees (Li et al., 2010; Chapelle and Li, 2011; Bouneffouf et al., 2020; Yancey and Settles, 2020; O’Brien et al., 2022), but are often too inefficient for personalization (Kinyanjui et…”

“The update rule follows the standard ridge-regularized LinUCB form for contextual bandits [26].”

“While multi-armed bandit (MAB) is a standard model for sequential decision-making [Lai and Robbins, 1985, Auer et al., 2002, Villar et al., 2015, Li et al., 2010, Bouneffouf and Rish, 2019], the feedback one ultimately cares about is often expensive to obtain.”

“These methods have been successful in recommendation systems [7] and adaptive sequence of curriculum [17].”

“…design is related to confidence based exploration methods in bandits and reinforcement learning, especially optimism-style methods that leverage uncertainty es-12 timates to balance exploration and exploitation Abbasi-Yadkori et al. (2011); Li et al. (2010); Zhang et al. (2025); Li et al. (2025).”

“D-PSGD cannot participate in the sparse heterogeneous or geographic settings due to mixed-architecture nodes, so the comparison is genuinely setting-specific rather than a 8 LinUCB [17] Reward + exploration 0 .”

“However, real-world environments are inherently non-stationary [7, 8].”

“…demand while repeatedly posting prices [Besbes and Zeevi, 2009, den Boer, 2015], with contextual variants incorporating buyer, product, or market features [Li et al., 2010, Chu et al., 2011, Abbasi-Yadkori et al., 2011, Agrawal and Goyal, 2013, Ban and Keskin, 2021, Javanmard and Nazerzadeh, 2019].”

“This phenomenon naturally arises in reinforcement learning [Wai et al. , 2018; Doan et al. , 2020], recommendation systems [Li et al. , 2010], and distributed data collection [Lopes and Sayed, 2007; Duchi et al. , 2011; Chen et al. , 2014; Madry et al. , 2017; Shah and Avrachenkov, 2018], where…”

“Remark 2: If an algorithm chooses policies with either of the following conditions , where ( i t , j t ) is the pair played at time t , then the algorithm will incur regret Ω( ϵT ) .”

“Contextual bandits are widely used for utility-driven ranking from logged interactions [6, 18], and memory-augmented contextual bandits are suitable when user feedback is sparse [26].”

“Strong regret bounds were established in [3], and the linear bandit model has since been widely deployed in online recommendation [4], web advertising [5], and economics [6].”

“To overcome this limitation, we try to introduce Contextual Combinatorial Multi-Arm Bandits ( C 2 MAB ) framework ( [18]).”

“This paper formulates online model selection as a contextual multi-armed bandit (CMAB) problem [10]–[12].”

“Contextual bandits provide a formal framework for learning action selection from contextual information with reward feedback, and have been used in a range of recommendation and decision systems [16], [17].”

“Contextual bandits with linear value functions, in particular LinUCB and its disjoint variant [6, 20], are the closest classical analogue of our linear scorer over workspace features.”

“Contextual bandits generalize the framework by conditioning decisions on observed features [17, 1], enabling personalization.”

“By leveraging the LinUCB algorithm [16], the proposed approach dynamically balances exploration and exploitation, enabling the system to learn effective retry strategies over time and adapt to evolving demand patterns and operational conditions.”

“Educational deployments also face small-sample constraints and limited interaction budgets [8].”

“Contextual bandit methods provide a mature technical route for local adaptive decision-making based on limited feedback [9].”

“Early formulations appeared in [20,23]; however, the Exp4 method employed in this work was introduced in [6] and can in fact be viewed as a contextual bandit method where the side information is presented in the form of experts.”

“Reinforcement learning and contextual bandit methods provide principled mechanisms for adaptive decision-making [14], [16].”

“To address environmental influences specifically, linear contextual bandit frameworks are widely utilized to map external factors to response probabilities [11].”

“These predictions are injected into Disjoint LinUCB [Li et al., 2010] as weighted pseudo-observations , where the weight w t ∈ [0 , 1) is determined by a pluggable decay schedule .”

“All three applications are instances of linear bandits[10, 19, 37], so a natural starting point may be to compare the regret of our algorithm with that of a linear bandit style algorithm which, in each step, uses the bandit feedback to estimate problem parameters (eg. mean job sizes for completion…”

“✓ – DRL – Li et al. [24] 2010 Bandits – – CB – Zhou et al. [26] 2020 Neural Bandits designated to host the control-plane.”

“This enables the system to refine its decision strategy through multiple interactions with the learner [6], [9].”

“This simplicity has enabled broad adoption across domains including recommendation systems (Li et al., 2010), clinical trials (Villar et al., 2015), and hyperparameter optimization (Li et al., 2018).”

“…are particularly attractive in this setting because they explicitly account for uncertainty: reviewers with wide posterior uncertainty are explored cautiously, while consistently reliable reviewers are prioritized when stakes are high (Auer, Cesa-Bianchi, and Fischer 2002; Li et al. 2010).”

“This work makes the following contributions: • We introduce exploration saturation [18, 22, 30] as a user-dependent, behavioral phenomenon that characterizes the limits of fairness-and novelty-driven recommendation, re-framing exploration as a bounded, user-experienced process rather than a…”

“One of the most popular contextual-bandit algorithms is LinUCB, proposed by Li et al. [25].”

“The goal is to learn a policy that maps contexts to arms in order to minimize regret over time [7], [12].”

“Common approaches include ϵ -Greedy [5], Linear Upper Confidence Bound (LinUCB) [4]–[6], Thompson Sampling (TS) [7], and variants such as Bootstrapped TS (BTS) and Weighted BTS [8].”

“Prominent examples include dependent bandits in [4], X-armed bandits in [5], linear bandits in [6], contextual bandits with side observation in [7], and combinatorial bandits in [8].”

“Most PE techniques focus on estimating a user’s preference based on previous data, such as news articles or food recommendations, using contextual multi-armed bandits (Li et al., 2010) and Gaussian processes (Bonilla et al., 2010).”

“Exemplified by methods such as LinGreedy [24] and LinUCB [26], CMABs extend traditional bandits by incorporating contextual state representations, allowing incremental algorithms to leverage environmental information when making recommendations.”

“Unlike static policies, Xrbit employs a LinUCB agent that learns to navigate the trade-off between latency, energy, and heat in real-time [12].”

“Online decision-making problems, notably multi-armed bandits (MAB) [2] and stochastic linear bandits (SLB) [3], constitute a fundamental class of sequential learning tasks with wide applications in recommendation systems [4], wireless communications [5], online advertising [6], and adaptive control…”

“Specifically, we employ a contextual bandit formulation [12], where feasible actions are ranked using context-dependent utility estimates and confidence bounds.”

“Reinforcement learning and contextual bandits provide principled tools for allocating limited computation based on observed context [16, 19, 38].”

“The implementation of LinUCB and LinTS was obtained from a package called striatum , while the implementation of NeuralUCB and NeuralTS was obtained directly from the GitHub repository of the authors.”

“In online advertising systems [Li et al., 2010], click fraud introduces adversarial corruption [Lykouris et al., 2018], while user-interaction signals can also exhibit heavy-tailed behavior [Jebarajakirthy et al., 2021].”

“All algorithms are adapted to linear reward models, where rewards are assumed to follow a linear function of the context features, a common assumption in contextual bandit research due to its interpretability and computational efficiency [1].”

“In recent years, big data-driven prompt learning has provided a new approach to addressing these challenges [16].”

“…the best action from the available choices based on the maximum potential reward that can be obtained for the current context while selecting an action randomly with a small fixed probability (ε), following an ε -greedy strategy, in order to explore the environment for future decision making [21].”

“It has a wide range of potential applications, such as recommender systems, and has been studied extensively from a theoretical perspective (Auer et al., 2002a; Cesa-Bianchi and Lugosi, 2006; Li et al., 2010).”

“LinUCB [14] maintains an upper confidence bound for the reward of each arm.”

“…methods [Auer et al., 2002, Beygelz-imer et al., 2011, Sen et al., 2018, Ito, 2024] as well as UCB-type approaches [Agarwal et al., 2014, Li et al., 2010] achieve sublinear regret using importance-weighted loss estimators, where the observed loss is scaled by the inverse of the…”

“Li et al. (2010) consider finite action spaces and build such confidence bound directly on the estimated regret for each possible action.”

“Our analysis adopts the common assumptions of the contextual linear bandit framework (Li et al., 2010; Abbasi-Yadkori et al., 2011; Liu et al., 2025c; Cheng et al., 2026) specialized to persistent prototypes and latent clusters.”

“Thecontextual multi-armed bandit problem is an extension of the traditional multi-armed bandit problem, with applications in recommendation [14], advertising [10], and more.”

“Within the research on recommendation systems, contextual bandits (CB) algorithms are commonly adopted as a lightweight framework for context-dependent decision making with an exploration-exploitation trade-off [28].”

“Conversely, modelling this environment as a Restless Multi-Armed Bandit (RMAB)—where worker fatigue and readiness are treated as hidden Markov states [5]—using statistical learning algorithms such as LinUCB [6] and Thompson Sampling [7] naturally balances the workload over time.”