A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“A model of transition between states through time steps is not considered here either; instead, a series can be viewed as an episode of making a decision [17]– [19].”

“A parallel line, contextual bandits, replaces per-arm means by a reward model shared across arms, linear [24], neural [25], tree-based [26], or, concurrently with this work, a prior-data fitted network [27]; all score a fixed enumerated action set and bound regret against its best arm, whereas our…”

“Selecting optimal stimuli for individuals can be framed as a contextual bandit problem [22], balancing exploration and exploitation [23].”

“Contextual bandits extend this framework by using side information, such as route distance, novelty, time of day, and rider history [18], [19].”

“LinUCB learns a linear model of the reward and chooses actions using an upper confidence bound [2], [3].”

“Contextual bandits give low-regret online decision policies with linear-model efficiency (Li et al., 2010; Agrawal and Goyal, 2013) and have been applied to sequential mobile-health decisions (Tewari and Murphy, 2017).”

“Among foundational contributions, Li et al. [9] introduced Lin-UCB for news recommendation, demonstrating effective personalization using user and item features.”

“Li et al. ( 2010 ) extended bandits to the contextual setting via LinUCB, with strong empirical results in news article recommendation.”

“Our specific contribution, EXP4-LoRA, combines LinUCB [4] for linear reward estimation with the EXP4 algorithm [5] for exponential-weight exploration, yielding a policy that converges in under 500 steps on mixed-domain benchmarks.”

“It is vast in scale, often containing tens to hundreds of attributes spanning diverse aspects [10, 24].”

“…are standard, namely the REINFORCE estimator for policy gradients [23], the contextual-bandit formulation in which each arm carries its own feature vector [24], and upper-confidence-bound action selection for the finite-armed bandit that governs operator choice in the executable study [25].”

“The multi-armed bandit model captures these problems in its simplest form: every action produces an immediate reward while also generating information for future decisions (Li et al., 2010; Gittins et al., 2011; Villar et al., 2015).”

“8 ) LinUCB [39, 40]: This approach formulates the compute allocation as a contextual multi-armed bandit problem.”

“Such online-learning models arise in a wide range of applications, including pricing [Cohen et al., 2020, Kleinberg and Leighton, 2003], advertising [Hu et al., 2026, Weed et al., 2016, Wen et al., 2025], and search and recommendation systems [Li et al., 2010, Wen et al., 2026b].”

“Our stochastic analysis builds on the OFUL framework of Abbasi-Yadkori et al. (2011) and the contextual bandit work of Li et al. (2010).”

“Reliance on logged data without any mechanism for exploration reinforces feedback loops, leading to “rich-get-richer” dynamics and suboptimal long-term performance [22, 27, 36, 38].”

“[13, 2] In this paper, we argue that historical data from completed A/B tests can be systematically leveraged to address the pre-deployment uncertainty problem of adaptive experimentation.”

“Such signals could support preference learning (Christiano et al., 2017), personalized ranking through contextual bandits (Li et al., 2010), and carefully bounded reinforcement learning (Sutton and Barto, 2018); personalized reranking is one practical first step.”

“After the loop of EM algorithm, sufficient statistics A , A f b and b f are updated (Lines 19-23), utilizing the closed-form solutions for online ridge regression (Li et al., 2010).”

“In this setting, multi-armed bandit methods are attractive because they support incremental online adaptation [3, 12, 15].”

“We develop a mechanism based on the Contextual Multi-Armed Bandit, specifically utilizing the LinUCB algorithm [12], to manage the exploration-exploitation trade-off involved in local optimization.”

“Initially, we explored linear-contextual bandit methods such as LinUCB, Hybrid UCB [26], and LinkUCBGlobal [27], but these showed limited performance, converging to accuracies well below α C due to non-linear relationships between contexts and actions.”

“It has broad applications in online experimentation (Scott, 2015), personalized recommendation (Li et al., 2010), adaptive treatment allocation (Tomkins et al., 2021), and portfolio selection (Huo and Fu, 2017).”

“We deliberately adopt a contextual-bandit formulation [14]: each epoch is treated as an independent decision conditioned on its context, rather than as one step of a fully sequential MDP.”

“It can be proven in [14] that, when the components of c t,n are independent given the corresponding rows of G t,n , and for any δ > 0 and x t,n ∈ R d , with probability at least 1 − δ , the following holds: where α = 1 + ln( 2 /δ ) / 2 is a constant.”

“Similarly bandit algorithms such as, ε - greedy [24], Upper Confidence Bounds [25], and Bayesian bandits [26] update action-selection strategies online but allow abrupt shifts in behavior following individual reward observations.”

“The deployable selector is related to multi-armed and contextual bandits [14], [15], [16], which also choose among discrete arms but do not include the swarm-refinement and execution semantics.”

“Selecting a strategy from the runtime context that the AP MLD can observe, such as the per-band SLD count and load, without explicit labels for the underlying environment, matches the contextual-bandit framework [5], for which LinUCB [6] is a natural candidate.”

“Exploration is bounded on purpose: the bandit may try alternative encoders only within a budget per feature and only on a shadow path that does not affect served embeddings, so a bad exploratory choice never reaches a production model [32].”

“This section provides a self-contained derivation of the LinUCB algorithm (Li et al. 2010) that underlies the worker selection mechanism of WILC.”

“LinUCB applies this framework to personalized recommendation (Li et al. 2010), while linear-bandit analyses and linear Thompson sampling characterize exploration and regret under linear reward models (Abbasi-Yadkori, Pál, and Szepesvári 2011; Agrawal and Goyal 2013; Abeille and Lazaric 2017).”

“We compare against several off-the-shelf algorithms: LinUCB [12], a contextual bandit with upper confidence bounds applied in cloud scaling [4] and used here with the disjoint-arm variant [7]; Proximal Policy Optimization (PPO) [21], which stabilizes policy gradient updates via a clipped surrogate…”

“Directly analogous to exploitation in bandit recommendation [31], this strategy surfaces items most likely to be positive.”

“The Upper Confidence Bound (UCB) (Li et al. 2010) algorithm is aimed to balance exploration and exploitation by estimating the confidence intervals of the expected rewards for each POI.”

“Inrecommender systems, exploration has been studied in contextual bandit form [16] and via off-policy correction for policy-gradient learners [8].”

“The system actively shifts recommendations away from mastered concepts toward new challenge areas, whereas collaborative filtering methods tend to perpetuate historical preferences regardless of evolving competencies 59 .”

“LinUCB is used as the basic online decision architecture [21,22].”

“LangSelect combines this observation with contextual-bandit machinery (Li et al., 2010; Zhou et al., 2020) and verification/fallback, making language choice explicit and empirically checked inside coding-agent workflows (Yang et al., 2024; Xia et al., 2025; Zhang et al., 2024; Bouzenia et al.,…”

“…used and efficient algorithms for linear contextual bandits [37] which computes confidence interval efficiently in closed form when the reward model is linear by assuming that the expected reward of an arm a is linear in its d-dimensional feature x t , a with some unknown coefficient vector θ [38].”

“…including clinical trials [15], wireless communication [14], high-dimensional decision and co-ordination problems ([16], [17], [11]) recommendation systems ([21], [4]), financial optimization [5], and economically motivated incentivized explorations ([13], [32], [22], [8], [10], [28]).”

“The modified model underpins classical algorithms such as LinUCB [24] and LinTS [1], which perform well in low-dimensional settings with light-tailed reward.”

“Applied and linear variants provide standard contextual benchmarks (Li et al. 2010; Chu et al. 2011), with confidence-set methods for linear stochastic bandits also central (Abbasi-Yadkori, Pál, and Szepesvári 2011).”

“Li et al. (2010) develop contextual-bandit algorithms for news recommendation, and Li et al. (2011) show that serving a uniformly random slice of traffic permits unbiased offline evaluation of any policy by rejection sampling on the logs.”

“Constrained LinUCB modeli ile de˘gerlendirilmektedir [12], [13].”

“…with expensive evaluations—whether in Bayesian Optimization (BO) for molecular design [Xie et al., 2025], contextual bandits for recommendation [Li et al., 2010], or engineering systems design [Cheon et al., 2024, Paulson and Tsay, 2025]—routinely combines an offline phase of uninformed…”

“…physician choosing among treatments, an online platform selecting product recommendations, or a policymaker allocating interventions may rely on predictions to probe the likely outcomes and guide their actions (Manski, 2004; Li et al., 2010; Athey, 2015; Gao et al., 2024; Feuerriegel et al., 2024).”

“These challenges are also prevalent in diverse domains, including healthcare [7], recommendation systems [8], games [9], and economic modeling [10].”

“AgentRank computes scores via confidence-weighted combination:”

“Using the previous reference values, we define the normalized cost and use its negative as the reward, a) Contextual Bandit Scheduler (LinUCB) [23]: The contextual bandit scheduler models function placement as a K - armed contextual bandit in which each worker corresponds to one arm.”

“Contextual bandit methods are well suited to repeated decision problems in which a controller must balance exploration and exploitation under uncertainty using only current-context information [8]–[10].”

“The scoring and parameter update procedures follow the contextual bandit formulation introduced in [12].”

“Learned ranking under a budget connects to contextual-bandit and log-anomaly work [17], [18], but the learner here is constrained and auditable.”

“Within the feasible set, RA-TTC selects online via a resource-constrained contextual bandit, LinUCB [17-18], 𝑚𝑚∈𝐴𝐴 ( 𝑡𝑡 ) where is the context of difficulty and resource state and is the exploration coefficient.”

“Contextual bandits (e.g., LinUCB) work for personalization [9]- [10].”

“To address the first challenge, we assume a linear reward structure, a common approach in personalized problems like article recommendation [Li et al., 2010, 2011] and healthcare [Tewari and Murphy, 2017].”

“…within the broader literature, five controllers are compared against the proposed framework : an equal-allocation Static baseline, classical Round-Robin , a pure-exploitation Greedy best-arm policy, a LinUCB contextual bandit [11], and a deep reinforcement-learning agent based on PPO [12].”

“…advanced with the emergence of neural and online-learning approaches, such as two-tower networks (He et al., 2017), attention-based models inspired by Neural Machine Translation (Zhou et al., 2018), contextual-bandit recommendation (Li et al., 2010), and Graph Neural Networks (Wang et al., 2019).”

“Li et al. [79] formalize personalized news recommendation as a contextual bandit balancing exploration and exploitation under per-user feedback, providing a canonical template for updating a generally trained system based on individual interaction.”

“Contextual Bandits has been successfully applied to many different domains including education ( [6]), health ( [7], [8]), tourism ( [9]), and digital marketing such as ad-placement and news recommendation ( [10]).”

“When the cetane number θ k is observable, the LinUCB algorithm [28, 29] is applied with disjoint linear models, where each discrete action u ∈ U maintains its own parameter vector.”

“31% on the Flickr and Face-book datasets, respectively, as k increases from 5 to 25 in SpillCB LinUCB : 0% .”

“…through Pr [ y t = k | x t ] = [ σ ( W ∗ x t )] k , where σ ( · ) is the multinomial logistic function: This framework captures a wide range of applications such as recommendation systems [Li et al., 2010, 2017], where different user responses to a recommended item are mapped to different rewards.”

“To evaluate the performance of the proposed CADCO framework, the following baselines were adopted for comparison: LinUCB [36], DKN [37], NAML [38], LSTUR [39], and NRMS [40].”

“Assuming the reward is a linear function of the context, i.e., r ( a k ) = θ ⊤ a c k , we can leverage the existing LinUCB algorithm [17].”

“The most widely used method for solving contextual ban-dit problems with continuous features is the Linear Upper Confidence Bound (LinUCB) algorithm, originally proposed by [7].”

“In contrast, online learning algorithms typically rely on optimism in the face of uncertainty (Li et al., 2010; Abbasi-Yadkori et al., 2011), which encourages exploration and achieves sublinear online regret.”

“To address this, we reformulate secrecy rate maximization as a multi-armed contextual bandit problem, a reinforcement learning framework suited for one-shot decision-making where actions do not alter future states [19], [20].”

“It has been applied with slight variations to different applications: from algorithmic pricing [Trovo et al., 2015], recommendation systems [Li et al., 2010], and decentralized networking [Appavoo et al., 2018], to revenue maximization in auctions [Croissant et al., 2020], to name just a few.”

“At each step t , under context x t , LinUCB uses the estimated ˆ θ a and A a to select an action using the following upper confidence bound (UCB) rule [10]: where α > 0 is a tunable parameter that balances exploration and exploitation of the environment.”