A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“For n ∈ S t , based on ridge regression [11], the estimated ˆ μ i ( t ) can be updated as follows: where B i ( t ) indicates the observed contextual information of edge node i .”

“K-LinUCB: LinUCB [3] is a contextual bandit algorithm that selects one arm in each round.”

“It is usually encountered in personalized advertising, recommendation systems, and medical, among other areas [3]–[5].”

“One intuitive modeling approach involves establishing a linear relationship between the action features and the expected rewards, known as linear contextual bandits [11], [12], [2].”

“The agent aims to maximize the cumulative reward over 𝑇 steps: A popular algorithm for this problem is LinUCB [13], which assumes the expected reward is approximately a linear function of the context.”

“Specifically, it estimate epistemic uncertainty via an information matrix (Abbasi-Yadkori et al., 2011; Li et al., 2010; Chu et al., 2011) in the learned representation space, and combine this uncertainty with the surrogate’s predicted performance to construct an acquisition score.”

“Online adaptation with feedback using methods such as contextual bandits [34] remains future work.”

“LinUCB [Li et al., 2010] introduced contextual bandits with linear reward models and confidence-based exploration.”

“This additive structure follows the hybrid linear upper-confidence-bound (LinUCB) algorithm [LCLS10].”

“The classical O ( d √ T log( T )) regret analysis of LinUCB/OFUL combines self-normalised concentration with the elliptical-potential lemma (Li et al., 2010; Abbasi-Yadkori et al., 2011).”

“…the effect of prompts sent at different moments (Klasnja et al., 2015, 2019), and contextual bandits, a method also used for news recommendation (Li et al., 2010), and reinforcement learning personalise which prompt to send and when (Tewari and Murphy, 2017; Yom-Tov et al., 2017; Liao et al.,…”

“Additionally, we compute the uncertainty associated with each candidate behavior configuration as [13] During the learning phase, we use uncertainty as an exploration bonus to encourage the algorithm to try behavior configurations for which it has less information.”

“The field subsequently shifted toward implicit behavioral feedback and ranking-oriented learning, incorporating pairwise ranking and contextual bandits [18, 24].”

“…outcomes, learning from logged feedback, and a value-based business objective—place post ad-click destination automation squarely in the setting of contextual bandits and policy learning [4]–[7], which learn decision rules from logged interaction data rather than predicting outcomes in isolation.”

“Since only the selected configuration is executed, the learner receives contextual-bandit feedback [14, 16].”

“It is architecture-based on PPO-based coordination [3], LinUCB contextual bandits [4], federated learning [7], SHAP-based explanation [8], Page-Hinkley drift detection [11], and FHIR R4 interoperability [12].”

“…terms of the spawn LLM-based judge the relative timing may differ substantially, potentially changing the effective spawning rate. and select actions, a variety of strategies can be constructed by bor-rowing ideas from reinforcement learning, bandit algorithms, and simple search heuristics [2, 33].”

“Under standard explore-exploit trade-offs (e.g., UCB-style policies [Li et al., 2010]), the effective score for a creator’s-content is: This produces a self-reinforcing feedback loop as shown in Figure 1.”

“Inspired by the exploration–exploitation principle commonly studied in recommender systems and bandit algorithms [8, 9], EAC introduces an exploration bonus that favors categories that are both underrepresented in the user’s profile and insufficiently covered in the recommendation list being…”

“In recommender system research, this matches the cold-start problem for new content: how to allocate the first exposure of new content between “exploration and exploitation” (Li et al., 2010; Wu et al., 2016).”

“…deployment budget cap (no search budget; Table 9); a fixed full committee; a rule-based regime router (rules in Appendix C); a LinUCB bandit [29] on a fixed 15-architecture menu over z k ; AFlow-style workflow search [14, 19]; MaAS/AutoMaAS-style adaptive selection [20, 9], which is the…”

“Bandit methods can adapt recommendations and online decisions based on observed user feedback (Li et al., 2010; Scott, 2015).”

“Both probing policies maintain separate statistics and use a context-conditioned, UCB-inspired selection rule (Auer et al., 2002; Li et al., 2010).”

“A segunda altera a política de decisão com exploração explícita [Li et al. 2010].”

“LinUCB [42].”

“In linear contextual bandits, the expected reward is assumed to be linear in the context features (Auer, 2002; Abe et al., 2003; Li et al., 2010).”

“The Multi-Armed Bandit routers (see Appendix B) each solve only part of the problem: Thompson Sampling ignores the query context, LinUCB and its Thompson variant rely on a linear reward model that breaks down under domain shift, and REINFORCE captures non-linear context but cannot use prior knowledge of which models s…”

“LinUCB (Li et al., 2010) maintains ( c t , V t , q t ) , initialized with V 1 = λI and q 1 = 0 .”

“…fixed dataset of past interactions (Aggarwal, 2016), or online , as a sequential decision problem with an exploration–exploitation trade-off, usually modeled as a (contextual) bandit (Agarwal et al., 2009; Agrawal and Goyal, 2013; Lattimore and Szepesvári, 2020; Li et al., 2010; Song et al., 2022).”

“Contextual bandits provide a framework for sequential decision-making in applications such as recommendation systems [24, 25], medicine [5, 7, 32], finance [31, 42], and operations research [4, 16].”

“In recommendation, for example, feedback is observed only for displayed items (Li et al. 2010).”

“It covers settings such as policy optimization and reinforcement learning [46, 47], contextual bandits and online recommendation [48], continual learning [49], and federated learning with time-varying client distributions [50].”

“The public release provides only the reduced user features as context and supports offline policy evaluation [1–3].”

“The underperformance of LinUCB and NeuralUCB highlights the complexity of the adaptive selection task.”

“In parallel, contextual ban-dits and reinforcement learning (RL) provide decision rules that balance exploration and exploitation using side information [10, 26, 37, 39].”

“We compare PC-LinUCB with exhaustive LinUCB (Li et al., 2010), rarely switching OFUL (RS-OFUL; Abbasi-Yadkori et al., 2011), CBSCFD (Chen et al., 2020), PSI-LinUCB (Shustova et al., 2025), and Sub-Elim (Yang et al., 2022).”

“Contextual-bandit algorithms learn from this partial feedback (Li et al., 2010; Zhou et al., 2020; Riquelme et al., 2018).”

“For category k in time step t , its extended context vector is defined as the splicing of original context and causal embedding: where [ ] denotes vector concatenation and 𝐞𝐞 𝑘𝑘𝑡𝑡 is the causal embedding produced by the encoder described in Sect.”

“Each episode of T =100 cycles generates domain-specific subtasks; a L IN UCB contextual bandit [30] proposes a human– AI collaboration mode, which the policy engine may override via caps, guards, or the HRS.”

“The search policy is a LinUCB variant (Li et al., 2010).”

“Linear bandits provide a canonical framework for sequential decision making with correlated rewards and have become a fundamental model for optimizing machine learning systems under uncertainty (Li et al. 2010).”

“The contextual-bandit rule of Li et al. [2010] selects interventions.”

“Contextual bandits learn action values from selected-action feedback using linear confidence bounds (Li et al., 2010; Abbasi-Yadkori et al., 2011), parameter sampling (Agrawal & Goyal, 2013), or neural confidence models (Zhou et al., 2020).”

“Transitioning to a contextual bandit [1, 10, 14] architecture will allow the system to incorporate real-time user state and historical preferences, tailoring the visual gateway to individual user profiles.”

“A diagonal LinUCB policy (Li et al., 2010; Chu et al., 2011) then selects an arm by combining predicted child-workflow quality with an uncertainty-driven exploration bonus.”

“…et al., 2022; Andrychowicz et al., 2021; Yu et al., 2026; Cui et al., 2025; Shen, 2025) to directed bonuses based on visitation counts (Lai, 1987; Li et al., 2010; Jin et al., 2020) or prediction error (Pathak et al., 2017; Burda et al., 2018), with recent extensions to language models (Bai et…”

“Contextual bandits use shared features to generalize from observed rewards and balance predicted performance against uncertainty (Li et al., 2010; Chu et al., 2011).”

“Sentence-BERT k-NN is slightly more accurate on 20 Newsgroups, LinUCB is best on Mushroom, and item kNN has the highest MovieLens accuracy.”

“We compare uniform allocation, best-static, the hand rule, the fixed feed-back controller, adaptive-KL (Ziegler et al., 2019), a contextual bandit (Li et al., 2010), NeuralUCB (Zhou et al., 2020), PBT (Jaderberg et al., 2017 Evidence for the three components.”

“The LinUCB algorithm, a representative contextual bandit method, assumes that the expected reward is a linear function of the context features and adaptively learns the feature weights through online updates [1, 18].”

“…contextual bandit and constrained reinforcement learning approaches that incorporate participant-specific information, biomarker trajectories, and multimodal physiological signals to further advance personalized neuromodulation while maintaining clinically defined safety constraints [20], [27].”

“We evaluate three deterministic baselines: no retrain, fixed interval, and threshold persistence, against one learned policy (LinUCB) [12].”

“Christakopoulou et al. [13] are the first to apply this strategy to address the user cold start problem, using a multi-armed bandit algorithm [47, 66] to balance exploration and exploitation.”

“Baselines We compare TDGE-LinUCB, TDGE-NeuralUCB, and TDGE-NeuralTS with their corresponding item-level backbones: LinUCB [15], NeuralUCB [37], and NeuralTS [35].”

“Following the contextual-bandit formulation (Li et al., 2010), we assume that the nonstationarity of f t is explained by a latent covariate state z t , whose space is denoted as Z .”

“Contextual bandits use observed features [2, 26], while clustered variants share information through inferred groups [32, 47].”

“This leads to the direct method (DM) (Li et al., 2010), defined as (cid:98) V DM ( π ) = 1 /N =1 (cid:98) g ( (cid:101) z j ) .”

“Sequential formulations made intervention, exploration, and delayed reward explicit [14, 18, 30].”

“Bandit methods have been extensively discussed in the retrieval and ranking context [1, 3, 12, 17].”

“Algorithm 2 UCB Algorithm, adapted from [3,6] 1: Data: Observe Y t 8: V t ← V t − 1 + a t a ⊤ t 9: Proof.”

“[1] proposed the LinUCB algorithm and successfully applied it to personalized recommendation.”

“Contextual bandits model sequential decision problems in which the learner observes context-dependent actions and receives feedback only for the selected action [1, 13, 15].”

“Online-learning formulations with context model reward conditioned on observed context [28,49], while restless and recovering formulations model rewards that evolve over time [38,53].”