A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“We remark that this is a common practice in the linear bandit literature [LCLS10, CLRS11], and the regret bound for the consequent algorithm is typically loosen by a factor of √ d [AYPS11].”

“CA-RAG is conceptually related to contextual bandit formulations in which queries are contexts and bundles are actions [10], [11].”

“Optimizing a black-box function with costly and noisy evaluations is a common problem, and has wide applications such as hyperparameter tuning (Snoek et al., 2012), A/B testing (Kohavi et al., 2009), recommendation systems (Li et al., 2010), and reinforcement learning (Lizotte et al., 2007).”

“Due to the introduction of context information, contextual bandits have been adopted widely in various areas, such as news article recommendation (Li et al., 2010), personalized medicine (Tewari and Murphy, 2017), design of clinical trials (Villar and Rosenberger, 2018), and advertisement…”

“Stochastic contextual bandits provide a mathematical framework for sequential decision making under uncertainty, with applications ranging from personalized recommendation systems (Li et al., 2010; Qin et al., 2014) to adaptive clinical trials (Durand et al., 2018; Villar et al., 2015).”

“In healthcare and related domains, offline contextual bandits have been used to model decision-making under uncertainty, where the objective is to optimize top-𝑘 utility rather than predictive accuracy [27], [33].”

“Based on this idea, we propose Symphony-Coord, a decentralized multi-agent framework that models agent selection as an online contextual bandit problem [Li et al., 2010, Lattimore and Szepesvári, 2020].”

“This problem can be naturally formulated as a contextual bandit problem [16].”

“The detailed derivation of V a,t follows the LinUCB formulation presented in [29].”

“For pricing, we consider: (c) static pricing (STP), which posts a fixed price; (d) random pricing (RDP), which samples uniformly from a price range; (e) linear pricing (LinP), which fits a linear response curve [76]; and (f) convex pricing (ConP), which models smoother but non-linear responses [77].”

“Within this class of methods, one of the most popular algorithms is the well-known LinUCB [12].”

“…adaptive clinical trials, contextual bandits, batch bandits, sequential policy learning, and dynamic pricing (Lattimore and Szepesvári, 2020; Garivier and Kaufmann, 2016; Chow and Chang, 2011; Zenati et al., 2022; Li et al., 2010; Perchet et al., 2016; Zenati et al., 2023; Qiang and Bay-ati, 2016).”

“News [7, 31, 35].”

“(Li et al., 2010) 4.”

“Examples include weather patterns that affect navigation, patient histories that shape medical outcomes, and evolving user preferences in recommender systems Li et al. (2010).”

“It decides whether to use another subflow or not when the fastest subflow is window-limited by leveraging LinUCB algorithm [37] in contextual MAB theory [38].”

“Despite the diverse applications of CB (Li et al., 2010), we are unaware of any previous research applying CB to implement dynamic trust calibration.”

“In this work, the reward is defined as the number of relevant documents identified in the screened set, consistent with prior work in active learning [33,34].”

“Based on contextual MAB, we adopt LinUCB [29] to derive how FPSe-lector deterministically selects an offloading path, considering: 1) LinUCB empowers FPSelector with direct generalizability without requiring pre-training and converges quickly to react to the current path and edge status; 2) LinUCB…”

“The seeker explicitly states admiration for Johnny Depp, especially in Alice in Wonderland (2010) .”

“In the early stages of news recommendation research, scholars predominantly relied on hand-crafted feature engineering to construct user profiles [Li et al. , 2010; Garcin et al. , 2013; Son et al. , 2013].”

“The multi-armed bandit (MAB) problem is a framework [33] for sequential decision-making under uncertainty, with applications in search engines ([29]), clinical trials ([13], [2]), recommendation systems ([3], [22]), and cognitive radio networks ([12]).”

“Causal reinforcement learning presents challenges in terms of scalability and convergence, especially when applied to online recommender systems [22].”

“Bandit algorithms are widely used to solve online decision-making problems such as personalized recommendation Li et al. (2010a) and digital advertising Li et al. (2010b).”

“It has been widely applied across domains such as online advertising and product recommendations (Li et al., 2010; Schwartz et al., 2017), education and tutoring systems (Clement et al., 2015), and healthcare and clinical trials (Tewari and Murphy, 2017; Villar et al., 2015).”

“This is usually approached from the objective of maximizing the expected return, which has led to impressive successes in games (Silver et al., 2018; Vinyals et al., 2019) and content recommendation (Li et al., 2010).”

“Such structure enables efficient generalization and has been successfully applied across a wide range of domains, including clinical trials and precision medicine (Lu et al., 2021) to recommender systems (Li et al., 2010) and inventory control (Jin et al., 2021a).”

“NTRL (Figure 1) is a contextual bandit [19] agent that is trained in a simulated environment to maximize a hand-crafted reward function that promotes long combats with high damage dealt to the party while avoiding a total annihilation of the players, causing the end of the game session.”

“At each iteration 𝑡 , players independently choose an arm Δ from a based on action selection strategies like 𝜖 -greedy or UCB [4, 25, 31, 48], with experiments showing that 𝜖 -greedy is more efficient and effective.”

“Offline RL focuses on learning effective policies solely from a pre-collected behavior dataset and has demonstrated significant success in practical applications [29, 31, 24].”

“Specifically, the parameter θ h is updated in Line 9 using its closed-form solution (Li et al., 2010), while the extraction network ϕ h ( · , · ) remains fixed.”

“For LinUCB, α was searched over (0.01, 0.1, 1, 2, 4, 10) and for NeuralUCB we optimized the network hidden size (16, 32, 64, 128, 256, 512), ν (0.1, 1, 10), λ (10 − i , i = 1 , 2 , 3 , 4), and the learning rate (2 × 10 − i , 5 × 10 − i , i = 1 , 2 , 3 , 4), as in Zhou The motor imagery features were extracted from the…”

“This problem has been extensively studied and finds applications in diverse domains, including healthcare systems [13, 36], recommender systems [19, 61], and cyber-physical systems [62, 110].”

“0 PDT, no tuning 0.0 ± 2.1 6.3 ± 0.8 8.3 ± 0.6 Hill-climbing 5.8 ± 3.8 7.9 ± 1.6 6.2 ± 4.0 ZORankSGD -0.6 ± 30.9 4.4 ± 16.7 3.1 ± 22.1 ϵ -greedy Ψ 9.0 ± 0.6 9.4 ± 0.3 9.6 ± 0.2 UCB Ψ 9.2 ± 0.5 8.8 ± 0.9 8.8 ± 0.9 TS Ψ 9.7 ± 0.1 9.7 ± 0.2 9.4 ± 0.5 ϵ -greedy 8.9 ± 0.5 9.5 ± 0.3 8.9 ± 0.7 UCB 9.4 ± 0.5 9.5 ± 0.3 9.3 ± 0…”

“This modeling framework has broad real-world applications, including domains such as clinical trials [4, 5], recommender systems [6], and resource allocation problems [7].”

“This algorithm uses deep neural networks for high dimensional and nonlinear features combined with exploration strategies (e.g., UCB, Thompson sampling, etc.)[9-11].”

“Many other policies have been proposed to address these varying conditions (Agrawal, 1995a; Li et al., 2010; Yue et al., 2012; Besbes et al., 2014).”

“Figure 6a benchmarks TS-Conf’s convergence across distinct feature dimensions, with d values set to [5, 10, 15, 20], all the while maintaining a consistent noise variance of σ 2 = 1 .”

“The classical CMAB algorithm, disjoint Linear UCB (LinUCB) (Li et al., 2010) assumes a linear relationship between the context embedding x t and the reward E [ r t,a | x t ] = x ⊤ t θ a .”

“In Li et al. [41], the authors used a contextual bandit algo-rithm, termed LinUCB, to improve news article recommendation.”

“Upon the advancements of deep learning methodologies, Rein-forcement Learning (RL) has witnessed a substantial surge in its prevalence, ranging from robotics [11], strategy games [25] and recommendation systems [21].”

“R6 Li et al. [2010], and Kuaishou Gao et al. [2022b] releases a large random traffic dataset with over 300 million rows.”

“RL algorithms, often inspired by biological learning systems [1], eschew the reliance on pre-labeled data typical of Supervised Learning.”

“The stochastic contextual bandit problem has been widely studied in the literature (Bubeck et al., 2012; Lattimore & Szepesv´ari, 2020), with broad applications in healthcare (Bastani & Bayati, 2020), personalized recommendation (Li et al., 2010), etc.”

“Bandit and (online) reinforcement learning (RL) algorithms are commonplace in recommendation ( e.g. , Li et al., 2010) and revenue management ( e.g. , Kallus and Udell, 2020, Qiang and Bayati, 2016), where adapting decisions means we are continually improving our recommendations or pricing while…”

“Bandit problems have significant implications across diverse real-world applications, such as recommendation systems [17] and clinical trials [23].”

“As a promising branch to solve the MAB problem, UCB-type algorithms calculate the confidence bound of all arms and select the arm that has the largest upper confidence bound[5, 10, 8, 18, 22].”

“Li et al. [45] proposed a LinUCB algorithm, which models rewards as a linear function of actions and context and then selects the optimal action based on the UCB principle.”

“√ Li et al. (2010) proposed LinUCB with ˜ O ( d T ) regret bound and applied the al-gorithm to news recommendation.”

“For instance, we have h ( x , θ ) = x ⊤ θ in linear contextual bandits [28], [30], [31], and h ( x , θ ) = µ ( x ⊤ θ ) for generalized linear bandits [32]–[35], where µ ( · ) is a link function.”

“LinUCB maintains linear models for actions, adjusting them with observed features and rewards.”

“It has a wide array of applications, such as crowdsourcing [2], recommendation systems [3], sponsored search auctions [4], service procurement [5], etc.”

“Contextual multi-arm bandit frameworks are particularly beneficial in non-stationary environments—dynamically changing scenarios (Li et al. 2010)—with small action spaces.”

“…into two paradigms: online learning, where the policy can constantly interact with the environment and make responsive update to real-time feedback (Li et al., 2010; Dudik et al., 2011; Agarwal et al., 2012, 2014); and offline learning, where the policy is learned from the data collected in…”

“Recent approaches have extended the success of contextual bandits in news and Ad recommendations [18, 9] by modeling the problem as a Markov decision process and solving it using deep reinforcement learning [28].”

“…(Shahriari et al., 2015), the improved data protection of federated bandit optimization makes it a better choice for applications involving sensitive data, such as recommender systems (Li et al., 2010), clinical trials (Villar et al., 2015) and sequential portfolio selection (Shen et al., 2015).”

“The five features above describe each candidate, and the reward measures the novelty of the resulting framework behavior.”

“A prominent example is the linear contextual bandit, in which rewards are linear in context–action features (Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013).”

“We solve our problem as a contextual bandit (Langford and Zhang, 2008; Li et al., 2010; Latti-more and Szepesvari, 2019) with K experts, where each LLM is an expert and the context is an embedding of the prompt.”

“The companion GDCB framework [Miroshnichenko, 2026a] introduces a supervised context scaler δ θ ( x ) : X → R > 0 that is calibrated by ridge regression independent of the bandit’s posterior update [Li et al., 2010, Agrawal and Goyal, 2013, Hong et al., 2021].”

“However, these approaches often require complex cost-sensitive classification oracles to maintain computational efficiency (Li et al. 2010).”

“LinUCB (Li et al., 2010): Linear reward models with UCB.”

“We evaluate LinUCB (Li et al., 2010) and ACRouter’s orchestrator (Zhou et al., 2026).”

“…given contextual features x i,a,t describing the potential interaction between agent i and peer a , agent i selects a peer according to the LinUCB [20] rule: where the first term estimates the expected reward of querying peer a , and the second term is an uncertainty bonus that encourages…”

“…formulated as a contextual multi-armed bandit [8] where each arm corresponds to a specific intervention (query rewriting, retrieval depth or mode adjustment, cross-encoder reranking, or index refresh), and a LinUCB policy [9] learns to balance expected factual recovery against computational cost.”

“Adaptive thresholds are updated via OGD [22] ( θ t +1 = proj[ θ t − η t g t ] ) or LinUCB [11] .”

“2: The regrets of SME-OFU, OFUL [7], and LinUCB [12] under uniform reward noise (Figure 2( Writing we can conclude that with probability no less than 1 − δ , This completes the proof of Theorem 1.”

“The contextual bandit framework is a canonical abstraction for these settings (Li et al., 2010).”

“If the loss is l n ( u, z ) = z , but the actions enter the scalar parametric model nonlinearly, y n = a n ( u n ) ⊤ θ ⋆ + v n , the problem reduces to the contextual bandit problem (Li et al., 2010).”

“Contextual bandits (CB) have become a dominant paradigm for personalized medicine due to their ability to map patient features (contexts) to treatments (actions) with strong regret guarantees [6] [7].”

“Contextual bandits [2] extend the traditional MAB model by incorporating rewards dependent on context and selected actions.”

“Future work includes contextual routing to select models/guardrail settings based on case features [12], learned calibration of weights w j in Eq.”

“The resulting low-dimensional structure enables efficient generalization and has seen wide application, from recommender systems (Li et al., 2010) and inventory control (Jin et al., 2021a) to clinical trials and precision medicine (Lu et al., 2021).”

“Contextual bandits and online learning study sequential learning from partial feedback, including large and structured action spaces (Li et al., 2010; Abbasi-Yadkori et al., 2011; Krishnamurthy et al., 2016; Zhu et al., 2022), but typically model rewards directly over context-action pairs rather…”

“To achieve online adaptivity under environmental drift, we formulate the UIS configuration selection as an online optimization problem through the lens of contextual multi-armed bandits (Li et al., 2010).”

“…e.g. , users, patients, customers, or regions) across many randomized occasions, and a range of modern experimental designs produces data of precisely this form, including mobile-health trials, sequential A/B tests, contextual bandit experiments, and switchback experiments [LCLS10; JKPW22; BSZ23].”

“A key contribution was [12] (2010), in which the LinUCB algorithm was proposed for personalised news article recommendation (Yahoo!”