A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“The primary challenge lies in the dynamic nature of user-item interactions, where both the context and underlying user preferences continuously evolve, creating what is known as the "exploration-exploitation dilemma" in recommendation settings [2].”

“…efficient way to balance exploration and exploitation, but they lack mechanisms to exploit spatial or semantic structure in high-dimensional data (Li et al., 2010; Bubeck & Cesa-Bianchi, 2012), which is critical in in settings like geospatial search, with structured and highly correlated…”

“Contextual variants [1, 24] extend this framework to settings where the reward distribution depends on observed contextual features.”

“These challenges are related to moving-target problems in RL, studied in contextual bandits [17], input-driven environments [18], and state-dependent baselines [19].”

“This constitutes a concrete implementation of a structural prior → test → revised prior mechanism: the hypothesis is the factor predictor, the test is a rolling-window predictive check, and the revised hypothesis is represented by switching or expanding model class when the hypothesis fails.”

“At the decision level, BRIGHT employs a linear contextual bandit with Bayesian Thompson sampling [17], [18] to select, at each decision epoch, from a small set of monitoring templates, each specifying a total monitoring budget and class-dependent base probing periods.”

“On the other hand, multi-armed bandit techniques [75] are lightweight enough [24, 54], but make too many assumptions about the relationship between features and rewards [38].”

“In contrast, PLFair and MMF demonstrate comparatively weaker effectiveness.”

“Cold-start exploration is one of the most persistent challenges in large-scale recommender systems [7, 11].”

“Linear bandits have many practical applications, such as personalized news recommendations (Li et al., 2010), medical treatments (Lansdell et al., 2019), online advertisement (Tian, 2025) and stock trading (Ji et al., 2024), to name a few.”

“The success of LinUCB in a large-scale industrial application provided landmark evidence of the practical value of bandit algorithms, catalyzing a powerful feedback loop.”

“OFU is commonly used in bandits such as the Upper Confidence Bound (UCB) family of algorithms [2, 17, 38].”

“Its applications encompass a wide range of domains, including recommendation systems (Li et al., 2010, 2011), auction de-sign (Avadhanula et al., 2021; Nguyen, 2020; Gao et al., 2021), resource allocation (Fu et al., 2022; Verma et al., 2019), and network routing and scheduling (György et al.,…”

“Li et al.[10] pioneered large-scale applications with LinUCB for personalized news recommendations.”

“In practice, the added complexity of more advanced methods such as SupLinUCB is often unnecessary, a conclusion supported by both prior theoretical and empirical studies (see e.g., Li et al. 2010, Chu et al. 2011).”

“To model the arm-specific rewards, we employ the standard disjoint context encoding [3]: the agent observes a set of arm features { x t,a } a ∈{ 0 , 1 } where x t, 0 = [ x t , 0 ] and x t, 1 = [ 0 , x t ].”

“This bandit setting is popularly known as contextual bandits ([9], [10], [11], [12], [13], [14], [15], [16], [17]) and has many real-life applications in areas like online advertising, online recommendations, and e-commerce.”

“Contextual MAB has exhibited excellent performance in recommendation systems, such as the LinUCB algorithm by Li et al. [41], which utilizes user features for personalized recommendations.”

“Adaptive Routing with Contextual Bandits: The Persona Coor-dinator frames the selection of both personas and coordination protocols as a contextual decision-making problem [1, 26].”

“LinUCB algorithm uses the idea of "optimistically estimating underexplored options" and combines context features to calculate confidence intervals, which is widely used in news recommendation and advertising [1].”

“The contextual bandit variant enriches this model by incorporating side information, such as user or item features, to guide action selection (Li et al. 2010).”

“…demonstrated substantial improvements in recommendation system performance, with LinUCB achieving click-through rate improvements of 12.5% over context-free methods and consistent performance gains across different pool sizes ranging from 10 to 250 articles in news recommendation scenarios [5].”

“[7] evaluates bandit algorithms in a contextual setting, showcasing their capabilities in sparse data conditions.”

“In the linear CMAB setting, the expected reward of each a is assumed to be a linear function of the context, enabling efficient algorithms such as LinUCB and Thompson Sampling with provable regret bounds [15].”

“The Linear Upper Confidence Bound (LinUCB) algo-rithm is selected as the foundation due to its computational efficiency, theoretical performance guarantees, and its proven ability to effectively balance the exploration-exploitation trade-off in real-time systems [17].”

“For example, studies such as [9]–[11] applied bandits on the Yahoo and MovieLens datasets to recommend news articles and movies.”

“Furthermore, the essence of embodied recommendation is a sequential decision-making process, where agents must balance between ”exploiting” existing knowledge to recommend items of known user preference and ”exploring” unknown domains to discover users’ potential interests [7].”

“The analysis includes state-of-the-art algorithms— Upper Confidence Bound (UCB) [14], Linear UCB (Lin-a UCB) [15], and Optimal Sampling for Unimodal Ban-dits (OSUB) [16]—as well as a newly proposed lightweight contextual algorithm, the Epsilon-RMSProp Linear Ban-dit (E-RLB), which combines…”

“• LinUCB [20] • Random Bidding • Static Bid Price • Deep Q-Network (DQN) [23] • Actor-Critic RL The results indicate that CAB-DRL consistently outperforms all baseline models, achieving a +14 .”

“Unlike sequential reinforcement learning, the dataset consists of single-turn interactions, making the problem more appropriately modeled as a contextual bandit [2], [11] with multi-objective feedback [8], [16].”

“Conventional collaborative filtering models (which assume a relatively static item catalog) struggle here [5].”

“Computationally, the task of course selection can be modeled as a contextual multi-armed bandit problem [6], [7].”

“Bayesian approaches, such as online changepoint detection [7], infer regime changes by modeling uncertainty in transition probabilities over time.”

“In constructing the context vector in Algorithm 1, we use the metrics provided in Table 1 to define a five-dimensional state context vector, as illustrated in Equation (6), with the following five aspects that influence path discovery: Path prediction for state ( PP ) indicates a state’s ability to discover unique pat…”

“Contextual bandits have been widely studied and applied in various domains, including personalized recommendation systems Li et al. (2010); Yang et al. (2020); Guo et al. (2020), dynamic pricingMisra et al. (2019); Mueller et al. (2019), and healthcare Woodroofe (1979); Mate et al. (2020).”

“(2) Under standard linear–stochastic assumptions, with probability at least 1 – δ , so the confidence radius is The decision index is [3].”

“(Li et al. 2010) It operates under a limited trust budget, allowing controlled exploration while maintaining consistency and reliability.”

“To balance exploration and exploitation, we use an optimistic estimate of service quality by computing an upper confidence bound (UCB) (derived in [17], [18]) of the service quality (see Line 4 in Algorithm 2): where α i,t = σ log √ λ and δ denotes the confidence level.”

“In both linear and GLM settings, deterministic methods based on optimism in the face of uncertainty (OFU) [7, 38, 1, 39] and randomized approaches such as Thompson Sampling (TS) [10, 6, 3] or Perturbed History Exploration (PHE) [31, 32, 35] have been extensively studied.”

“Contextual-multi-armed bandits (CMAB) LINUCB style Li et al. (2010) : At the beginning of trial t , the agent observes a context x t .”

“Randomized control trials (RCTs), or A/B tests, are a fundamental tool for assessing the effectiveness of interventions across multiple disciplines, including healthcare[6], social sciences [7] and e-commerce platforms[29].”

“Owing to their simplicity and flexibility, contextual bandits have been widely applied in practice, including personalized recommendations (Li et al., 2010), mobile health (Nahum-Shani et al., 2016), and online education platforms (Cai et al., 2021).”

“A widely studied and practically successful instance is the linear contextual bandit model [Lang-ford and Zhang, 2007, Li et al., 2010].”

“Recent work has extended the MAB framework to incorporate richer modeling assumptions, including contextual information [Li et al., 2010] and hierarchical structures [Martin et al., 2022].”

“Large-scale digital platforms have further adapted contextual bandits and RL to optimize user engagement [22 – 24].”

“Once the root cause is identified, CloudHeal dynamically selects self-healing actions using a Contextual Multi-Armed Bandit (CMAB) algorithm [11], allowing continuous policy adaptation based on real-time system feedback.”

“Online learning is a topic with rich connections to several fields of interest and many real-world applications such as portfolio selection [26], personalized recommendation systems [27], and online advertising [30]; see Cesa-Bianchi and Lugosi [8], Hoi et al. [19] for a more comprehensive overview.”

“For example, LinUCB [7] treats each item as an arm, recommending an item based on contextual information.”

“evaluate the performance of an algorithm or a policy using historical Work done during an internship at RBC Borealis.”

“Contextual bandit framework [1] has been applied in a wide range of modern applications, such as online advertising [2], personalized recommendation [3, 4], etc.”

“At the same time, video recommender systems generally face a classical decision dilemma - the trade-off between exploration and exploitation, i.e., how to correctly trade off between recommending content with known user preferences and discovering new potential user interests [4].”

“…Lihong, et al. applied the LinUCB algorithm to personalized news recommendation, which is a contextual Bandit algorithm, constructing linear confidence to facilitate online learning of both user characteristics and article characteristics leading to a significant reduction in cumulative regret [4].”

“LinUCB improved news recommendation at Microsoft [2 ], [14] , while contextual Thompson policies boosted ad ranking at Facebook [15] .”

“Bandit and reinforcement learning algorithms are well-suited for such problems and have seen academic and commercial success [Li et al., 2010, Chapelle and Li, 2011, Bouneffouf et al., 2020, Yancey and Settles, 2020, O’Brien et al., 2022].”

“They have been widely deployed in practice for tasks such as online personalization and recommendation systems [12, 27].”

“MAB-based methodologies have been effectively utilized across multiple domains, such as node selection in edge computing [8, 9], web-based news recommendation systems [12], and online advertising [13].”

“We adopt an explore and exploit experimental set-up, in the paradigm of reinforcement learning [17] and multi-armed bandits [12, 13, 16].”

“…bandit (MAB) problems Lattimore and Szepesv´ari (2020) have emerged as a fundamental framework for online decision making under uncertainty, with broad applications ranging from adaptive drug discovery to recommendation systems Mad-hukar et al. (2017); Qin et al. (2014); Li et al. (2010, 2011).”

“With context, actions, and costs defined, the reinforcement learning loop is complete [15]–[17].”

“Contextual bandits have been used in RL for action selection [5] and hyperparameter tuning [8].”

“This model has wide-ranging applications including recommender systems that sequentially offer products recommendations to users to maximize long-term revenue [LCLS10], or clinical trials in which the goal is to find the best treatment for a sequence of patients [Tho33].”

“Although, under a more complex attack AT-DPT outperforms all other algorithms, and matches CRLinUCB in the clean case.”

“Second, we investigate the impact of learning the model through interactions within a contextual bandit framework [Li et al., 2010] rather than using a precollected dataset.”

“'s personalized news recommendation system, precisely situated on its front-page interface [6].”

“…problem as a contextual bandit problem where a learning algorithm sequentially selects articles to serve users based on contextual information about the users and articles and adapting its article-selection strategy based on the feedback to maximize total rewards (Li et al. 2010).”

“The cumulative regret Reg ( T ) of CoCoB consists of two terms, defined as: The first term is the regret analysis of item-bandit, which follows the typical √ T -style term seen in linear bandit regret analyses [12,26].”

“In standard online learning settings, e.g., basic multi-armed bandits [10] or linear bandits [11], the feedback is usually a scalar stochastic reward or loss, which is only related to the action (probe) taken by the algorithm.”

“Combining reinforcement learning (RL) with expressive deep neural networks has demonstrated great potential in a range of applications, from robotics [1] and strategy games [2] to recommendation systems [3].”

“The contextual bandit problem has attracted a great deal of attention because it is a faithful model of many real-world problems, such as person-alised advertising (Abe & Nakamura, 1999), personalised news recommendation (Li et al., 2010) and medical treatment (Durand et al., 2018).”

“Most common bandit algorithms for recommender systems[12], [9] consider each arm as the item to recommend.”

“Despite being a simple model, MAB has found wide-ranged applications including online recommendations (Li et al., 2010), wireless communications (Gai et al., 2010), resource allocation (Liu et al., 2021), clinical trails (Aziz et al., 2021), etc.”

“We run our bandit-based prompt-tuning method with UCB [9] and ϵ -greedy exploration strategies.”

“A key aspect here is the balance between exploitation (using existing knowledge) and exploration (gaining new experience), which can be mathematically expressed as a contextual multi-armed bandit problem [8].”

“This cost function is flexible and is customary in contextual optimization (Sadana et al., 2024) and contextual bandit (Li et al., 2010; Lattimore and Szepesvári, 2020).”

“…and visit count, specifically by choosing the action with the highest score according to the following Upper Confidence Bound (UCB) formula [13]: 𝑎 𝑘 = arg 𝑚𝑎𝑥 {𝑄(𝑠, 𝑎) + 𝑐𝑃(𝑠, 𝑎) √∑ 𝑏 𝑁(𝑠, 𝑏) 1 + 𝑁(𝑠, 𝑎) } (18) where 𝑁 represents the visit count, 𝑄 the average value,…”

“A linear link between the predicted reward and the situation was presupposed by LinUCB 13-14 .”

“The multi-armed bandit problem (MAB) is a classic framework in sequential decision making, capturing the exploration-exploitation trade off faced in many real-world domains, for example, recommendation systems [1, 2, 3], clinical trials [4, 5], online advertising [6], adaptive routing [7] and so on.”

“In [25–27], the authors propose contextual MAB models to learn the personalized news article recommendation, the relation between the context of tiles and the QoE, and the automatic optimization scheme of wireless base station parameters.”

“These algorithms have achieved success in various domains ranging from personalized online advertisement and recommender systems [22, 37, 3] to clinical trials [36, 2, 33] and adaptive routing in communication systems [24, 21].”

“• We implement a prototype of ASMF UZZ 1 , locate bug-triggering code segments through JavaParser [24] and PR, and implement scheduling policies through LinUCB [25] for replication and further optimization for future use.”

“…is widely used in domains requiring personalization, experimentation, or optimization under uncertainty, including recommender systems, healthcare, education, finance, and energy management (Li et al., 2010; Tewari & Murphy, 2017; Lan & Baraniuk, 2016; Soemers et al., 2018; Chen et al., 2020).”