A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“Since LinUCB [38] and KernalUCB [53] are outperformed by the above baselines, we will not include them for comparison.”

“…A plethora of methods has been developed and successfully employed to solve cold-start issues, including side-information enhanced [11], [42], [43], [44], [45], [46], exploration-exploitation trade-off based [47], [48], [49] and few-shot learning related [9], [13], [44], [50], [51], [52].”

“Traditionally, the study of such structured bandit problems has relied on knowledge of the problem structure like linear bandits [Li et al., 2010, Abbasi-Yadkori et al., 2011, Degenne et al., 2020], bilinear bandits [Jun et al., 2019], hierarchical bandits [Hong et al., 2022a,b], Lipschitz bandits…”

“The study of MAB has important applications in various fields, such as online advertising [1], recommendation systems [2], clinical trials [3] and resource allocation in network systems [4], where decisions must be made sequentially in the face of uncertainty.”

“Over the years, there have been attempts to combine the predictive prowess of machine learning models with the decision-making capabilities of bandits [6, 7, 8, 9].”

“Let the context vector be We adopt a LinUCB-style algorithm [8].”

“(cid:2) LinUCB [32]: which selects the CDT and resolution based on the minimum upper confidence bound of the utility based on the classic MAB theory.”

“In this section, we build the Contextual Multi-Armed Bandit learning-based Caching Update (CMAB-CU) algorithm utilizing the LinUCB scheme [39].”

“The eight routers are LinUCB and its shared-parameter variant (Li et al., 2010), linear Thompson sampling (Agrawal and Goyal, 2013), ensemble sampling (Lu and Van Roy, 2017), discounted Lin-UCB (Russac et al., 2019), CUSUM-and GLR-based change-detection LinUCB (Liu et al., 2018; Besson et al.,…”

“In Section 5 we provide an example instantiation of our proposed framework for an adaptive question answering system that uses a Contextual Multi-Armed Bandit (CMAB) for optimization [41].”

“CB algorithms find applications in various fields, including robotics (Srivastava et al., 2014), clinical trials (Aziz et al., 2021), communications (Anandkumar et al., 2011), and recommender systems (Li et al., 2010).”

“Personalized AI solutions can support users in many possible use cases, typically by learning user behavior through available data [40, 42, 46].”

“Our fully autonomous baseline is LinUCB [12], the state-of-the-art for acquiring unseen bite-sized food items like banana and apple slices [6, 7].”

“• LinUCB [Li et al. , 2010; Abbasi-Yadkori et al. , 2011]: The standard linear contextual bandit framework designed for infinite arm sets.”

“MABs have been successfully used for tasks such as news article recommendation (Li et al., 2010), data selection in neural machine translation (Kreutzer et al., 2021), model selection from a pool of multiple NLP systems (Haffari et al., 2017), and crowdsourced worker selection for annotation (Wang…”

“Tomkins et al. also use random effects in their Thompson-Sampling [Russo and Van Roy, 2014; Thompson, 1933] contextual bandit algorithm [Li et al. , 2010], IntelligentPooling.”

“If we restrict the setting to bandit feedback with fixed number of experts, it becomes a special case of contextual multi-armed bandits (Auer et al., 2002; Li et al., 2010).”

“T result of Theorem 8 is to be compared with the standard linear bandits, where LinUCB is the algorithm often used in practice (Li et al., 2010), achieving D √ T cumulative regret.”

“This formulation connects to the contextual bandit framework [20]: the signal vector S ( r ) serves as the context, model selection is the action, and response quality is the reward.”

“Bandit algorithms have been widely applied in domains such as online recommendation systems (Li et al., 2010), clinical trials (Villar et al., 2015), hyperparameter tuning (Li et al., 2018), and more recently applications with LLMs (Shi et al., 2024; Chen et al., 2024).”

“Bandit optimization is a very important framework for sequential decision making with numerous applications, including recommendation systems (Li et al. 2010), health-care (Tewari and Murphy 2017), AI for Social Good (AI4SG) (Mate et al. 2022), hyperparameter tuning in deep learning (Bergstra et…”

“(Non-stationary) contextual bandits In contextual bandit problems, the agent is presented with features x t +1 , and must choose an action (arm) that yields the highest expected reward (Li et al., 2010).”

“Li et al., 2010; Chu et al., 2011) and the Thompson sampling (TS; Thompson, 1933; Agrawal and Goyal, 2013) strategies.”

“We observe that we can regard the data valuation as a type of context associated with each arm, and thus the pricing problem can be formulated as a contextual bandit problem [40].”

“Inspired by LinUCB [14], we enhance the traditional MAB method by integrating contextual information, reinforcing its capabilities as the decision-making agent at each step.”

“Active exploration is common in RSs, e.g., using (contextual) ban-dit methods to present novel items to users to refine models of their preferences (Li et al. 2010).”

“Bandit algorithms effectively handle exploration-exploitation face-offs [8,19,21].”

“Our solution can be viewed as projecting the returns into the low-dimensional space represented by MALinZero using a contextual linear bandit problem formulation [12, 20].”

“The Algorithm Selector module in the AEE component implements a contextual multi-armed bandit approach using the LinUCB (Linear Upper Confidence Bound) algorithm [117] to dynamically select the most suitable encryption algorithm.”

“Third, MOMAB neglects the contextual information in real-world applications, such as user preferences and news features in news recommendation systems, which could be used to guide the decision-making process (Li et al. 2010).”

“This recognition has sparked a growing interest in studying adaptive learning algorithms in the setting of contextual bandits [24, 28, 4] and reinforcement learning (RL) [37].”

“…warmPref-PS algorithm, we consider the following baselines: (i) (vanilla) PS , a PS algorithm that does not use the offline dataset, (ii) LinTS from Li et al. (2010) and Agrawal & Goyal (2013), and (iii) naive-PS , which is essentially the warmPref-PS in Algorithm 2 but assumes β “ 0, i.e., models…”

“Linear Upper Confidence Bound Algorithm for Contex-tual Bandit Problem (LinUCB) comes as a more efficient solution for the exploitation-exploration paradigm [7], where uncertainty is introduced for each arm.”

“In the course to machine learning, AI alignment has shown up explicitly before in the field of Recsys and information retrieval where learning-to-rank from bandit feedback and user preference have been studied intensively [1, 61, 76, 85].”

“Front Page Today Module dataset, which contains over 33 million events, the authors in [173] conclude that contextual algorithms outperform non-contextual ones in scenarios where user and content dynamics are constantly changing, such as web services.”

“We follow the conventions of Sumers et al. (2022) and study a special case of linear contextual bandits (Li et al., 2010) in which each context is a subset of the action space that the agent is allowed to choose from in that context.”

“Online learning with bandit feedback is a powerful paradigm for modeling sequential decision-making cases (Robbins 1952), such as clinical trials (Villar, Bowden, and Wason 2015), news recommendation (Li et al. 2010), and website optimization (White 2012).”

“Unlike prior works [22], [23], [24], [25], ours requires neither explicit contexts as parameters of the reward functions nor a context-reward relationship known a priori.”

“Most of the existing contextual bandit algorithms assume a linear relationship between the contexts-control space and the associated reward [54]; or assume a certain structure in the reward function [55].”

“Since it inherently involves sequential actions and the exploration-exploitation trade-off, the MAB framework has been applied to online recommendations (Li et al., 2010; Zong et al., 2016; Wang et al., 2017) as well as a myriad other domains (Bouneffouf et al., 2020).”

“Contextual Bandits (naïve, LinUCB) [39]: simplified baselines ignoring temporal dependencies.”

“Beyond these applications, PORL is widely applied in areas such as autonomous driving Levinson et al. [2011], personalized content recommendations Li et al. [2010], medical diagnosis Hauskrecht and Fraser [2000], and games Brown and Sandholm [2019], where decision-making under partial observability…”

“Platforms such as Google News and Flipboard utilize natural language processing (NLP) and collaborative filtering to curate and rank articles based on relevance, thereby increasing reader engagement and retention (Li et al. , 2010).”

“While much of the literature assumes i.i.d. losses and arbitrarily chosen action sets (for which a well-known algorithm is LinUCB (Li et al., 2010)), we focus on the complementary regime: the action sets are drawn i.i.d. from a fixed distribution D , while the losses may be chosen adversarially.”

“More recently, Czapp et al. [17] design a LinUCB [14, 49] contextual multi-armed bandit algorithm to select the prompt with the highest predicted CTR from a pre-defined pool of prompts associated with the given context (defined by user, item, and location features), which is designed to balance the…”

“Given an arm-set X ⊂ B , the aim is to select arms X j ∈ X adaptively to minimise cumulative regret: A natural approach, given our previously constructed confidence sequences, is to employ the LinUCB algorithm (Abbasi-Yadkori et al., 2011).”

“Compared to the low-dimensional results in Figure 1(1), our methods ESTOR and STOR outperform LinUCB and LinTS, and ESTOR in particular performs better than DR Lasso Bandit.”

“To improve training efficiency, we propose a novel contextual bandit-based worst-case reward distribution estimator that learns to identify the worst-case distribution group g ∈ G for each state-action pair ( s, a ) by training a contextual bandit (CB) model (Li et al., 2010; Russo et al., 2018).”

“Specifically, NeuralUCB is a natural extension of Linear Upper Confidence Bound (LinUCB) (Li et al., 2010; Chu et al., 2011), which uses a DNN-based random feature mapping to approximate the underlying reward function.”

“Over the past few decades, bandit algorithms have gained widespread use in various real-world applications, such as recommender systems (Li et al., 2010), online advertising (Schwartz et al., 2017), and clinical trials (Woodroofe, 1979).”

“We employ LinUCB [Li et al., 2010], a contextual bandit algorithm for the arm or RM selection.”

“…context-action pairs, the mean reward of each action is assumed to be parameterized by an unknown function of action-context features, e.g., linear (Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal & Goyal, 2013b), GLM (Filippi et al., 2010; Li et al., 2017; Jun et al.,…”

“In contrast, contextual bandits (e.g., LinUCB [19]) improve the quality of the recommendations, by adjusting the recommendations based on a given context, which typically holds information for the users and items.”

“The Multi-armed Bandit (MAB) has proven to be a powerful framework to model various decision-making problems with great applications to medical trials [35], personalized recommendation [23], and hyperparameter learning [10, 21], etc.”

“A contextual bandit is a popular and practical framework for online learning under uncertainty (Li et al., 2010).”

“The significance of this problem lies in its wide applicability, ranging from recommendation systems (Li et al., 2010) to online advertising (Avadhanula et al., 2021) where the learner must balance between exploration (acquiring information about the model) and exploitation (pulling the arm…”

“Some works assume a structure in the reward function, e.g., a linear relationship between the contexts and the reward (Li et al. 2010; Abbasi-Yadkori et al. 2011; Abeille et al. 2017).”

“While other contextual multi-armed bandit algorithms can be used, in this work, we specifically evaluated the performance of two popular schemes, Linear Thompson Sampling (LinTS) [4] and Linear UCB (LinUCB) [16].”

“This line of work is popularly known as contextual bandits (Li et al., 2010; Chu et al., 2011; Agrawal and Goyal, 2013; Li et al., 2017).”

“At each memory decision point we apply the Upper Confidence Bound rule [1, 11], with the exploration bonus computed using the state-specific visit count: where c is the UCB exploration coefficient (Appendix C, 8).”

“The setting just introduced provides a general framework for sequential decision making under uncertainty (Langford & Zhang, 2008), capturing many problems of practical interest (Li et al., 2010).”

“5-0.5B [19]) occupy varying GPU memory, impacting KV cache capacity.”

“Early work applied multi-armed bandit algorithms to optimize ranking under click-through feedback (Li et al., 2010), later extending to combinatorial and semi-bandit settings that model item interactions (Chen et al., 2013).”

“Unlike standard contextual bandits [Langford and Zhang, 2007, Li et al., 2010, Chu et al., 2011], which commit to a single decision per round, CA-CMAB tracks both the best achieved performance per prompt and the cumulative cost of all attempts (Figure 2).”

“The Upper Confidence Bound (UCB) score [24] is computed for each candidate.”

“Specifically, standard algorithms for stochastic linear bandits without delay (e.g., Li et al. (2010); Abbasi-Yadkori et al. (2011)) all construct UCB/LCB for each action by constructing an ellipsoidal confidence set for θ .”

“Most of the literature on CLB focuses on the active learning setting for regret minimization [18], [19], [20].”

“Thelinear upper confidence bound (LinUCB) algorithm is a representative algorithm for solving the contextual MAB problem since it works well in various environments with low computational complexity [36], [37].”

“…but in many practical scenarios it is theoretically viable (Liu et al., 2022) and has lead to successful applications in complex domains like robotics (Zhu et al., 2017), poker (Brown and Sandholm, 2019), real-time strategy games (Vinyals et al., 2019) and recommendation systems (Li et al., 2010).”

“…of interventions is critical, which is the case for complex inventory management use cases [3, 13, 19–21 , 34 Contextless or linear bandits [4, 18] are a natural setup for adaptive experimentation, allowing for the introduction of statistical power constraints to guide the decision process…”

“[28] and Kuaishou [16]) intentionally insert small random traffic attempting to improve recommendation diversity or explore users’ interests [25, 40], which will cause all recommendation probabilities to be non-zero.”

“We adopted a standard process for producing unbiased offline, off-policy estimates of algorithm performance ([12]): using a genuine campaign in which discounts were randomly assigned across customers, we (a) re-sampled customers to create a dataset in which discounts were uniformly distributed (in…”

“In off-policy evaluation (OPE, Li et al. 2010), a logging policy π 0 interacts with the bandit for n rounds and collects a logged dataset D = { ( x i , a i , r i ) =1 .”

“Exploration strategies are also investigated in such a two-stage system, where LinUCB [28] is deployed individually [20].”

“Neural Bandits (Zhou et al., 2019; Zhang et al., 2021) integrate the adaptive exploration of ban-dit algorithms (Auer, 2003; Agrawal and Goyal, 2012; Li et al., 2010) with neural networks’ superior ability to predict rewards under uncertainty.”

“…Li et al., 2010a; Abbasi-Yadkori et al., 2011) have become a reference solution to the problems of online decision optimization in a wide variety of applications, including recommender systems (Li et al., 2010a), clinical trials (Durand et al., 2018), and display advertising (Li et al., 2010b).”

“In this section, we merged Lyapunov optimization with the LinUCB [21] algorithm.”

“Contextual bandits are a particularly lightweight choice for such control, since they can trade off exploration and exploitation while conditioning decisions on feature vectors [Li et al. , 2010].”

“We adopt the LinUCB strategy (Li et al., 2010), which augments the predicted reward with an exploration bonus that reflects the learner’s uncertainty about each action in the current context.”