A contextual-bandit approach to personalized news article recommendation

“In contrast to theunguidedexploration strategy adopted by ǫgreedy, another class of algorithms generally known as upper confidence bound algorithms [4, 7, 17] use a smarter way to balanc e exploration and exploitation.”

“In contrast to the unguided exploration strategy adopted by ǫ - greedy , another class of algorithms generally known as upper con-fidence bound algorithms [4, 7, 17] use a smarter way to balance exploration and exploitation.”

“D EEP reinforcement learning has been successful in myriad kinds of tasks ranging from playing Atari games [1], to autonomous driving [2] [3], robotic manipulation [4] [5], recommendation system [6] [7], economics [8] [9] and even training large language models [10] [11] etc.”

“…across diverse domains related to sequential decision-making, including healthcare intervention (Tewari and Murphy, 2017; Rabbi et al., 2018), recommendation systems (Li et al., 2010; Kallus and Udell, 2020), and dynamic pricing (Kleinberg and Leighton, 2003; Wang et al., 2021), to name a few.”

“A more classical approach is the LinUCB algorithm [2], [3], [4], [5], which typically considers the objective function as a linear weighted function of context.”

“We adopt the hybrid linear model in LinUCB proposed by Li et al. [1] to learn from both user context and option context.”

“A 𝑘 -armed linear bandit uses a linear model for rewards: 𝑟 𝑡 = 𝑥 ⊤ 𝑡 𝑎 𝑡 + 𝜀 𝑡 , where 𝑥 𝑡 ∈ R is the feature vector, ( 𝜃 ) = 1 ∈ R are the arm reward coefficients, and 𝜀 represents subgaussian noise [7, 21].”

“For example, there are numerous large-scale datasets for benchmarking bandit algorithms for recommendation systems [51, 50, 74, 69, 26].”

“This raises two questions: (i) How do we define the notion that Z i is uniquely recoverable from the distribution of observable data p ( H T ) ?”

“More specifically, compositionality in our work is defined through: (1) structured units with instance-specific compositions; and (2) component interactions that are formalized through an interaction graph and the Markov assumption (Equation 1).”

“Multi-armed bandit (MAB), or its contextual variant, is a standard formulation for many sequential decision making problems in statistics, engineering, psychology, e-commerce and online advertising (Li et al., 2010; Sutton & Barto, 2018).”

“Compared with traditional bandit, LinUCB [18] introduces contextual features into bandit to achieve a better balance [19, 20].”

“It has been applied in various domains, including clinical trials, recommendation systems, and online advertising [6, 10, 11, 16, 20].”

“…with the highest upper confidence bound is chosen according to the equation: where 𝑥 𝑡 ( 𝑎 ) is the feature vector for action 𝑎 at time 𝑡 , 𝜃 𝑎 is the parameter vector for action 𝑎 which we want to estimate, and 𝛼 is the hyperparameter controlling the exploration/exploitation trade-off.”

“Lastly, all the experiments in Figure 3 use the same regularizer for both online and offline algorithms, and we still observe a gap in performance.”