Bayesian Active Learning for Classification and Preference Learning

“Perhaps the most ubiquitous is active learning for SVMs [Tong and Koller, 2001, Seung et al., 1992], where the volume of Version Space (VS) is used as a proxy for the posterior entropy.”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“We no longer need to build our entropy calculation around the type of posterior approximation (as in [MacKay, 1992, Krishnapuram et al., , Lawrence et al., 2003]) but are free to choose from many of the available algorithms.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“Information Theoretic approaches: Maximum Entropy Sampling (MES) [Sebastiani and Wynn, 2000] explicitly works in dataspace (Eqn.”

“…parameter space (exponentially hard with dimensionality), or sampling (from which it is notoriously hard to estimate entropies without introducing bias [Panzeri and Petersen, 2007]), these papers make Gaussian or low dimensional approximations and calculate the entropy of the approximate posterior.”

“For its part, BatchBALD [84] opts to expand BALD [67] to the batch query context; this approach no longer calculates the mutual information between a single sample and model parameters, but rather recalculates the mutual information between the batch samples and the model parameters to jointly score the batch of sampl…”

“Another measure (also known as (BALD, [27]) is the mutual information between data-points and weights.”

“Common selection criteria include entropy (Dagan & Engelson, 1995), uncertainty sampling (Lewis & Gale, 1994) and expected informativeness (Houlsby et al., 2011).”

“Notice that there are other methods that can measure the prediction uncertainty, such as the mutual information between labels and parameters [19], but we found that the prediction variance works better in our experiments.”

“Perhaps the most obvious related literature is that of Bayesian active learning (BAL) [37, 45, 56, 65, 77, 78, 91].”

“Houlsby et al. (2011) propose Bayesian Active Learning by Disagree-ment (BALD), a method that chooses data points that maximize the mutual information between predictions and model’s posterior probabilities.”

“We denote Monte Carlo Dropout Disagreement by DO-BALD and its Bayes-by-Backprop counterpart as BB-BALD.”

“For the MC Dropout and BNN SVI which incorporate stochastic sampling in testing, we set the 10 forward passes through the model and adopt the BALD [20] method to quantify the model uncertainty as suggested by [27].”

“The authors propose a simple idea to turn the original BALD algorithm of Houlsby et al. [2011] into a batch active learning algorithm via a softmax function over the current uncertainties”

“It is also used to guide methods that require explorations such as active learning [78], continual learning [134], Bayesian optimisation [70], and reinforcement learning [82].”

“Various metrics have been proposed to quantify these uncertainties in classification tasks [32, 35], including predictive entropy [36], variation ratio [37] and mutual information [36, 38].”

“One approach that is now commonly adopted [Houlsby et al., 2011, Gal, 2016, Depeweg et al., 2018, Smith and Gal, 2018, Mobiny et al., 2021] is based on information-theoretic quantities derived from Shannon entropy.”

“We briefly discuss connection of EDDI information reward with BALD (Houlsby et al., 2011) and.”

“Note that this has some interesting connections with uncertain sampling as presented in (Houlsby et al., 2011).”

“Hence, it is successfully integrated into active learning by [10, 16, 19, 28].”

“Houlsby et al. (2011) presented an approach based on Bayesian information theory, in which examples are selected in order to maximally reduce the entropy of the posterior distribution over classifier parameters.”

“We do this by combining the general acquisition strategy BALD (MacKay 1992; Houlsby et al. 2011) with our probabilistic model and Monte-Carlo Dropout (Gal 2016).”

“The main technical contribution of this paper consists of combining BALD and BDA for generating new labeled samples that are informative for the training process (see Fig.”

“Similarly, we obtain For this model, BALD [2, 4] can also be evaluated in closed form: We can make a direct comparison with BALD by treating the squared norm of a data point with itself as an acquisition function, α ACS ( x n ; D 0 ) = (cid:104)L n , L n (cid:105) ˆ π, F , yielding, Viewing α ACS…”

“…which aims to select the most beneficial samples from an unlabeled data pool based on criteria including uncertainty (Lewis and Gale, 1994; Houlsby et al., 2011; Gal et al., 2017), diversity (Huang et al., 2010; Sener and Savarese, 2018), and hybrid objectives (Du et al., 2017; Yang et…”

“Houlsby et al. (2011) and Daniel et al. (2015) proposed active GP regression methods for classification and reward learning, respectively.”

“They were the first to use MC dropout in computing uncertainty metrics like entropy, standard deviation, and Bayesian active learning disagreement (BALD) (Houlsby et al., 2011).”

“…classification (Rasmussen and Williams, 2005), representation learning (Lawrence, 2005), state space modelling (Wang et al., 2005), active learning (Houlsby et al., 2011), reinforcement learning (Deisenroth, 2010), black-box optimisation (Snoek et al., 2012), and numerical methods (Mahsereci and…”

“Bayesian Active Learning by Disagreements ( BALD ) [20, 26] chooses data points that are expected to maximize the information gained from the model parameters ω , i.e. the mutual information between predictions and model posterior: [29] maximizes the mean standard deviation of the predicted…”

“[71]), which does not require fictitious posterior updates:”

“Active learning has traditionally focused on pool-based methods, which choose points to label from a fixed reservoir (Lewis & Catlett, 1994; Settles & Craven, 2008; Settles, 2009; Houlsby et al., 2011).”

“…to label at each step out of all possible regions in the pool, (ii) H is an uncertainty sampling method that selects the regions with maximum cumulative pixel-wise Shannon entropy, (iii) B picks regions with maximum cumulative pixel-wise BALD (Houlsby et al., 2011b; Gal et al., 2017) metric.”

“We adopt the Bayesian Active Learning by Disagreement (BALD) acquisition function [19] to sample the most important inferences with the approximation from the Monte Carlo (MC) dropout samples as”

“BALD needs to aggregate”

“Perhaps the best-known instance of adaptive experimental design is active learning, wherein the designs to be chosen are the data points for which to acquire labels (Atlas et al., 1990; Settles, 2010; Houlsby et al., 2011; Sener & Savarese, 2018).”

“47 ), in their application to histopathological image classification.”

“QUERY STRATEGIES RANDOM DATA - BASED DATA UNCERTAINTY discriminative[34] REPRESENTATIVENESS CLUSTERING flat[101, 68] hierarchical[18,77] SET CONSTRUCTION core-set[84,78] MODEL - BASED MODEL UNCERTAINTY UNC-IE[87] EXPECTED PARAMETER CHANGE expected gradient length [86,104] expected weight change [96] ADVERSARIAL DFAL[22…”

“In [42], Houlsby et al. state that test points x ∗ that maximize MI are points on which the model is uncertain on average, as the model parameters W erroneously produce predictions with a high degree of confidence.”

“The variant Offline is our semantic map prediction model without fine-tuning with an active strategy, BALD adapts the BALD objective (Houlsby et al., 2011) for actively training our ensemble, and Entropy refers to our model fine-tuned with an entropy objective for active training (Shannon, 1948).”

“Using the insight that the BAS score formally expresses the mutual information between c and z% (for the given x%), Equation 6 can be rewritten in a different form that is computationally far more convenient, as it does not require expensive posterior updates for a continuum of imaginary data, z%, to compute IPðcjz%;…”

“Estimating model uncertainty through modelling the (approximate) posterior p ( θ | D ) in Bayesian model averaging is a popular technique (Houlsby et al., 2011; Gal et al., 2016; Smith & Gal, 2018; Ulmer et al., 2020), but comes with the disadvantage of additional computational overhead.”

“This is because the acquisition proposal distribution, a stochastic relaxation of the Bayesian Active Learning by Disagreement (BALD) objective (Houlsby et al., 2011), is effective (c.f. § 4).”

“…by Disagreement) builds upon Monte-Carlo Dropout by proposing a decision theoretic objective; it acquires examples that maximise the decrease in expected posterior entropy (Houlsby et al., 2011; Gal et al., 2017; Siddhant and Lipton, 2018) – capturing “disagreement” across different dropout masks.”

“Ideally, we would take an optimal experimental-design approach (21, 22) and estimate, for a given image, how much seeing the response would reduce our uncertainty about which model generated the data (assuming that one of the models underlies the observed human responses).”

“The aim is to select the next best view that maximizes the Information Gain [12, 15, 18] be-tween candidates views x ∈ D pool and D train , where D pool is the pool of candidate views: where H [ · ] is the entropy [14].”

“In the literature, the use of mutual information for uncertainty quantification can be seen in works by Houlsby et al. (2011); Gal, Islam, and Ghahramani (2017) Active Learning by Disagreement (BALD) and via knowledge uncertainty in work by Depeweg et al. (2017); Malinin, Mlodozeniec, and Gales…”

“For further details, we refer the reader to [23].”

“Bayesian perspective over the model parameters is also applicable (Houlsby et al., 2011).”

“When y and y ∗ are discrete we can write This can be estimated using samples, θ i ∼ p φ ( θ ) (Houlsby, 2014): Relatedly Houlsby et al (2011) identified a closed-form approximation of BALD for the particular case of using a probit likelihood function, a Gaussian-process prior and a Gaussian…”

“Common selection criteria include entropy [25], boosting the margin for classifiers [1, 12] and expected informativeness [20].”

“BayesianActiveLearningbyDisagreement (BALD) [20] is an acquisitionmethod that utilises the uncertainty estimates via mutual information between the model predictions and model parameters.”

“This criterion is equivalent both to minimising the expected information entropy of the posterior distribution and to Bayesian active learning by disagreement (BALD) [5] (see Methods).”

“We have analyzed two approaches to estimate uncertainty within classification: tractable view of the mutual information (MI) 30,32 and bias-corrected MI. 33 The MI between the prediction y and the posterior over the model parameters w captures a different measure of uncertainty.”

“ω0 if the label y = Y 0 were observed for a new data point x, given the training dataset D [24].”

“We did not compare to other ensemble-based approximations, such as mutual information (Houlsby et al., 2011).”

“We evaluate the epistemic uncertainty using Bayesian active learning by disagreement (BALD) [21] for the activity recognition task.”

“In Houlsby et al. (2011), Bayesian active learning by disagreement (BALD) was proposed to select samples that maximize the mutual information between model parameters and model predictions.”

“We model our uncertainty using an approximate posterior p(θ | D), given by training an ensemble or dropout network on our offline dataset D. Houlsby et al. (2011) show that the information gain of a potential query can be formulated as: I(θ;Y | D) = H(Y | D)− Eθ∼p(θ|D)[H(Y | θ,D)].”

“This selection is typically based on criteria such as uncertainty sampling which focuses on epistemic uncertainty (Mussmann & Liang, 2018; Houlsby et al., 2011; Kirsch et al., 2019; Nguyen et al., 2022).”

“…extensively explored strategies for sequential data acquisition in various domains, including concept learning [56, 3], active learning [16, 58, 36, 23, 55], Bayesian optimization [43, 44, 49, 5, 60, 35, 64, 24, 13, 21, 68, 22], reinforcement learning [39, 29, 61], and robot planning with…”

“Our MIbased diversity term resembles a common active learning criterion (Houlsby et al., 2011), but a key difference is that we directly optimize a set of models with respect to our criterion.”

“Other approaches use classifier confidence (Lewis and Gale, 1994), predicted expected error (Roy and McCallum, 2001), or model disagreement (Houlsby et al., 2011).”

“Other settings include: active learning (Sadigh et al., 2017; Houlsby et al., 2011; Eric et al., 2008), which is focused exclusively on learning an accurate model rather than maximizing utility of decision-making; learning with more structured preference feedback (Radlinski and Joachims, 2005; Shivaswamy and Joachims,…”

“Our method is more related to disagreement-based active learning (Houlsby et al., 2011; Siddhant and Lipton, 2018; Shen et al., 2017), which queries for instances where multiple models disagree the most, and is usually combined with model uncertainty measurements (Gal and Ghahramani, 2016).”

“…a GP with kernel k̃→scale(Xtrain) at x. Unlike in the classification setting, the noise distribution ε ∼ N (0, σ2) in the GP model is independent of x, which renders different acquisition functions like maximum entropy (Shannon, 1948; MacKay, 1992b) or BALD (Houlsby et al., 2011) equivalent to Eq.”

“Our work differs from ABCD in a few ways: we work with both linear and nonlinear SCMs by using state-of-the-art posterior models over DAGs (Lorch et al., 2021), we apply BO to select the value to intervene with, but we also prepare the batch using soft BALD (Kirsch et al., 2021) which is significantly faster than the g…”

“Moreover,to measure different sources of uncertainty, mutual information (MI; [75, 77]) between ensemble predictions and their parameters is often used in NLP models [78, 79, 80].”

“We consider three strategies for learning preferences over Y = ˜ f ( X ) : random search, BALD, and a novel AF called EUBO which is introduced in §4.1 and aims to find the best query over a known set of outcomes.”

“However, the Bayesian active learning by disagreement (BALD) proposed by Houlsby et al. (2011) leverage mutual information between outputs y i and model parameters θ trained on a dataset D : Let p cmi be a probability of a tag c for a token i that is predicted by a m -th stochastic pass of a model…”

“We evaluate the model uncertainty and predictive uncertainty us-ing Bayesian active learning by disagreement (BALD) (Equation 6) and predictive entropy (Equation 7), respectively.”

“In this work, we focus on the “Bayesian Active Learning by Disagreement (BALD)” (Houlsby et al., 2011) acquisition strategy because it demonstrated very good performance in comparison to other strategies in the experiments by Gal et al. (2017) as well as in our own preliminary experiments.”

“This measure corresponds to the mutual information between predictions and model parameters W [30].”

“Table 4 shows the performance of SGD, MC dropout, SGLD, and APD using the fcNN1 network (784- 100-10) with VR, entropy, and BALD.”

“For example, if the quantity of interest is g ( a ), then BALD seeks to find a ′ such that the information gain measured by the entropy decay is maximized.”

“In addition, human evaluation focuses on individual images and does not necessarily measure how well the generative model reflects the data distribution.”

“Our technique currently uses uncertainty sampling, but other techniques from psychophysics or Bayesian active learning may prove better suited for this application (King-Smith et al. 1994; Kontsevich & Tyler 1999; Roy & McCallum 2001; Settles 2009; Houlsby et al. 2011).”

“In (f), we can observe that the high BALD area has considerable overlaps with the correct predictions.”

“Epistemic uncertainty is simply the difference [27], he(x) := ht(x)−ha(x).”

“From our experiments, the Core Set [12] and Hierarchical [41] representativeness techniques as well as the Breaking Tie [21] and BALD [13] uncertainty heuristics stand out from other techniques.”

“Bridging this generalised perspective back to how aleatoric and epistemic uncertainty have been discussed in past work, we provide new insights on BALD, a popular information-theoretic objective for data acquisition (Gal et al, 2017; Houlsby et al, 2011; Lindley, 1956).”

“AL and example selection Active learning (AL) (Lewis and Catlett, 1994; Settles and Craven, 2008; Settles, 2009; Houlsby et al., 2011) is a well-studied field that investigates how machine learning algorithms might automatically select helpful additional data points to maximize their performance.”

“Among the multiple methods to estimate the uncertainty in DBAL, we use the Bayesian Active Learning by Disagreement (BALD) [15] as the estimation method – it showed the best generality in our experiments.”

“It can be instantiated through different acquisition functions, such as maximum entropy [28], BALD [15], and variation ratios [11].”

“Mutual information between the posterior over W and the prediction y∗ quantifies the uncertainty in the BNN’s output [26].”

“BALD [11] and BATCHBALD [19] do not perform well against the other strategies, and BATCHBALD does not perform as well as BALD in our setting.”

“Estimating uncertainty reliably and efficiently has remained an open problem with many important applications such as guiding exploration in Reinforcement Learning (Osband et al., 2016) or as a method for selecting data points for which to acquire labels in Active Learning (Houlsby et al., 2011).”

“Mutual information 53 .”