Bayesian Active Learning for Classification and Preference Learning

“Perhaps the most ubiquitous is active learning for SVMs [Tong and Koller, 2001, Seung et al., 1992], where the volume of Version Space (VS) is used as a proxy for the posterior entropy.”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“We no longer need to build our entropy calculation around the type of posterior approximation (as in [MacKay, 1992, Krishnapuram et al., , Lawrence et al., 2003]) but are free to choose from many of the available algorithms.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“Information Theoretic approaches: Maximum Entropy Sampling (MES) [Sebastiani and Wynn, 2000] explicitly works in dataspace (Eqn.”

“Thus, more advanced criteria such as Bayesian active learning by disagreement (BALD) are proposed [33].”

“Others have proposed collecting as much information as possible by prioritizing inspection of uncertain samples [22], [23], [24].”

“– BALD: (Bayesian Active Learning by Disagreement [14]) measures the mutual information between data and weights, and it can be interpreted as seeking data points for which the parameters (weights) under the posterior disagrees the most.”

“Causal Bayesian Experimental Design (CBED) uses Bayesian Optimal Experimental Design to select interventions that maximize mutual information (MI) between new data and existing graph beliefs, with MI estimated via a BALD-like method (Tigas et al., 2022; Houlsby et al., 2011).”

“…divergence-based algorithms (Ducoffe and Precioso, 2018; Margatina et al., 2021; Zhang et al., 2022b), disagreement-based (Seung et al., 1992; Houlsby et al., 2011; Gal et al., 2017; Siddhant and Lipton, 2018; Kirsch et al., 2019; Zeng and Zubiaga, 2023), gradient-based (Settles et al.,…”

“We study the presence of active learning bias in a class of active learning methods called information theoretic active learning, where the modeler must pick an objective function capturing the informativeness of observations (Houlsby et al., 2011).”

“S3 shows empirically that this is the case, by evaluating BALD and BALV on the same benchmark problems used in the main text.”

“…by Houlsby et al. (2011)), which allows us to define it in a non-parametric setting (i.e., when y = f ( x ) + ϵ ): see details on the equivalence in Appendix B. Nonetheless, the EIG remains a challenging criterion to compute, as it involves nested Monte Carlo estimation (Rainforth et al., 2018),…”

“Active learning strategies have included uncertainty-based sampling [29, 1, 9] and information gain [25].”

“The information gain for a pairwise label, ya,b, is the reduction in entropy of the distribution over the utilities, f , given ya,b. Houlsby et al. (2011) note that this can be more easily computed if it is reversed using a method known as Bayesian active learning by disagreement (BALD), which…”

“…functions are given below: BALD Acquisition Function: One popular acquisition function is Bayesian Active Learning by Disagreement (BALD) [14], which is the expected information gain, measured by the reduction in Shannon entropy of the model parameter φ from labeling x across all…”

“In this paper, we select the Bayesian Active Learning by Disagreement (BALD) as our active-learning strategy (Houlsby et al. 2011, 2012), which determines the queried point, 𝐬 𝐐 , by maximizing the objective containing two entropy quantities: where H [ y s , D ] is called Shannon entropy and…”

“Subsequently, online and sequential settings were considered, motivated by applications in which the feedback is provided in an online manner, e.g., by a human [Yue et al., 2012, Yue and Joachims, 2009, Houlsby et al., 2011].”

“The Shannon entropy H ( · ) of the predictive distribution is a common measure of total predictive uncertainty [14].”

“…overcome this challenge, we develop two different approaches for infomax active learning in discrete latent variable models: one based on sampling (Houlsby et al., 2011; Bak & Pillow, 2018) and another based on variational inference (VI) (Blei et al., 2017), which we describe in the next two…”

“[25] defines the aleatoric uncertainty within the BNN as mean entropy or the Bayesian Active Learning by Disagreement Objective (BALD) [29] of model prediction: where H ( .”

“The first body of work proposes approaches for exploring new cases, from the simple but intuitive random examination [7] to more advanced active learning solutions using uncertainty [8] and diversity [9], [10].”

“MC Dropout relies on maximum entropy, while DBAL maximises information with BALD [21].”

“Houlsby et al. (2011) sample many networks with Monte Carlo simulation and propose an objective function that takes into account the disagreement and confidence of the predictions coming from these networks.”

“…model estimates performance distributions , we define the EIG of a policy-task pair using a Bayesian Active Learning by Dis-agreement (BALD) [20] formulation for probabilistic models The first term represents the marginal entropy over Q ij , which quantifies the total uncertainty in Q ij .”

“This is closely related to the disagreement term in Bayesian active learning [27].”

“Specifically, we follow (Houlsby et al., 2011; Wang et al., 2023) to leverage information gain of the model parameters to estimate how certain the estimator model is to the triple with respect to the true preference.”

“(2) We can interpret the agent’s behavior from Equation 2 as trying to maximize marginal uncertainty about y t , while minimizing model uncertainty about what value y t should take on given a particular x (Houlsby et al., 2011).”

“…hierachical LegalBERTs with the tex-tified allegation information on the original dataset from (Santosh et al., 2022) Aleatoric uncertainty, on the other hand, stems from inherent ambiguity and can be considered as the variability in experiment outcomes (Houlsby et al., 2011; Gal et al., 2016).”

“For each DAL cycle, they query the most informative training images according to the uncertainty-based query strategy Bayesian active learning by disagreement (BALD) [41] and then manually label them.”

“We use BALD (Houlsby et al., 2011) as the acquisition function for the deep ensemble and self-supervised BNN, because they provide epistemic uncertainty estimates, while we use predictive entropy for SimCLR.”

“Houlsby et al. (2011) also presented a formula for calculating mutual information without the costly computation of the expected posterior entropy.”

“…active learning methods usually focus on designing various sample selection strategies that can broadly be grouped into uncertainty-based [6], [11], [21], [22], [23], [24], [25], [26], [27], [28], [29] and diversity-based meth-ods [7], [8], [10], [12], [14], [18], [30], [31], [32], [33], [34].”

“Such a model should be confident in its predictions when being accurate, and be uncertain about inaccurate ones (Houlsby et al. 2011).”

“In many situations [12, 22, 33], it will be useful to distinguish the reducible uncertainty caused by the model’s lack of in-domain knowledge (called epistemic uncertainty or model uncertainty ) from the irreducible uncertainty caused by the inherent stochasticity or noise in the input data (called…”

“The most common are (i) Max Entropy, which measures the amount of information that can be extracted from a process, (ii) BALD (Houlsby et al. 2011), which chooses items that maximize the mutual information of prediction and model posterior, or those producing disagreeing predictions with high…”

“501 Our Uncertainty-aware selection algorithm with 502 the BALD measure further reduces the annotation 503 complexity by around 37% (compared with Ran- 504 dom Mixing).”

“Thus, we introduce an approximate strategy for the meta-agent: proposing environments that provide the most information gain over its belief distribution, as commonly done in active learning [32, 33] and robot active exploration [34].”

“• BALD [40] : The active selection strategy is to maximize the expected reduction of parameter entropy after retraining the model on the new training set.”

“Our implementation differs from Houlsby et al. (2011) in one aspect.”

“At iteration t + 1 , given the current observed data D t , us-ing the mutual information (Houlsby et al. 2011; Gal, Islam, and Ghahramani 2017) as a way to represent uncertainty, we propose the acquisition function as, (1) where y denotes the predicted function value corresponding to the data point…”

“Acquisition functions are typically categorized into uncertainty-based (36; 29; 37; 38) and distribution-based (39; 40; 41; 42) approaches, with some relying on external modules such as auxiliary predictors (38; 43; 44).”

“Following Houlsby et al. [2011] we can utilize conditional differential entropy to establish the following decomposition with Then a measure of aleatoric uncertainty can be defined as Intuitively, by fixing a first-order distribution, epistemic uncertainty is removed, allowing aleatoric uncertainty…”

“While active learning focuses on observations and feature acquisition on variables, our approach leverages a proper modeling of uncertainty at both training and prediction time to improve reliability, by focusing on both observations and variables with different modalities.”

“…reformulation in terms of the output space, rather than the complex parameter space and is known as Bayesian Active Learning by Disagreement (BALD) [193], denoted as Active Learning through uncertainty quantification has been been beneficial for medical [194]–[201], multi-view [202], remote…”

“In the context of minimizing epistemic uncertainty about an unknown target, several methods utilize the most informative selection policy using mutual information for active learning setting (Gal et al., 2017; Houlsby et al., 2011; Nguyen et al., 2021; Hübotter et al., 2024; Kirsch, 2024).”

“Concretely, for the model uncertainty, inspired by the Bayesian Active Learning by Disagreement [40], we find that the information gain is a good measure for the model uncertainty.”

“Until now, various uncertainty measures, as well as corresponding active learning algorithms, have been proposed (Nguyen et al., 2022; Houlsby et al., 2011; Jung et al., 2023); see Appendix A for a more comprehensive overview of the related work.”

“[ 39] integrates both generalization difficulty and data representation by extending [40] to diverse batch acquisitions.”

“Probabilistic labeling and uncertainty estimates are further crucial elements of any robust and ef fi cient AI-based phase space exploration approach 25,26 .”

“To address this issue, Bayesian Active Learning by Disagreement (BALD) queries the most uncertain example across the ensemble models but confident in the single model (Houlsby et al., 2011).”

“This heuristic outperformed popular baselines, such as BALD (Houlsby et al., 2011) and clustering-based approach.”

“B. Selecting samples with highest mutual information between their predicted labels and the weights, BALD [39,7], which was initially applied in [7] with T forward passes in MC-dropout.”

“While a flawless moderator is generally assumed in the review of interactive machine learning approaches like active learning (Burkhardt et al., 2018; Gal et al., 2017; Houlsby et al., 2011), the assumption does not have to apply to all real scenarios (Sheng et al.”

“A prominent approach, Bayesian Active Learning with Disagreement (BALD), even selects informative points that are also predicted to be less noisy (Houlsby et al., 2011; Gal et al., 2017).”

“Over the past few years, the rapid development of active learning has resulted in various sampling strategies, such as uncertainty sampling [18], query-by-committee [9] and information gain [11].”

“We start with the BALD ob-jective, which is the mutual information between the model’s parameters and the target label of a given data point.”

“In this study, we evaluate the model uncertainty using Bayesian active learning by disagreement (BALD) [17], which quantifies mutual information between parameter posterior distribution and predictive distribution.”

“Houlsby et al. (2011) showed that this objective function has a closed-form expression, where the first term can be approximated by some popular Gaussian approximation methods, such as Expectation Propagation (see “Appendix A”) or a Laplace approximation, and the second term can be approximated by…”

“…= H(Efα(p)[P (y|p)])− Efα(p)[H(P (y|p))] = − ∑ j αj α0 ( log αj α0 − ψ(αj + 1) + ψ(α0 + 1) ) This metric is useful when measuring uncertainty for out-of-distribution or adversarial examples, and a variation of it was used in the context of active learning (Houlsby et al., 2011).”

“Part of this chapter is documented in a technical report [Houlsby et al., 2011].”

“Wecomputeclasscentroids f μ c g c 2 C and shared across classes covariance matrix Σ .”

“Active learning can be used to either learn the agent’s utility model or to successively present them with better recommendations (Houlsby et al. 2011; Dragone, Teso, and Passerini 2018).”

“This selection is typically based on criteria such as uncertainty sampling which focuses on epistemic uncertainty (Mussmann and Liang, 2018; Houlsby et al., 2011; Kirsch et al., 2019; Nguyen et al., 2022b).”

“Ranking, comparison or preference feedback is considered in dueling bandit models (Yue et al., 2012; Dud´ık et al., 2015; Wu & Liu, 2016; Yan et al., 2022; Bengs et al., 2021; Sz¨or´enyi et al., 2015; Agnihotri et al., 2023, 2019; Hu et al., 2022) but it is akin to active learning from preferences…”

“BALD (Houlsby et al., 2011) is a priori the strongest baseline since it is a contextual active learning algorithm, unlike the others.”

“These scores are determined through acquisition functions such as entropy [45] or BALD [16].”

“[19] introduced this as Bayesian active learning by disagreement (Bald).”

“Bayesian Active Learning by Disagreement (BALD) [43] and its batch version (i.e., BatchBALD) [44] selected samples that maximize the reduction in expected entropy [45] by computing the mutual information between model predictions and model parameters.”

“Active learning AL algorithms can be classified into heuristic-based methods such as uncertainty sampling (Settles & Craven, 2008; Houlsby et al., 2011) and diversity sampling (Brinker, 2003; Joshi et al., 2009); and data-driven methods which learn the acquisition function from the data (Bachman et…”

“We will stick to the natural way of doing active-learning via querying the point with the highest information gain between the observation and all uncertain variables in the model [see MacKay (1992); Houlsby et al. (2011)] to investigate which impact our prior has on the active-learning performance.”

“First, it selects the next salient TS-A for profiling at the beginning of each year, where the salient TS-A is selected with a Bayesian active learning strategy (Houlsby et al. 2011).”

“In other words, it captures how strongly the model parameters and the model predictions for a given data point are coupled [24].”

“- BALD Select the instance with Bayesian Active Learning by Disagreement (BALD) (Houlsby et al. 2011): a(x) = H[y|x]− Ep(ω)[H[y|x, ω]] with ω denoting a model in the ensemble”

“4,26,29,30,35 We can write it as the entropy of the average softmax vector minus the average of the entropy of the individual softmax vectors:”

“Common AL strategies are either based on model uncertainty (Houlsby et al., 2011; Yuan et al., 2020), or promoting the diversity in sample distribution (Bodó et al., 2011; Sener and Savarese, 2018).”

“In GP classification and preference learning, I ( c, f | x , D ) can be approximated and maximized at low cost (Houlsby et al., 2011).”

“After the model had seen roughly 10% of the data, we collected examples that were: (i) selected by the AL strategy (BALD) as examples on which the model has low confidence, and (ii) grouped by the clustering procedure in the same cluster based on model aware Siamese similarity scores.”

“• BALD (Gal et al., 2017; Houlsby et al., 2011): Selects the node which has the the largest mutual information value between predictions and model posterior.”

“We can classify most methods into a few categories: uncertainty (Settles, 2012; Houlsby et al., 2011; Gal et al., 2017), margin (Tong and Koller, 2001; Balcan et al., 2006; Jiang and Gupta, 2019), clustering (Dasgupta and Hsu, 2008; Maggioni and Murphy, 2019), and look-ahead (Zhu et al., 2003b; Cai…”

“Most relevant to this paper are works on uncertainty sampling [13,12] and Bayesian active learning [11].”

“The first term is the entropy of the expected response, and the second term is the expected conditional entropy of the response given the belief about the model parameters (for more details, see Houlsby et al., 2011).”

“In order to work on lower dimensional output space, equation 4 is rearranged [9].”

“(iii) Analytical computation : GPs allow for the analytical computation of ∆ t ( h ) and I t ( h ) via the BALD approximation (Houlsby et al., 2011), bypassing the need for computationally expensive ensemble retraining.”

“MI is computed by measuring the difference between the uncertainty of the average prediction (i.e., the PE) and the average uncertainty of the individual model predictions [18].”

“This paper used Bayesian Active Learning by Disagreement (BALD) [25] to conduct active federated learning.”

“For example, representative sampling methods require pair-wise comparison throughout the dataset and sequential greedy acquisition [17].”

“…including mixed-effects modeling [Foster et al., 2019], model selection [Cavagnaro et al., 2010], hy-* Work done while at Aalto University. perparameter selection [Houlsby et al., 2011], Bayesian optimization [Hernández-Lobato et al., 2014], and graph discovery [Branchini et al., 2023].”

“This has the potential to be problematic for a wide variety of possible applications involving sequential decision-making, such as Bayesian experimental design [7, 45], active learning [2, 27, 50], Bayesian optimization [19, 23], and contextual bandit problems [35, 49].”

“Indeed, a GP prior on the latent utility u(x) ∼ GP(0,k(x,x′)) induces a GP prior on q([xi,x j]) = u(xi)− u(x j) by linearity [Houlsby et al., 2011]: q([xi,x j]) ∼ GP(0,kp([xi,x j], [xi,xj]), where”

“• To maintain the same measure of information content learned from both benign and adversarial training instances, we formulate an information gain (mutual infor-1 also known as Mutual Information (Houlsby et al., 2011; Gal et al., 2017) mation) objective.”