Bayesian Active Learning for Classification and Preference Learning

“Perhaps the most ubiquitous is active learning for SVMs [Tong and Koller, 2001, Seung et al., 1992], where the volume of Version Space (VS) is used as a proxy for the posterior entropy.”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“We no longer need to build our entropy calculation around the type of posterior approximation (as in [MacKay, 1992, Krishnapuram et al., , Lawrence et al., 2003]) but are free to choose from many of the available algorithms.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“Information Theoretic approaches: Maximum Entropy Sampling (MES) [Sebastiani and Wynn, 2000] explicitly works in dataspace (Eqn.”

“For our study, we employ Entropy, EqualAL, and BALD to represent single-sample acquisition methods due to their direct applicability to segmentation tasks.”

“Prevalent approaches to uncertainty quantification in the literature (Houlsby et al., 2011; Gal, 2016; Depeweg et al., 2018; Smith and Gal, 2018; Mobiny et al., 2021) rely on information-theoretic measures derived from Shannon entropy (Shannon, 1948).”

“Our contributions can be summarized as follows: • We propose a preference-based MOO algorithm by combining BO and preference learning in which both the MOO objective f ( x ) and the DM preference (utility function) are modeled by a Bayesian manner.”

“Note that there are other possible acquisition functions, e.g. , the popular predictive variances and BALD (Houlsby et al., 2011).”

“The BALD acquisition function calculates the mutual information between the data samples and the model weights.”

“BALD aims to select instances with the highest marginal uncertainty about the class label but simultaneously high confidence for the individual settings of the model’s parameters.”

“Our problem can be generally viewed as learning preferences from human feedback [27, 28, 31].”

“Similar techniques are also used in (Houlsby et al. 2011; Gal, Islam, and Ghahramani 2017), where they select data with higher information gain for active learning.”

“To select informative pseudo-labeled samples, we adopt the Bayesian Active Learning by Disagreement (BALD) measure [14].”

“As a comparison, BALD”

“We maximize the expected information gained about the true reward r ∗ given the user’s answer w (Houlsby et al., 2011).”

“…is another uncertainty sampling approach that seeks the sample point producing the largest decrease in expected posterior entropy; this approach has been shown to yield the sample for which individual configurations of an underlying latent function most disagree about the outcome (Houlsby et al .”

“Classical approaches typically adopt a Bayesian framework (Lindley, 1956; Houlsby et al., 2011): they construct a surrogate model of the response function, update this model with new data, and select future experiments by optimizing an acquisition function such as expected improvement or…”

“Our formulation of data selection has connections to classic and deep active learning [Houlsby et al., 2011, Bickford Smith et al., 2023, Kirsch, 2023], which are deeply rooted in optimal Bayesian experimental design [Lindley, 1956, Rainforth et al., 2024], whose goal is to select a set of…”

“To further demonstrate the effectiveness of our sample identification strategy, we add more experiments on ImageNet-C to compare our CEMA with more strategies, namely SENTRY (Prabhu et al., 2021) and BALD (Houlsby et al., 2011).”

“Bayesian Active Learning by Disagreement (BALD) [65,66] suggests a sequential decision process to select the next data collection at a point where the information gain is maximized for model parameters in a classification task.”

“Then, the informativeness score is obtained by an acquisition function like entropy [37], or BALD [18] for classification tasks or total variance (TV) [46] for regression tasks.”

“Additionally, it has been shown that proper data selection can reduce the need to annotate every possible sample (MacKay, 1992; Holub et al., 2008; Houlsby et al., 2011).”

“Gal, Islam, and Ghahramani (2017) present the Monte Carlo dropout methods to combine Bayesian neural networks with BALD.”

“There are two main categories, namely, uncertainty-based methods, which select samples based on some pre-defined uncertainty metric [44, 45, 46, 47], and distribution-based methods, which query representative samples of sufficient diversity out of the unlabeled set [48, 49].”

“Thus, we design a Bayesian active learning approach for training our model, inspired by classic mutual-information-based approaches (Houlsby et al., 2011; Hern´andez-Lobato et al., 2014).”

“The logarithmic score is typically assumed implicitly in both the standard classification (Houlsby et al., 2011; Gal, 2016) and the NLG setting (Malinin and Gales, 2021; Kuhn et al., 2023) to derive uncertainty measures.”

“Houlsby et al. (2011) introduce Bayesian Active Learning by Disagreement (BALD), which queries points with high mutual information between the model parameters and the class label.”

“Our main objective is to identify the next-best-view that maximizes the Expected Information Gain [13, 6, 11] among potential viewpoints x ∈ D pool in comparison to D train , where D pool denotes the collection of available views: where H [ · ] is the entropy [12].”

“Data points offer a classifier different amounts of information gain when included in the training data [Lindley, 1956, Houlsby et al., 2011].”

“Bayesian Active Learning by Disagreement (BALD) (Houlsby et al., 2011; Kirsch et al., 2019) selects uncertain points that maximize the mutual information between model predictions and parameters, i.e., with the highest Expected Information Gain (EIG) about model parameters.”

“; Dagan and Engel-son, 1995) , Bayesian Active Learning by Disagree-ment (BALD; Houlsby et al., 2011) and Discrimi-native Active Learning (DAL; Gissin and Shalev-Shwartz, 2019) .”

“BALD (Houlsby et al.”

“To handle noise in the expert responses, we could model aleatoric uncertainty in addition to epistemic uncertainty, perhaps through the BALD objective (Houlsby et al., 2011; Noseworthy et al., 2021).”

“Using Bayesian methods, researchers have developed information based criterion for the problem of active learning using Gaussian Processes (GPs) on classification problems (Houlsby et al. 2011 tainty on image classification for neural networks.”

“The study in [14] uses predictive entropy with the Gaussian Process Classifier to yield a Bayesian active learning method called BALD.”

“…we obtain by labelling sample x with label y. Specifically, given the model parameters w and the current labelled dataset D, we can measure the mutual information between the posterior distribution p(w|D), and the model prediction p(y|x, w) (Houlsby et al., 2011): I[y, w|x, D] = H[p(y|x, D)]− E p(w”

“These heuristics decide based on different strategies such as entropy [48], maximizing the error reduction [43], or information theory [20, 25].”

“Where no labeled data set of valid and invalid sequences is available, we propose a novel approach to active learning for sequential tasks inspired by classic mutual-information-based approaches (Houlsby et al., 2011; Hern´andez-Lobato et al., 2014).”

“The improvement of performance of AL with BERT is however limited when compared to what it achieved for older NLP models [34,30,24] and even more so when compared to image classifiers [14,5,9].”

“The second term in Equation (9d) is intractable but can be approximated very well by replacing the binary entropy function by a squared exponential function as proposed in (21):”

“Houlsby et al. (2011) show that the information gain of a potential query can be formulated as: Intuitively, the information gain will be maximized when the first term is high, meaning that the overall model has high entropy, but the second term is low, meaning that each individual hypothesis θ from…”

“Furthermore, our approach is also aligned with Bayesian uncertainty quantification methods (Houlsby et al., 2011; Kandasamy et al., 2015; Kampffmeyer et al., 2016; Gal & Ghahramani, 2016; Alex Kendall & Cipolla, 2017; Gal et al., 2017; Kirsch et al., 2019; Mukhoti et al., 2021; Kirsch et al., 2021)…”

“The informative sample selection approach selects the most uncertain datum for the predictor (Lewis and Gale, 1994; Scheffer et al., 2001; Yang et al., 2015; Seung et al., 1992; Freund et al., 1992; Houlsby et al., 2011; Kirsch et al., 2019).”

“We employ BALD [21] with Monte-Carlo Dropout [17].”

“…to a wide variety of practical explorative-learning tasks, e.g., in active learning and Bayesian optimization to select informative data examples (Houlsby et al., 2011; Gal et al., 2017; Brochu et al., 2010; Fishel and Loeb, 2012), and in reinforcement learning to learn through interactions…”

“This is due to the fact that the UQ framework is dependent on the evaluation of a full ensemble of independent networks; while this is generally acceptable during training, since the networks can be evaluated in parallel, it can be problematic for inference when computational resources are limited…”

“From [22], the next best view objective is to maximize the reduction in entropy.”

“Most active learning acquisition functions belong to one of a few categories: uncertainty [35, 17, 14], margin [37, 2, 20], clustering [13, 30], and look-ahead [42, 9].”

“When using a Bayesian model with random variable ω for the weights, one can further split this term into H[Z | X] = I[Z;ω | X] + H[Z | X, ω], so uncertainty stemming from weight uncertainty and independent noise (Houlsby et al., 2011; Kirsch et al., 2019).”

“Another well-known acquisition function is BALD (Houlsby et al., 2011) which is based on mutual information.”

“– BALD: (Bayesian Active Learning by Disagreement [11]) it measures the mutual information between data and weights, and it can be interpreted as seeking data points for which the parameters (weights) under the posterior disagrees the most.”

“217 Despite its lack of theoretical guarantees, information gain is a commonly used effective approach 218 in adaptive active learning [29, 57, 58].”

“Uncertainty is a fundamental concept in machine learning, reflecting the fact that we usually lack complete information about the model’s predictions or parameters (Gal & Ghahra-mani, 2016; Houlsby et al., 2011; H ¨ ullermeier & Waegeman, 2021; Tonolini et al., 2024).”

“BALD We compute the BALD score [23] for each data point in X train and select the m data points with the highest score.”

“In the context of assisting humans, information-theoretic methods have primarily been used to reason about the human’s goals or rewards [32–34].”

“(4)) introduced by Houlsby et al. [2011] and popularized, for instance, by Gal [2016], Depeweg et al. [2018], Smith and Gal [2018] is the measure (C2) .”

“BNNs allow the application of several different uncertainty-based acquisition functions, such as BALD [22], Max Entropy [19, 23], and Mean Std [24].”

“We compare our method against DMFAL (Li et al., 2020a), BMFAL-Random (Li et al., 2022a), BMFAL (Li et al., 2022a) and MF-BALD (Gal et al., 2017) as baselines, using the same hyperparameter settings as in the literature.”

“Instead, drawing analogies from Bayesian Active Learning [42, 52], we introduce a new information theoretic objec-tive that maximizes the information gain from an entire dataset collected by a population of exploration agents (see Figure 2).”

“MCD [7] proposes a solution to fair training using i.i.d sampling over the existing BALD [19] based sampling.”

“In this paper, we focus on active learning for binary classification and, consider the Bayesian Active Learning by Disagreement (BALD) information criterion introduced in Houlsby et al. (2011):18”

“While active learning has been explored for learning reward/preference functions from limited interaction in the RL context [19, 9], our primary contribution is the application of this principle to a direct online RLHF setting.”

“While historically confidence estimation has mostly been applied to classification (Houlsby et al., 2011), multiple-choice (Jiang et al., 2021), and short-form QA (Xiong et al., 2024), with the advent of LLMs it has increasingly been considered for long-form generation.”

“By contrast, our objective is to actively decide what additional data to gather, akin to active data allocation and learning methods including Bayesian experimental de-sign [18, 19], information gain maximization [20, 21], and active learning [22].”

“Thisdynamicapproachtovalidationwasfound toincreasethetrainingefficiencyandmodel’sperformance,as demonstratedinourablationstudies.”

“…processes (GPs) and Bayesian neural nets (BNN) [] have enabled principled ways of predicting epistemic uncertainty, which led to better information-based decision making strategies [Houlsby et al., 2011, Hennig and Schuler, 2012, Wang and Jegelka, 2017, Gal et al., 2017, Wang et al., 2023].”

“• BALD [12, 17]: BALD is a representative bayesian active learning method that has been used as a strong active learning baseline in recent literature.”

“Instances leading to a maximal score relate to points on which the algorithm is uncertain on average and for with the model parameters are referenced with a high disagreeing prediction [61,62].”

“Bayesian active learning by disagreement (BALD) [15] is based on a probabilistic Gaussian process classifier.”

“Examples of utility functions include the Information Gain [1], Upper Confidence Bounds [2], or the Expected Improvement [3].”

“…, D ]) as the mutual information which estimates the expected reduction in uncertainty about the prediction Y that would be obtained if we knew the model parameters θ : It has been shown to be an effective measure of epistemic uncertainty (Houlsby et al., 2011; Gal et al., 2017; Smith & Gal, 2018).”

“1), besides the entropy method, we also use other methods such as random, Bayesian Active Learning by Disagreement (BALD) [79], to verify the validity of our class balanced strategy.”

“BALD (Houlsby et al., 2011) estimates the mutual information (MI) between the datapoints and the model weights, the idea being that points with large MI between the predicted label and weights have a larger impact on the trained model’s performance.”

“To measure information, we use the Bayesian active learning by disagreement acquisition function [26], where images from an unlabelled pool are chosen to maximise the mutual information I(y,W | z,Z,Y ) ∈ [0, 1] between model predictions and the posterior.”

“AL has been widely studied in image recognition [6, 23, 46, 58], information extraction [9, 22, 45], and text categorization [18, 35], using uncertainty-based sampling [8, 23], information gain [19], or theoretical dropout-based frameworks [13, 24, 38].”

“In addition, Bayesian Active Learning by Disagreement (BALD) [6] measures the uncertainty according to the probabilistic distribution of the model output via Bayesian inference, where an approximation by dropout is usually incorporated [5].”

“Beside meta-learning, another approach to reduce the number of required training data points is active learning [33], [34], [35],”

“BALD/Expected Information Gain (EIG).”

“● Bayesian active learning by disagreement (BALD) [13].”

“The method in [12] applies predictive entropy to the Gaussian Process Classifier to develop a Bayesian approach for active learning called BALD.”

“BALD calculates the mutual information between the predictions and model posterior (Houlsby et al., 2011; Gal et al., 2017): BALD[x] =Entropy[x]+ T∑ t=1 ∑ j Pt(y = j|x,wt) logPt(y = j|x,wt) (4) Random selection (Rand) is performed the baseline comparison.”

“We evaluate the model uncertainty using Bayesian active learning by disagreement (BALD) [13], which quantifies mutual information between parameter posterior distribution and predictive distribution.”

“In practice, the total predictive uncertainty of a model can be viewed as the combination of aleatoric and epistemic components.”

“Taking an information-theoretic perspective [21], we would then aim to choose actions such that uncertainty about the latent function is minimized .”

“We evaluated a number of acquisition methods: random, BALD (Houlsby et al, 2011), EPIG (Bickford Smith et al, 2023), maximum entropy (Settles & Craven, 2008), BADGE (Ash et al, 2020), greedy k centres (Sener & Savarese, 2018), k means (Pourahmadi et al, 2021), ProbCover (Yehuda et al, 2022) and…”

“5] × [15 , 60] ⊂ R 2 .”

“Uncertainty-based methods [6, 7] are the most popular because of their simplicity [8].”

“Specifically, we compare to the following popular deep active learning methods: BALD (Houlsby et al., 2011), BatchBALD (Kirsch et al.”

“Finally, Batchbald is a recent Bayesian approach that extends the popular BALD [28] method.”

“ours model kinematic uncertainty and take actions to reduce that uncertainty [26], [16], [27].”

“Here, we compute the mutual information between predictions and model posterior for uncertainty measurement [8]: where H is the entropy of given instance point and E denotes the expectation of the entropy of the model prediction over the posterior of the model parameter.”

“This in turn advises the user how the prediction should be used in downstream tasks [Houlsby et al., 2011, Kirsch et al., 2019].”

“…select data samples that the current model is most uncertain of, where uncertainty can be measured by the posterior probability of a binary classifier [8], the entropy of class posterior probabilities [9], best-versus-second best margin [9], and the disagreement among several models [10].”