Bayesian Active Learning for Classification and Preference Learning

“Perhaps the most ubiquitous is active learning for SVMs [Tong and Koller, 2001, Seung et al., 1992], where the volume of Version Space (VS) is used as a proxy for the posterior entropy.”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“We no longer need to build our entropy calculation around the type of posterior approximation (as in [MacKay, 1992, Krishnapuram et al., , Lawrence et al., 2003]) but are free to choose from many of the available algorithms.”

“Although the latter approach was proposed several decades ago [Lindley, 1956, Bernardo, 1979], it is not always straightforward to apply the criteria to complicated models such as nonparametric processes with infinite parameter spaces.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“[Guestrin et al., 2005, Krause et al., 2006] also use mutual information.”

“Our active learning framework for GPC can be extended to the important problem of preference learning [F¨urnkranz and H¨ullermeier, 2003, Chu and Ghahramani, 2005].”

“Information Theoretic approaches: Maximum Entropy Sampling (MES) [Sebastiani and Wynn, 2000] explicitly works in dataspace (Eqn.”

“B) (Houlsby et al., 2011).”

“Finally, Houlsby et al. (2011) has proposed a information-theoretic policy Bayesian active learning by disagreement (BALD), which measures the mutual information between data points and model parameters and is adopted in the deep uncertainty literature (Gal et al., 2017; Kirsch et al., 2019;…”

“MI is approximated in different ways: AIT [Scherrer et al., 2021] uses F-score inspired metric to implicitly approximate MI; CBED [Tigas et al., 2022] incorporates BALD-like estimator [Houlsby et al., 2011]; ABCD [Agrawal et al., 2019] uses estimator based on weighted importance sampling.”

“MCP, Core-set, BALD, and (occasionally) DFAL are the best performing metrics and outperform random sampling.”

“[23] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.”

“Most active learning acquisition functions for statistical belong to one of a few categories: uncertainty [42], [21], [14], margin [47], [2], [22], clustering [13], [28], and look-ahead [56], [5].”

“The characteristics of the others are not applicable due to the use of dropouts (BALD, DropOut-Entropy) and model independence (MCP, DFAL, EGL, Core-set).”

“We note that, the conditional entropy term in (6) is distinct from the standard aleatoric uncertainty in conventional learning in which the model parameter φ τ , and not the hyperparameter θ , is assumed to be unknown [9].”

“Using decision and information theory are two general methods for active learning [9].”

“It has the be er results than the variation ratios, predictive entropy, and Bayesian Active Learning by Disagreement (BALD) methods [10, 12, 19], which select samples based on uncertainty information.”

“Formally, at each iteration t , we identify the maximally informative preference pair through information-theoretic criteria proposed by Houlsby et al. [47] I r ; w ˇˇ y , y 2 , D t = H (cid:2) r ˇˇ y , y 2 , D t (cid”

“…inference over rewards, (ii) tracking sequential posterior contraction to demonstrate epistemic uncertainty reduction, in the spirit of Bayesian active learning (Houlsby et al., 2011; Gal et al., 2017), and (iii) verifying inferred objectives through policy-level utility in RLHF fine-tuning.”

“The hybrid works [42], [43], [44], [45], [46], [47], [48] combine both paradigms such as by measuring uncertainty as to the gradient magnitude [44] at the final layer of neural networks and selecting gradients that span a diverse set of directions.”

“In conventional NLP, model uncertainty is often used for tasks like active learning or confidence calibration (Kendall and Gal; Gal and Ghahramani, 2016; Houlsby et al., 2011; Guo et al., 2017).”

“The existing AL methods mainly evaluate the informativeness of samples either from uncertainty [22], [23], [24] or from diversity [25], [26], [27], [28].”

“…to obtain this information elsewhere include: • comparing the candidate pool directly to existing training set distributions [42]; • extracting local output gradients with respect to the input variables from the trained NN [43]; • extracting uncertainty estimates from the trained NN [44, 45].”

“Formally, when the expectation is taken with respect to the prior predictive distribution of the data, this expected KL divergence coincides with the mutual information between the parameters and the observations [33, 38, 39].”

“This attribute has made it widely applicable in fields like active learning (Houlsby et al., 2011; Gal et al., 2017), uncertainty estimation (Gal & Ghahramani, 2016; Kendall & Gal, 2017; Lakshminarayanan et al., 2017; Russell & Reale, 2021), image reconstruction (Dorta et al., 2018), human pose…”

“One popular approach is uncertainty sampling, which prioritizes data points where the model exhibits the highest uncertainty [9, 14, 18, 37].”

“Lastly, Random refers to a random sampling process designed to mimic real-world randomness. proof-of-concept RL agent that, while performing below radiologists’ levels, demonstrates potential to reduce decision-making and establish prompt stop criteria for more efficient SAM-assisted segmentation…”

“We employ 4 variants (SR, SMP, PV, BALD in Sec.”

“Variance-based or information-theoretic methods (estimated by entropy or mutual information), such as Bayesian Active Learning by Disagreement (BALD) [22], are considered under this category.”

“We validate DCoM through an extensive empirical study using various vision datasets (Section 4), demonstrating that DCoM consistently achieves superior performance across all budget ranges.”

“The BALD (Bayesian Active Learning by Disagreement) algorithm, used by Houlsby et al. (2011) and later (Kirsch et al. 2019), enhances active learning by selecting data points that maximize the mutual information between the model’s predictions and its parameters by identifying samples where the…”

“Additionally, LADA (Kim Moreover, approaches like the Core-Set Approach (Sener and Savarese, 2017) and Bayesian Active Learning (BALD) (Houlsby et al., 2011) offer strategies for selecting informative instances, with Core-Set identifying a compact, diverse subset of unlabeled data, and BALD…”

“A higher variability in these outputs for unlabeled data samples indicates greater uncertainty, suggesting the prioritiza-tionofsamplesforlabeling[36].”

“Bayesian AL [8,9] is a subset of AL where the model trained is Bayesian.”

“Shoham & Avron (2023) recently analyzed experimental designs for global one-shot learning in overparameterized models.”

“Overall then, our approach differs from previous work in mainly two aspects: first, the experiment design criteria differs from information theoretic approaches by using the expected control performance in order to focus learning in control-relevant regions, and second, our approach to solving the…”

“DataCOPE deals with a fundamentally different task from active learning (Lewis, 1995; Sebastiani and Wynn, 2000; Mussmann and Liang, 2018; Houlsby et al., 2011; Kirsch et al., 2019; Nguyen et al., 2022).”

“Popular approaches rely on sequentially picking one data point per round [Houlsby et al., 2011, Gal et al., 2017] though the addition of a single data point causes minimal change to validation accuracy while increasing the cost of model retraining.”

“Within these heuristics, Bayesian Active Learning by Dis-agreement (BALD) (Houlsby et al. 2011) performs extraor-dinarily well by using the mutual information between the model prediction and the model parameters to estimate sam-ples’ acquisition scores.”

“In this flavor, there are several works specifically focused on applications such as large-scale classification, preference learning and posterior approximation (Pinsler et al., 2019; Houlsby et al., 2011; Kandasamy et al., 2017b).”

“If high response variability stems mainly from dispersion across values of the focus, this indicates that each value of the focus makes distinct predictions, facilitating identification of the correct value (Houlsby et al., 2011).”

“The original BALD criterion [13] is formulated as the conditional mutual information I(θ, y | x,Dtrain) between unknown (unobserved) output y and latent parameters θ, conditioned on input variable x and observed data Dtrain.”

“The first is Mutual Information (MI) or BALD [7]: UMI = − ∑C c=1 P̄ c log P̄ c+ 1T ∑T t=1 ∑C c=1 P c t logP c t .”

“CIFAR-10 Next, we evaluate absolute model performance on CIFAR-10 [21], comparing traditional AL methods (BALD [17], Entropy [32]) with our novel Fig.”

“Many active learning approaches, especially the ones that are not transductive, maximize the expected information gain, also called BALD (Houlsby et al., 2011), as an acquisition function—or, equivalently, minimize the model uncertainty.”

“Recent works, including both Bayesian (Houlsby et al., 2011; Gal et al., 2017; Kirsch et al., 2019) and non-Bayesian approaches (Sener & Savarese, 2017; Ash et al.”

“Our proposed acquisition function is defined as the mutual information (Houlsby et al., 2011; Gal et al., 2017) between Q̃k(x) and the BNN parameters ω, i.”

“In particular, flexible Bayesian methods are useful for indicating disagreement in the posterior distribution, which can be settled through external information like human labels.(98)”

“The recent acquisition function calculates the hypothetical disagreement by f θ on a data instance, Houlsby et al. 2011).”

“Others have proposed collecting as much information as possible by prioritizing inspection of uncertain samples [22], [23], [24].”

“The method in [12] applies predictive entropy to the Gaussian Process Classifier to develop a Bayesian approach for active learning called BALD.”

“Bayesian Active Learning by Disagreement (BALD) (Houlsby et al. 2011; Gal, Islam, and Ghahramani 2017; Siddhant and Lipton 2018) measures uncertainty according to the probabilistic distribution of model outputs via Bayesian inference (Zhu et al.”

“…uncertainty is re-quiredIn fact, there are some uncertainty metrics which explicitly need the epistemic uncertainty, such as predictive variance for regression [Gal and Ghahramani, 2016b] and Bayesian Active Learning by Disagreement objective (BALD) [Houlsby et al. , 2011] for classification.”

“We highlight challenges in applying traditional uncertainty-based approaches [9, 11] to this problem, showing that different types of the agent uncertainty become misleading signals for making query decisions.”

“We take BALD [20] as an uncertainty estimate.”

“Selecting unlabeled data follows the flow of original uncertainty pool-based active learning with Bayesian active learning by disagreement (BALD) [20], and highly uncertain”

“(2) BALD (Bayesian Active Learning By Disagreement) looks for the superpixel x that maximizes the decrease in conditional entropy caused by the posterior [9, 16]: x∗ = arg max x∈Du H[y|x,Du]− Ep(ω|Du)H[y|x,Du]”

“…explicitly based on defining uncertainty measures, which is typically taken to be entropy to yield the expected information gain [43, 53], though they can vary in what variables are targeted for learning, e.g., in AL, we can either take w to be model parameters [29] or downstream predictions [62].”

“Bayesian active learning (Bickford Smith et al, 2023, 2024; Gal et al, 2017; Houlsby et al, 2011; Kirsch et al, 2019; MacKay, 1992a,b) is the application of BED to sequentially selecting data to use in training a predictive model for outputs, ˜ y ∈ Y given inputs ˜ x ∈ X .”

“This principle underlies acquisition functions such as BALD (Houlsby et al., 2011) and Batch-BALD (Kirsch et al., 2019).”

“The corresponding Expected Information Gain, defined as EIG ( Ω ; ) , is the difference between the prior entropy of Ω and the expected posterior entropy after observing Y NV S i [19]: where ) .”

“On the other hand, high epistemic uncertainty signals that collecting more training data in this region is likely to improve the model, which makes it suitable for guiding model development and active learning (Houlsby et al., 2011; Gal et al., 2017; Smith et al., 2023).”

“This objective is indirect; reducing global parameter uncertainty does not guarantee a targeted reduction in CATE uncertainty (Houlsby et al., 2011; Jesson et al., 2021).”

“Active learning uses the information gain of the probabilistic surrogate model to guide the selection, such as BALD (Houlsby et al., 2011) and EPIG (Smith et al., 2023); therefore, it is an exploration-only process.”

“Their difference equals the mutual information (BALD score [7]), or Jensen–Shannon–style divergence between experts.”

“…Gain (EIG) EIG maximizes the expected reduction in uncertainty about the model’s predictions after observing a concept label, which can be approximated by computing the difference between the expected entropy and the entropy of the expected prediction after sampling (Houlsby et al., 2011).”

“This technique was initially introduced for the Gaussian Process Classifier (Houlsby et al., 2011) and later adopted by other researchers (e.g., Gal, 2016; Gawlikowski et al., 2023).”

“The Bayesian Active Learning by Disagreement (BALD) approach, proposed by Houlsby et al. [35], was later adapted for causal inference by Jes-son et al. [36].”

“Stage 3: Human Refinement implements Bayesian Active Learning by Disagreement (BALD) [16] to prioritize ambiguous cases (e.g., neologisms like (cid:2037)(cid:1996)ﺍ(cid:1960)(cid:1617) , ”tires”) for expert review.”

“IG-based approaches are to be contrasted against a popular learning policy called “Bayesian active learning by disagreement”, or BALD (Houlsby et al., 2011), which minimizes uncertainty in the model’s parameters and may lead to suboptimal decisions in various scenarios (Smith et al., 2023).”

“This concept of “epistemic uncertainty reduction” is particularly useful in data acquisition tasks following Bayesian principles (Houlsby et al., 2011).”

“The AL algorithms for classification problems are heavily discussed compared with the regression problem [for example, Houlsby et al., 2011, Zhao et al., 2021, Bickford Smith et al., 2023].”

“To capture the uncertainty in the model, a common measure is the conditional differential entropy (CDE): After we get the uncertainty of the distribution, the epistemic uncertainty can be expressed as described by [15]: where 𝐼 (·) refers to the mutual information and 𝜃 is the parameters of…”

“Active learning based on Bayesian approaches has been extensively studied (Daniel et al., 2014; Biyik & Sadigh, 2018; Sadigh et al., 2017; Bıyık et al., 2024; Houlsby et al., 2011; Biyik et al., 2024), and many of these methods can be interpreted through the lens of hypothesis space removal.”

“However, disentangling epistemic from aleatoric uncertainty is a non-trivial task, as the conventional decomposition (Houlsby et al., 2011) of Shannon entropy (total uncertainty) into mutual information (epistemic uncertainty) and conditional entropy (aleatoric uncertainty) can yield inconsistent…”

“We stress that candidates x are here not samples from the belief density but are instead provided by some other unknown process, such as an active learning method [Houlsby et al., 2011].”

“Additionally, methods like mutual information between predictions and model posterior have been used to target data points that most influence the model’s posterior distribution [56].”

“Hybrid sampling strategies are proposed recently [24, 1, 25, 20], combining representative/diversity and uncertainty sampling, like BADGE [1], where the authors propose to use the gradient magnitude with respect to the classifier’s parameters as a measure of uncertainty.”

“To address this issue, acquisition functions designed explicitly for batch acquisition with NN classifiers have been developed, such as BatchBALD Kirsch et al. (2019).”

“Inspired by uncertainty estimation techniques commonly employed in active learning (AL) pipelines [21,46], we introduce a Bayesian self-training strategy that leverages an uncertainty-aware filtering technique for generating more reliable pseudo-labels.”

“In the CausalBALD (Jesson et al., 2021) approach, the acquisition function is altered to specifically target areas where the distributions of different treatment groups overlap, thereby maximizing sample efficiency for learning personalised treatment effects.”

“At each BO iteration t , with probability p fb , we add three expert feedback { ( x i 0 , x i 1 , ℓ i ) } 3 i =1 , where each of the pair ( x i 0 , x i 1 ) is proposed by either random selection or the BALD active learning algorithm (Houlsby et al., 2011).”

“Unlabeled samples can be valued by using the prediction disagreement from a probabilistic model (Houlsby et al., 2011), this disagreement can also be obtained from multiple models (Freund et al., 1997).”

“…selected which maximize the information gain at a single timestep (a greedy, but often good, approximation to optimizing over sequences of actions [4]): (2) One way to solve this maximization is via a simple pool-based approach: first generate a collection of M unlabeled samples, then score each…”

“The Bayesian active learning by disagreement (BALD) [36] heuristic relates to the mutual information between the probabilities of the predictions.”

“This disagreement is typically assessed by vote entropy (Argamon-Engelson and Dagan 1999), Kullback-Leibler divergence (McCallum and Nigam 1998), or mutual information between model predictions and model parameters, as employed in context of Bayesian Active Learning by Disagreement (BALD) (Houlsby et al. 2011).”

“In this specific acquisition function, the differential entropy serves as a proxy for information gain, meaning that it aims to select the next sample point x * that would maximize the information about the underlying latent function (Houlsby, Huszár, Ghahramani, & Lengyel, 2011).”

“Bayesian active learning by disagreement (BALD) [8] calculates mutual information between a model’s predictions and its parameters to identify top candidates for labeling based on high overall uncertainty and prediction disagreement.”

“It is shown in [11] that this is equivalent to BALD [12] in a regression setting.”

“Since the entropy does not represent model uncertainty, a Bayesian method for deep active learning was proposed in DBAL [14], which combined Bayesian convolutional neural networks (CNNs) [12, 13] with Bayesian active learning [18].”

“In 2011, [24] uses GP classification in the form of information theory to find the best solution with the biggest entropy.”

“In preliminary experiments, we studied the following: maximum entropy (Shannon, 1948), BALD (Houlsby et al., 2011), variation ratios (Freeman, 1965), and mean standard deviation.”

“Vote count in our baseline CRF is a decent query strategy, but information-theoretic quantities [4, 12] are among the state-of-the-art.”

“BALD (Bayesian Active Learning by Disagreement) [63], using an information-theoretic approach, in addition to uncertainty, measures a selected sample’s impact on the network’s uncertainty.”