Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs

On the CoQA benchmark, ASPIRE improves AUACC from 91.23% to 92.63%.

“on the CoQA benchmark, our method improves the AUACC from 91.23% to 92.63%”

After training θp via soft prompt tuning, the accuracy of LLMs is improved significantly.

“The results in Table 3 show that after training θ p via soft prompt tuning, the accuracy of LLMs is improved significantly.”

With approximately 2k training samples, ASPIRE can outperform baselines without soft prompt tuning significantly on different datasets.

“with ∼2k samples, ASPIRE can outperform the baselines without soft prompt tuning significantly on different datasets”

More training samples lead to enhanced performance for ASPIRE.

“More training samples lead to enhanced performance”

ASPIRE outperforms state-of-the-art selective prediction methods on question-answering datasets (CoQA, TriviaQA, and SQuAD) using OPT and GPT-2 models.

“We evaluate our method on a variety of question-answering (QA) datasets and show that it outperforms state-of-the-art selective prediction methods.”

• Also stated in: § 1 Introduction · plain

  “We show that ASPIRE achieves the state-of-the-art selective prediction performance on three question answering datasets: CoQA, TriviaQA and SQuAD, using OPT and GPT-2 models.”

• Also stated in: § 6 Experiments · plain

  “(A1) By learning self-evaluation, we can significantly improve selective prediction performance across different datasets and LLMs (see Table 1).”

Using decoding algorithms that can sample different high-likelihood answers (e.g., beam search) is important for ASPIRE to achieve good selective prediction performance.

“(A2) Using decoding algorithms that can sample different high-likelihood answers as A (e.g., beam search) is important for ASPIRE to achieve good selective prediction performance (see Table 4).”

On the CoQA benchmark, ASPIRE improves AUROC from 74.61% to 80.25%.

“improves the AUROC from 74.61% to 80.25%.”

The computational complexity of ASPIRE at test time is O(l_max), where l_max is the maximum length of the generated output sequence.

“Lastly, we note that the computational complexity of the proposed method at test time is O(l max ) with l max being the maximum length of the generated output sequence.”

More training samples lead to performance improvement.

“The results in Table 5 show that more training samples lead to performance improvement”

After prompt tuning, the accuracy of LLMs gets better.

“as the accuracy gets better”

• Also stated in: § I Complete Results · boosted

  “The results show that after training θ p via soft prompt tuning, the accuracy of LLMs is improved significantly.”

With 2K training samples, ASPIRE can outperform the baselines without soft prompt tuning by a large margin across different datasets.

“with 2K training samples, ASPIRE can outperform the baselines without soft prompt tuning by a large margin across different datasets.”

Using multinomial sampling with T = 2.0 or T = 0.1 leads to worse performance compared to other decoding algorithms.

“The results in Table 4 show that using multinomial sampling with T = 2.0 or T = 0.1 leads to worse performance compared to other decoding algorithms.”

On the TriviaQA dataset, the pre-trained OPT-30B model has better accuracy than the adapted OPT-2.7B model.

“on the TriviaQA dataset, although the pre-trained OPT-30B model has better accuracy than the adapted OPT-2.7Bmodel”

The proposed method ASPIRE significantly outperforms self-consistency.

“the proposed method ASPIRE still significantly outperforms self-consistency as shown in Table 18.”

After prompt tuning, perplexity becomes more meaningful in separating correct and wrong answers.

“and the perplexity becomes more meaningful in separating correct and wrong answers.”

The proposed ASPIRE with the adapted OPT-2.7B model significantly outperforms the Self-eval and P(True) baselines with the pre-trained OPT-30B model in selective prediction performance.

“the proposed ASPIRE with the adapted OPT-2.7Bmodel can significantly outperform the Self-eval and P(True) baselines with the pre-trained OPT-30B model in selective prediction performance.”

ASPIRE significantly outperforms the baselines across OPT and GPT2 models of different sizes for different values of the Rouge threshold γ.

“Additional results in Appendix I show that ASPIRE significantly outperforms the baselines across OPT and GPT2 models of different sizes for different values of the Rouge threshold γ.”

After prompt tuning, the AUACC of different methods is significantly improved.

“The results also show that after prompt tuning, the AUACC of different methods is significantly improved”

Setting α = 0.25 leads to the best performance.

“The results in Table 2 show that setting α = 0.25 leads to the best performance”

• Also stated in: § J The Effect of the Hyper-parameter α · plain

  “The results in Table 17 show that setting α = 0.25 leads to the best performance across different datasets and different models.”

On the TriviaQA dataset, the Self-eval and P(True) baselines with the pre-trained OPT-30B model have much worse selective prediction performance compared to the proposed ASPIRE with the adapted OPT-2.7B model.

“the Self-eval and P(True) baselines with the pre-trained OPT-30B model have much worse selective prediction performance compared to the proposed ASPIRE with the adapted OPT-2.7Bmodel.”

If γ = 0.9, then no wrong answers would be labeled as correct answers.

“We find that if we set γ = 0.9, then no wrong answers would be labeled as correct answers.”

On the CoQA and SQuAD datasets, the adapted OPT-2.7B model outperforms the pre-trained OPT-30B model in accuracy.

“On the CoQA and SQuAD datasets, the adapted OPT-2.7Bcan even outperform the pre-trained OPT-30B”

Using beam search leads to similar performance as using multinomial sampling with T = 1.

“The results also show that using beam search leads to similar performance as using multinomial sampling with T = 1.”

The proposed method ASPIRE significantly outperforms the baselines in terms of AUACC and AUROC metrics across different datasets and LLMs.

“The results in Table 1 show that the proposed method ASPIRE significantly outperforms the baselines in terms of the AUACC and AUROC metrics across different datasets and LLMs.”

Beam search decoding yields higher AUACC and AUROC than multinomial sampling for adapted OPT-2.7B with θp on CoQA, TriviaQA, and SQuAD.

“Adapted OPT-2.7B with θ p | Multinomial (T=0.1) Multinomial (T=1.0) Multinomial (T=2.0) Beam search | 92.04 92.60 92.02 92.63 | | 77.96 79.86 77.91 80.25 | 55.09 55.15 53.80 55.06 | 84.28 84.29 82.40 84.44 | 94.24 94.57 94.15 94.73 | 80.52 82.08 80.42 82.60”

Fine-tuning LLMs on training data from the target question-answering task using the standard LLM training loss can improve selective prediction performance.

“Fine-tuning LLMs on training data from the target question answering task using the standard LLM training loss can improve selective prediction performance.”

ASPIRE is useful for practical applications where high selective prediction performance and low inference costs are desired after deploying the LLM.

“Thus, the proposed method is useful for practical applications where high selective prediction performance and low inference costs are desired, after deploying the LLM.”

Predictive entropy and semantic entropy methods have a computational complexity of O(m * l_max), where m is the number of output sequences sampled for uncertainty estimation.

“The predictive entropy and semantic entropy methods have a complexity of O(m • l max ) where m is the number of output sequences sampled for uncertainty estimation, which is much larger than that of our method.”

Even after fine-tuning, some wrong answers may still have high likelihood and be generated by the LLM at test time.

“Even after fine-tuning, some wrong answers may still have high likelihood and be generated by the LLM at test time.”

Distinguishing correct and incorrect answers based on likelihood scores alone is challenging.

“Therefore, distinguishing correct and incorrect answers based on likelihood scores alone is a challenging task.”

Maximizing the likelihood of the ground-truth answer is not the same as minimizing the likelihood of wrong answers, because LLMs generate output sequences auto-regressively.

“However, maximizing the likelihood of the ground-truth answer is not the same as minimizing the likelihood of the wrong answers, since LLMs generate output sequences in an auto-regressive way.”

Freezing θ and the learned θp can avoid changing the prediction behaviors of the LLM when learning self-evaluation.

“Since the output sequence generation only depends on θ and θ p , freezing θ and the learned θ p can avoid changing the prediction behaviors of the LLM when learning self-evaluation.”

Training LLMs on a subset of the training data from question-answering tasks allows them to learn to distinguish between correct and incorrect answers on their own.

“We do this by training LLMs on a subset of the training data from the question-answering tasks.This allows the LLMs to learn to distinguish between correct and incorrect answers on their own.”

ASPIRE is much less computationally expensive than solutions that require generating multiple output sequences to obtain the uncertainty measure.

“This makes our method much less computationally expensive than solutions that require generating multiple output sequences to obtain the uncertainty measure.”

Self-consistency produces discrete selection scores, making it easier to achieve high AUACC compared to approaches that produce continuous selection scores.

“Since self-consistency produces discrete selection scores (in the above setting, the number of possible selection scores is 10) and we use the composite trapezoidal rule to compute AUACC, it is easier for self-consistency to achieve high AUACC compared to those approaches that produce continuous selection scores.”

The computational complexity of the self-eval method is O(l_max).

“For the self-eval method, the computational complexity is O(l max ) since we only need one additional forward pass to obtain the self-eval score.”

Setting a high temperature (T = 2.0) for multinomial sampling samples some random answers that might not have high-likelihood.

“If we set a high temperature (T = 2.0) for multinomial sampling, then we sample some random answers that might not have high-likelihood.”

The computational complexity of the P(True) method is O(m • l_max).

“For the P(True) method, the computational complexity is O(m • l max ) since we need to additionally generate m answers and need one forward pass to compute the P(True) score.”

The computational complexity of the proposed method ASPIRE is O(l_max).

“For the proposed method ASPIRE, the computational complexity is O(l max ) since we only need additional one forward pass to obtain the learned self-eval score.”

The computational complexity of the semantic entropy method is O(m • l_max).

“Thus, the computational complexity for semantic entropy is O(m • l max ).”

Self-evaluation approaches are not effective for high capacity LLMs.

“These demonstrate that the selfevaluation approaches are not effective for high capacity LLMs”

ASPIRE can significantly improve selective prediction performance even with a limited number of training samples.

“This underlines that our method, ASPIRE, can significantly improve selective prediction performance even with limited number of training samples.”

Setting a low temperature (T = 0.1) for multinomial sampling repeatedly samples the same high-likelihood answers.

“If we set a low temperature (T = 0.1) for multinomial sampling, then we repeatedly sample the same high-likelihood answers.”

The computational complexity of the predictive entropy method is O(m • l_max).

“For the predictive entropy method, the computational complexity is O(m • l max ) since we need to additionally generate m answers and compute the likelihood of those m answers.”

It is possible to adapt a smaller LLM to achieve better accuracy than a much larger LLM.

“which demonstrates that it is possible to adapt a smaller LLM to achieve better accuracy than a much larger LLM.”

The complexity of the proposed method ASPIRE in training time is the same as that of standard soft prompt tuning.

“Therefore, the complexity of the proposed method ASPIRE in the training time is the same as that of the standard soft prompt tuning.”

The complexity of training θp using the training objective (12) is the same as the complexity of standard soft prompt tuning.

“The complexity of training θ p using the training objective ( 12) is the same as the complexity of the standard soft prompt tuning.”

The computational complexity of the perplexity method is O(l_max).

“For the perplexity method, the computational complexity is O(l max ) since we only need additional one forward pass to obtain the perplexity score.”