On scalable oversight with weak LLMs judging strong LLMs

Compared to Khan et al. (2024), we do not see such a clear trend of the advantage for correct answers increasing with increasing Elo.

“However, compared to Khan et al. (2024) we don't see such a clear trend of this advantage increasing with increasing Elo.”

• cites Khan et al. (2024) sourceSupported

The results of Khan et al. (2024) on QuALITY generalise to other extractive tasks, even while using a different set of models.

“The results of Khan et al. (2024) on QuALITY generalise to other extractive tasks, even while using a different set of models: stronger judge models tend to be more accurate across protocols.”

• cites Khan et al. (2024) sourcePartly supported

  Khan et al. evaluate judges on QuALITY, and Figure 8 (Appendix A.2, p. 12) reports that stronger judge models are more accurate in debate and the baselines. It reports only minimal improvement in consultancy, so the trend does not clearly hold across all protocols. The extension to other extractive tasks and a different set of models is the citing authors’ conclusion, not a result tested in this source. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Denison et al. (2024) showed that current large language models can generalise behaviours which exploit inaccurate training signals.

“Denison et al. (2024) recently showed that current large language models (LLMs) can generalise behaviours which exploit inaccurate training signals: generalising from simple behaviours, such as sycophancy, to more complex ones, such as reward tampering, in which the model modifies its own reward administration.”

• cites Denison et al. (2024) sourceSupported

On closed tasks, for Pro 1.0, the difference between debate and QA without article is not statistically significant (p = 0.1).

“For Pro 1.0, the difference between debate and QA without article is not statistically significant (𝑝 = 0.1).”

Reducing best-of-N sampling for debater responses from 4 to 1 does not substantially reduce judge accuracy.

“We consider the debate protocol and reduce the best-of-N (BoN) sampling for debater responses from 4 to 1, i.e. standard sampling.We do not see substantial reduction in judge accuracy, suggesting our implementation of best-of-N is not helpful.”

In tasks without information asymmetry, when the judge is weaker than the debaters but not too weak, debate has either small or no advantage over QA without article.

“For other tasks, when the judge is weaker than the debaters (but not too weak), we find either small or no advantage to debate over QA without article.”

On closed tasks, for Pro 1.5, debate leads to worse judge accuracy than QA without article.

“For Pro 1.5, when the judge is the same as the debaters, debate actually leads to worse judge accuracy than QA without article, similar to some of the findings reported in Fig 31 of Khan et al. (2024).”

Reducing the number of turns from 3 to 1 in consultancy and debate has no significant effect on judge accuracy.

“We investigate what happens when we reduce the number of turns from 3 to 1 in consultancy and debate.We find no significant effect from the number of turns.”

Varying judge prompting from 0-shot to 6-shot does not show significant improvement.

“We look at varying 0-shot to 6-shot prompting for the judge, balancing the order of the correct answer across examples.Overall, the results do not show significant improvement with 6-shots.”

On multimodal tasks, consultancy is significantly worse than other protocols.

“Multimodal: consultancy is significantly worse than other protocols.”

On closed tasks, for Gemma7B, QA without article does better than debate.

“For Gemma7B, our weakest judge, QA without article does better than debate (as in the extractive case).”

On closed tasks, for GPT3.5, debate does better than QA without article (p = 0.037).

“For GPT3.5, debate does better than QA without article (with 𝑝 = 0.037).”

On closed tasks, debate increases judge accuracy compared to consultancy.

“We see debate increases judge accuracy compared to consultancy on closed tasks (previously this was only seen in the extractive task, QuALITY).”

In extractive QA tasks with information asymmetry, debate does not outperform QA with article.

“In extractive QA tasks with information asymmetry, debate outperforms QA without article as in the single task of Khan et al. (2024), but not QA with article.”

Debate outperforms consultancy across all tasks when the consultant is randomly assigned to argue for the correct or incorrect answer.

“We find that debate outperforms consultancy across all tasks when the consultant is randomly assigned to argue for the correct/incorrect answer.”

• Also stated in: § 1. Introduction · plain

  “We find that debate consistently outperforms consultancy across all tasks, previously only shown on a single extractive QA task in Khan et al. (2024).”

On multimodal tasks, the difference between debate and QA without article is not statistically significant (p = 0.11).

“The difference between debate and QA without article is not statistically significant (𝑝 = 0.11).”

In extractive QA tasks with information asymmetry, debate outperforms QA without article.

“In extractive QA tasks with information asymmetry, debate outperforms QA without article as in the single task of Khan et al. (2024), but not QA with article.”

Stronger debater models lead to higher judge accuracy, including for a weaker judge, across a range of tasks.

“Comparing across debater models/settings, we calculate Elo scores and show that stronger debaters lead to higher judge accuracy (including for a weaker judge) across a range of tasks, see Figure 4.”

• Also stated in: § Abstract · plain

  “Further, we find that stronger debater models increase judge accuracy, though more modestly than in previous studies.”

The judge accuracy does not differ with a low p-value between debate and consultancy protocols for the multimodal case.

“We note that the judge accuracy differs with a low 𝑝-value between debate and consultancy protocols, though not for the multimodal case (in which we only have one task).”

• AssumptionThe authors state this as an assumption: For each split (by model, protocol, dataset/task-type) the data is IID and approximately normally distributed about the mean.

We observe no consistent benefit to using best-of-N sampling on debater responses.

“We observe no consistent benefit to using best-of-N sampling on debater responses.”

In open debate, when the debater chooses incorrectly, the judge does a lot better at discerning this than in open consultancy.

“But when the debater chooses incorrectly, the judge does a lot better at discerning this.”

Judge decisions are more sensitive to debaters on extractive tasks than closed tasks.

“Judge decisions are more sensitive to debaters on extractive tasks than closed tasks.”

On extractive tasks, QA with article is best for all models.

“QA with article is best for all models, indicating that information is best used directly, even by a weak model, rather than filtered through oversight.”

For extractive tasks, open consultancy leads to higher judge accuracy than open debate.

“For extractive tasks, open consultancy does lead to higher judge accuracy than open debate, but in closed/multimodal we see roughly equal judge accuracy for our strongest debate judge, though lower accuracy for weaker debate judges than for corresponding consultancy judges.”

We observe no significant difference between the two turn styles.

“We observe no significant difference between the two turn styles.”

• Also stated in: § I.2. Sample debates · plain

  “However, we don't see a significant difference between the turn styles”

On all tasks, open consultancy leads to greater or equal win-rate than open debate.

“On all tasks, open consultancy leads to greater (or equal) win-rate than open debate, meaning the judge tends to agree with the consultant's choice more in a consultancy protocol, perhaps because the consultant doesn't face as strong a challenge as in open debate (where there is an equal strength opponent).”

On closed tasks there is less of a clear trend for the advantage of correct answers increasing with Elo.

“For extractive tasks it appears it gets slightly easier for honest debaters as models increase their convincingness, whereas on closed tasks there is less of a clear trend.”

In open debate, when the debater chooses correctly, the judge does a bit worse than in open consultancy.

“When the debater chooses correctly, the judge does a bit worse than in open consultancy.”

Weak judges do slightly better in consultancy compared to debate when the consultant/debater chooses correctly.

“weak judges do slightly better in consultancy compared to debate when the consultant/debater chooses correctly -however, weak judges do much worse in consultancy compared to debate when the consultant chooses incorrectly.”

• AssumptionRests on an assumption: The weak-strong dynamics of humans supervising superhuman AI can be analogised by the study's setup.

For extractive tasks it appears it gets slightly easier for honest debaters as models increase their convincingness.

“For extractive tasks it appears it gets slightly easier for honest debaters as models increase their convincingness, whereas on closed tasks there is less of a clear trend.”

Both Gemini-family judges tend to assign relatively lower Elo scores to GPT3.5 than expected based on relative capabilities of the models.

“We notice overall that both Gemini-family judges tends to assign relatively lower Elo scores to GPT3.5 than expected based on relative capabilities of the models.”

Stronger models tend to score higher Elo.

“We see it is often easier to argue for the truth, in particular on closed tasks, and that stronger models tend to score higher Elo.”

We observe no benefit to increasing the number of turns of each protocol.

“We observe no benefit to increasing the number of turns of each protocol.”

Weak judges do much worse in consultancy compared to debate when the consultant chooses incorrectly.

“weak judges do slightly better in consultancy compared to debate when the consultant/debater chooses correctly -however, weak judges do much worse in consultancy compared to debate when the consultant chooses incorrectly.”

• AssumptionRests on an assumption: The weak-strong dynamics of humans supervising superhuman AI can be analogised by the study's setup.

In open consultancy, the judge is equally convinced by the consultant, whether or not the consultant has chosen to argue for the correct answer.

“In open consultancy, the judge is equally convinced by the consultant, whether or not the consultant has chosen to argue for the correct answer.”

Changes to the inference setup (few-shot, chain-of-thought, turn style, swapping answer order) seem to have little effect.

“Changes to the inference setup (few-shot, chain-of-thought, turn style, swapping answer order) seem to have little effect.”

On TruthfulQA, in 6 of the 12 errors made by the judge, the posed question is naturally ambiguous and the dishonest debater succeeds in interpreting the question in a manner that supports their answer.

“In 6 of the 12 errors made by the judge, the posed question is naturally ambiguous and the dishonest debater succeeds in interpreting the question in a manner that supports their answer.”

On previously unstudied closed QA tasks, weak judges achieve higher accuracy in the debate protocol than in consultancy.

“We found that, on previously unstudied closed QA tasks, weak judges achieve higher accuracy in the debate protocol, than consultancy, and around the same as direct question answering.”

• AssumptionRests on an assumption: The weak-strong dynamics of humans supervising superhuman AI can be analogised by the study's setup.

In closed and multimodal tasks, weaker debate judges have lower accuracy than corresponding consultancy judges.

“For extractive tasks, open consultancy does lead to higher judge accuracy than open debate, but in closed/multimodal we see roughly equal judge accuracy for our strongest debate judge, though lower accuracy for weaker debate judges than for corresponding consultancy judges.”

On extractive tasks, stronger judge models tend to be more accurate across protocols.

“The results of Khan et al. (2024) on QuALITY generalise to other extractive tasks, even while using a different set of models: stronger judge models tend to be more accurate across protocols.”

Stronger debate models (as measured by Elo) tend to lead to marginally higher judge accuracy, though the effect is relatively small.

“A summary of our findings are that stronger debate models (as measured by Elo) tend to lead to marginally higher judge accuracy, though the effect is relatively small.”

We observe no significant difference between using single and both orderings.

“We observe no significant difference between using single and both orderings.”

We observe no consistent benefit to using few-shot prompting.

“We observe no consistent benefit to using few-shot prompting.”

On previously unstudied closed QA tasks, weak judges achieve around the same accuracy in the debate protocol as in direct question answering.

“We found that, on previously unstudied closed QA tasks, weak judges achieve higher accuracy in the debate protocol, than consultancy, and around the same as direct question answering.”

• AssumptionRests on an assumption: The weak-strong dynamics of humans supervising superhuman AI can be analogised by the study's setup.

All debaters display a positive difference between correct and incorrect Elo scores, indicating an advantage for being assigned the correct answer.

“All debaters display a positive difference, indicating an advantage for being assigned the correct answer.”

The judge accuracy differs with a low p-value between debate and consultancy protocols.

“We note that the judge accuracy differs with a low 𝑝-value between debate and consultancy protocols, though not for the multimodal case (in which we only have one task).”

• AssumptionThe authors state this as an assumption: For each split (by model, protocol, dataset/task-type) the data is IID and approximately normally distributed about the mean.

In open debate, the judge follows the debater's choice less frequently than in open consultancy.

“In open debate, in contrast, the judge follows the debater's choice less frequently than in open consultancy.”

It is often easier to argue for the truth, in particular on closed tasks.

“We see it is often easier to argue for the truth, in particular on closed tasks, and that stronger models tend to score higher Elo.”

In closed and multimodal tasks, for the strongest debate judge, open consultancy and open debate have roughly equal judge accuracy.

“For extractive tasks, open consultancy does lead to higher judge accuracy than open debate, but in closed/multimodal we see roughly equal judge accuracy for our strongest debate judge, though lower accuracy for weaker debate judges than for corresponding consultancy judges.”

As a consequence of sampling the same question twice with switched answer order, some of the samples will be correlated.

“As a consequence, some of the samples will be correlated.”

Information is best used directly, even by a weak model, rather than filtered through oversight.

“QA with article is best for all models, indicating that information is best used directly, even by a weak model, rather than filtered through oversight.”

Debate is likely more promising as a scalable oversight protocol than consultancy.

“Overall, we think these results suggest debate is likely more promising as a scalable oversight protocol than consultancy, though we emphasise that our findings only apply in the inferenceonly setting, and that a proper evaluation of scalable oversight protocols used for training may reveal other findings.”

The paper's findings apply only in the inference-only setting.

“Overall, we think these results suggest debate is likely more promising as a scalable oversight protocol than consultancy, though we emphasise that our findings only apply in the inferenceonly setting, and that a proper evaluation of scalable oversight protocols used for training may reveal other findings.”

Debate between LLMs has been primarily studied on a reading comprehension task, where the judge is artificially limited by only being allowed to see short verified quotes from the text provided by the LLM debaters.

“Thus far, debate between LLMs has been primarily studied on a reading comprehension task, where the judge is artificially limited by only being allowed to see short verified quotes from the text provided by the LLM debaters.”

Debate satisfies a key objective of scalable oversight-that judge accuracy increases as AI capabilities scale.

“This provides some evidence that debate satisfies a key objective of scalable oversight -that judge accuracy increases as AI capabilities scale.”

The use of information asymmetry limits the ability to fully evaluate the qualitative advantages of debate predicted by theory.

“While providing initial promising evidence on the power of debate, the use of this type of information asymmetry limits the ability to fully evaluate the qualitative advantages of debate predicted by theory.”

The ablation showing no improvement with number of turns is a mildly negative sign for debate as a training protocol.

“The ablation showing no improvement with number of turns is also a mildly negative sign, though perhaps will differ under training for debate, if models learn to use further rounds to better argue against the claim of their opponent in the prior round.”

It is sufficient to just randomise the answer position, rather than evaluating with both answer positions.

“This suggests it's sufficient to just randomise the answer position, rather than evaluating with both answer positions.”

The finding that stronger debate models tend to lead to marginally higher judge accuracy is a weakly positive indication for debate as a scalable oversight protocol.

“The takeaway is that this is a weakly positive indication for debate as a scalable oversight protocol -though one may have hoped to see a stronger effect on judge accuracy.”

The training signal provided by the weak judge in open debate is less likely to amplify incorrect answers than in open consultancy.

“Thus, the training signal provided by the weak judge in open debate is less likely to amplify incorrect answers than in open consultancy.”

The ablation on BoN revealing no improvement is a mildly negative sign for debate as a training protocol.

“The ablation on BoN revealing no improvement is a mildly negative sign for debate as a training protocol, as BoN is motivated as a proxy for training.”

Using weak judges to provide a training signal via consultancy runs the risk of amplifying the consultant's incorrect behavior.

“Thus, using weak judges to provide a training signal via consultancy runs the risk of amplifying the consultant's incorrect behavior.”

The potential asymmetry is unimportant.

“indicating the potential asymmetry is unimportant”