The Leaderboard Illusion

In principle, the BT model should still handle deprecation reliably because of the transitivity property.

“However, in principle, the BT model should still handle this reliably because of the transitivity property (Boubdir et al., 2024).”

• cites Boubdir et al., 2024 sourcePartly supported

  Section 3.1 describes transitivity as a desirable property that can allow model comparisons without direct head-to-head evaluations. But the paper discusses Elo rather than the BT model, does not address deprecation, and reports circumstances in which Elo rankings violate transitivity. It therefore does not establish that BT would handle deprecation reliably. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Overfitting is a particularly salient challenge in static evaluation settings, where fixed test sets are vulnerable to overfitting through repeated exposure, data contamination, or intentional tuning.

“Overfitting is a particularly salient challenge in static evaluation settings, where fixed test sets are vulnerable to overfitting through repeated exposure, data contamination, or intentional tuning (Deng et al., 2024;Golchin & Surdeanu, 2024;Roberts et al., 2023;Dong et al., 2024;Singh et al., 2024a).”

• cites Deng et al., 2024 sourcePartly supported

  The Introduction says benchmark data in training sets can inflate performance metrics, and §4.3 reports that deliberately fine-tuning on the MMLU test set produced a nearly 100% exact-match rate. This supports the risk from data contamination and intentional tuning, but the paper does not establish repeated exposure as a separate route to overfitting or explicitly compare static with other evaluation settings. Page numbers were not available in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Golchin & Surdeanu, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Roberts et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Various works have shown that LM-as-a-judge is correlated with human preferences.

“Various works have shown that this is correlated with human preferences (Dubois et al., 2023;Rafailov et al., 2023;Kim et al., 2023).”

• cites Dubois et al., 2023 sourceSupported
• cites Rafailov et al., 2023 sourceNot checked — no full text of the source
• cites Kim et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Benchmarks are rarely impartial and are shaped by the environments in which they are made.

“Research has found that benchmarks are rarely impartial and instead shaped by the environments in which the benchmarks are made”

• cites Aniba et al., 2010 sourcePartly supported

  The Conclusions and Perspectives section says benchmark design reflects a field’s scientific paradigm and that choices about what to include are statements of values. This supports the claim that benchmarks are shaped by their context, but the paper does not establish that benchmarks are rarely impartial. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Bartz-Beielstein et al., 2020 sourcePartly supported

  Section 4 (‘Algorithms’; page unavailable) calls the choice of algorithms ‘one of the most subjective choices’ in a benchmarking study, and Section 3 (‘Problem Instances’; page unavailable) says the choice of problem instances can heavily influence results. These passages support the point that benchmark design shapes outcomes, but they do not establish that benchmarks are rarely impartial or that they are shaped specifically by the environments in which they are made. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Static task-based leaderboards are often susceptible to data contamination and implicit overfitting.

“Static task-based leaderboards, such as Hugging Face's Open LLM Leaderboard (Fourrier et al., 2024;Gao et al., 2024) and OpenCompass (Contributors, 2023), aim to evaluate models across a broad range of skills but are often susceptible to data contamination and implicit overfitting”

• cites Deng et al., 2024 sourcePartly supported

  The Introduction says benchmark data can enter training sets and artificially inflate performance metrics, and the Contaminated Data section reports evidence of potential contamination in widely used benchmarks. The Limitations section only suggests possible overfitting in multi-choice tasks; it does not establish that static task-based leaderboards generally suffer from implicit overfitting. Page numbers are unavailable in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Golchin & Surdeanu, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Roberts et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Human voting has been shown to mitigate some biases associated with LLM-as-a-judge approaches.

“Human voting has been shown to mitigate some of the biases associated with using LLM-as-a-judge approaches”

• cites Raina et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Koo et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Shimabucoro et al., 2023 sourceCould not verify

  The Introduction and §§3.2–4.2 discuss biases in few-shot-learning performance estimators, not biases in LLM-as-a-judge approaches. Searches for human voting and human annotators judging language models found no passage addressing whether voting mitigates those biases. The indexed passages provide no page numbers. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Four models are positioned between the two identical checkpoints on the leaderboard.

“As a result, 4 models are positioned between our two identical checkpoints.”

Family A's models have a lower average Arena score across all models compared to Family B's.

“Although the performance of both model families is in a similar range, Family A's models have a lower average Arena score across all models compared to Family B's.”

Proprietary model providers collect significantly more test prompts and model battle outcomes than others.

“However, proprietary model providers collect significantly more test prompts and model battle outcomes than others.”

No private models were tested by academic labs during the observed period.

“Notably, we found that no private models were tested by academic labs during the observed period.”

Out of 243 public models, 205 have been silently deprecated.

“As shown in Figure 17, out of 243 public models, 205 have been silently deprecated.”

• Also stated in: § 5. Provide transparency into what models are being removed from the leaderboard. · plain

  “While 47 out of 243 public models are officially listed as deprecated, we measure that 205 have been silently removed by the organizers without informing the provider.”

OpenAI received an estimated 20.4% of all data on the arena.

“Providers like Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the arena, respectively.”

• Also stated in: § -Charles A. E. Goodhart · hedged

  “Google and OpenAI have received an estimated 19.2% and 20.4% of all test prompts on the arena, respectively.”

The paper finds large biases towards proprietary models in sampling rates.

“Our work finds large biases towards proprietary models in sampling rates.”

Deprecation disproportionately impacts open-weight and open-source models, creating large asymmetries in data access over time.

“We find deprecation disproportionately impacts open-weight and open-source models, creating large asymmetries in data access over time (Section 4.1).”

Private testing is an undisclosed policy.

“Private testing is an undisclosed policy”

Extreme disparities exist in access to data from Chatbot Arena.

“Extreme disparities exist in access to data from Chatbot Arena.”

The paper observes large differences in data access between providers, with 61.4% of all data going to proprietary providers.

“We observe large differences in data access between providers, with 61.4% of all data going to proprietary providers.”

The authors' open-weight model Aya Expanse appeared notably undersampled compared to proprietary models.

“we observed that our open-weight model appeared to be notably undersampled compared to proprietary models -a discrepancy that is further reflected in Figures 3, 4, and 5.”

• cites Dang et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Chatbot Arena frequently deprecates models.

“However, Chatbot Arena frequently deprecates models.”

Smaller startups, such as Reka, were found to have one active private variant live in the arena.

“In contrast, smaller startups, such as Reka, were found to have one active private variant live in the arena.”

Launching multiple model variants increased the amount of prompts collected from 5.9% with 1 variant to 19.4% with 3 variants.

“We note that even with our experiment of launching multiple model variants, we increased the amount of prompts collected from 5.9% with 1 variant to 19.4% with 3 variants.”

Proprietary models consistently received the largest share of data, ranging from 54.3% to 70.1%.

“Proprietary models consistently received the largest share of data-ranging from 54.3% to 70.1%.”

Cohere submitted 4 private variants as ablations for experiments measuring the lift from private testing.

“We note that during the same period, Cohere submitted 4 private variants-these ablations were part of experiments measuring the lift that could be expected from private testing that we detail in the experiments in Section 3.2 and Section 4.1.”

Even limited additional data can result in relative performance gains of up to 112% on the arena distribution, based on conservative estimates.

“We show that access to Chatbot Arena data yields substantial benefits; even limited additional data can result in relative performance gains of up to 112% on the arena distribution, based on our conservative estimates.”

Chatbot Arena has an undisclosed policy that allows a small group of preferred model providers to test many model variants in private before releasing only the best-performing checkpoint.

“In particular, we identify an undisclosed Chatbot Arena policy that allows a small group of preferred model providers to test many model variants in private before releasing only the best-performing checkpoint.”

In the simulation, testing just 10 variants yields a notable increase of approximately 100 points in the maximum score identified, compared to the baseline scenario with no private testing.

“In our simulation, testing just 10 variants yields a notable increase of approximately 100 points in the maximum score identified, compared to the baseline scenario with no private testing.”

A combined 83 open-weight models have only received an estimated 29.7% of the total data.

“In contrast, a combined 83 open-weight models have only received an estimated 29.7% of the total data.”

A combined 41 fully open-source models have only received an estimated 8.8% of the total data.

“In contrast, a combined 41 fully open-source models have only received an estimated 8.8% of the total data, collectively.”

Open-weight and fully open-source models receive significantly less data, in some cases receiving less than half the amount of data as proprietary developers.

“Open-weight and fully open-source models receive significantly less data, in some cases receiving less than half the amount of data as proprietary developers.”

Meta tested 27 private LLM variants in the lead-up to the Llama-4 release.

“At an extreme, we identify 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release.”

Substantial data asymmetries exist regarding the amount of feedback data given to proprietary vs. open-weight vs. open-source model providers.

“In addition, we find substantial data asymmetries are apparent regarding the amount of feedback data given to proprietary vs. open-weight vs. open-source model providers.”

Over an extended period, a handful of preferred providers have been granted disproportionate access to data and testing.

“This comprehensive analysis reveals that over an extended period, a handful of preferred providers have been granted disproportionate access to data and testing.”

Restricting private testing to a subset of providers can lead to scenarios where a weaker family of models (Family A) enabled with private testing can outperform a stronger family of models (Family B), which is restricted to a single submission.

“As we show in Figure 8, restricting private testing to a subset of providers can lead to scenarios where a weaker family of models (Family A) enabled with private testing can outperform a stronger family of models (Family B), which is restricted to a single submission.”

In a controlled experimental setting, increasing the arena training data proportion from 0% to 70% more than doubles the win-rates from 23.5% to 49.9% on ArenaHard.

“In a controlled experimental setting, we observe that increasing the arena training data proportion from (0% → 70%) more than doubles the win-rates from 23.5% to 49.9% on ArenaHard (Li et al., 2024c).”

• cites Li et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Certain model developers (most notably Meta, Google, Open AI and Amazon) have benefited from extensive private testing.

“We show that certain model developers (most notably Meta, Google, Open AI and Amazon) have benefited from extensive private testing.”

This allows providers to collect 100% of the test prompts submitted on the Arena.

“This allows providers to collect 100% of the test prompts submitted on the Arena.”

Google had 10 private models preceding the release of Gemma 3.

“We observed that Meta and Google had 27 and 10 private models, respectively, which preceded the releases of Llama 4 (Meta, 2025) and Gemma 3 (Google, 2025) models shortly thereafter.”

Undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired.

“We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired.”

Chatbot Arena has an unstated policy of allowing select providers to test many submissions in parallel.

“Chatbot Arena has an unstated policy of allowing select providers to test many submissions in parallel.”

Access to Chatbot Arena data yields substantial benefits.

“We show that access to Chatbot Arena data yields substantial benefits; even limited additional data can result in relative performance gains of up to 112% on the arena distribution, based on our conservative estimates.”

The number of silently deprecated models (205) is significantly higher than the 47 models officially listed as deprecated in FastChat.

“This is a significantly higher number than the 47 models officially listed as deprecated as part of Chatbot Arena's backend codebase, FastChat 5 .”

As the number of privately tested models increases, the expected maximum Arena Score discovered rises accordingly.

“As the number of privately tested models increases, the expected maximum Arena Score discovered rises accordingly.”

Among models that are silently deprecated, 66% are open-weight or fully open-source.

“Among the models that are silently deprecated, 66% are open-weight or fully open-source.”

Google received an estimated 19.2% of all data on the arena.

“Providers like Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the arena, respectively.”

• Also stated in: § -Charles A. E. Goodhart · hedged

  “Google and OpenAI have received an estimated 19.2% and 20.4% of all test prompts on the arena, respectively.”

Sampling rates vary significantly across providers.

“As shown in Figure 5, sampling rates vary significantly across providers.”

If private models tested by Meta on the vision leaderboard are considered, there are an additional 16 variants, bringing the total to 43.

“Indeed, if we also consider the number of private models tested by Meta on the vision leaderboard, we observe an additional 16 variants, bringing the total to 43.”

The paper observes large discrepancies across providers, with substantially higher sampling rates for OpenAI, Google, xAI, and Meta compared to others.

“We observe large discrepancies across providers, with substantially higher sampling rates for OpenAI, Google, xAI, and Meta compared to others.”

Identical checkpoints of Aya-Vision-8B submitted to Chatbot Arena in March 2025 obtained different final Arena scores (1052 vs. 1069).

“In Figure 9, we observe notable differences in final Arena scores between identical checkpoints (1052 vs. 1069).”

The scraped-random-sample was the only way to identify private variants being tested by various providers.

“Our scraped-random-sample was the only way to identify private variants being tested by various providers.”

Sampling rates disproportionately undersample open-weight and open-source models.

“and also disproportionately undersample open-weight and open-source models”

• AssumptionRests on an assumption: The scraped-random-sample is representative of battles over time on Chatbot Arena.
• AssumptionRests on an assumption: The classification of models into proprietary, open-weight, and open-source categories is valid.

A subset of the data sources used in this study have undergone pre-processing by Chatbot Arena.

“A subset of the data sources utilized for this study have undergone pre-processing by Chatbot Arena”

The number of multilingual prompts on the Arena has grown by 20% over 1.5 years, from 23.9% in April 2023 to 43.5% in January 2025.

“Overall, the number of multilingual prompts on the Arena has grown by 20% over 1.5 years, from 23.9% in April 2023 to 43.5% in January 2025.”

Private battles are removed from the datasets released by Chatbot Arena.

“Through this work, we also establish that private battles are removed from the datasets released by Chatbot Arena.”

205 models on Chatbot Arena have been silently deprecated by reducing their active sampling rates to near zero.

“In addition, as discussed in Section 4.1, 205 models on Chatbot Arena have been silently deprecated by reducing their active sampling rates to near zero (see Figure 17).”

Private testing volume varies widely across providers, creating unfair advantages.

“As illustrated in Figure 6, private testing volume varies widely across providers, creating unfair advantages.”

Prompt samples available to each provider may not be mutually exclusive, as each battle involves two models, allowing the same prompt to be sent to two different providers.

“We note that the prompt samples available to each provider may not be mutually exclusive, as each battle on the Arena involves two models (from the same or different model providers), allowing the same prompt to be sent to two different providers.”

High levels of prompt duplication exist on Chatbot Arena.

“In parallel, we observe high levels of prompt duplication.”

The scraped data snapshot covers January-March 2025.

“However, it covers a limited time period from January-March, 2025.”

A non-trivial portion of prompts in one month are either exact duplicates or near-duplicates of prompts from previous months.

“(2) a non-trivial portion of prompts in one month are either exact duplicates or near-duplicates of prompts from previous months.”

Meta is the provider with the highest number of private variants in the analysis.

“we find them to be the provider with the highest number of private variants in our analysis.”

• AssumptionRests on an assumption: Model self-identification reliably attributes private models to their respective providers.

The rankings derived from the dense comparison graph align closely with the models' rankings according to their true skill ratings.

“We observe that the rankings derived from the dense comparison graph align closely with the models' rankings according to their true skill ratings.”

Currently, providers who have engaged in private testing are allowed to retract submissions and only submit the best variant to the public leaderboard.

“Currently, providers who have engaged in private testing are allowed to retract submissions and only submit the best variant to the public leaderboard.”

The Chatbot Arena-FastChat backend codebase publicly lists 47 models explicitly marked as deprecated on GitHub.

“the backend codebase of Chatbot Arena-FastChatpublicly lists deprecated models on GitHub 25 , where 47 models are explicitly marked as deprecated.”

As the amount of arena-mix data increases, the models improve in their evaluation on the ArenaHard prompts.

“From Figure 10, we observe that as the amount of arena-mix data increases, the models improve in their evaluation on the ArenaHard prompts.”

On Chatbot Arena, notable shifts have been observed in the distribution of prompts evaluated over longer periods, with a consistent increase in the proportion of prompts from more complex categories, such as mathematics, coding, and multi-turn conversations.

“On Chatbot Arena, notable shifts have been observed in the distribution of prompts evaluated over longer periods, with a consistent increase in the proportion of prompts from more complex categories, such as mathematics, coding, and multi-turn conversations 19 .”

The characteristics of prompts on Chatbot Arena shift notably over time.

“(1) The characteristics of prompts do shift notably over time”

The training experiments are based on only a fraction of the data believed available to some proprietary model providers.

“as it is based on training with only a fraction of the data believed to be available to some proprietary model providers.”

Four times the number of explicitly marked deprecated models have been silently deprecated without warning.

“However, four times that number have been silently deprecated without warning.”

A sparse or disconnected comparison graph results in an inaccurate estimation of the models' skills.

“In contrast, a sparse or disconnected comparison graph results in an inaccurate estimation of the models' skills.”

Many models are deprecated over time, and their scores stop getting updated.

“However, as shown in Figure 13, many models are deprecated over time, and their scores stop getting updated.”

Rankings produced by the BT model are highly sensitive to model deprecation, particularly when the prompt distribution changes over time.

“As illustrated in Figure 14, our simulation shows that rankings produced by the BT model are highly sensitive to model deprecation, particularly when the prompt distribution changes over time.”

The proportion of languages outside of English has varied over time.

“For example, in Figure 11, we observe that the proportion of languages outside of English has varied over time.”

7.3% of prompts from December 2024 appear again in the exact form in January 2025.

“For instance, 7.3% of prompts from December 2024 appear again in the exact form in January 2025.”

Proprietary API data for the models tested on Arena reveals only a subset of battles.

“Although we also have proprietary API data for the models we test on the Arena, it only reveals a subset of battles.”

Sampling rates on Chatbot Arena vary greatly by provider.

“As shown in Figure 5, the sampling rates vary greatly by providers”

• AssumptionRests on an assumption: The scraped-random-sample is representative of battles over time on Chatbot Arena.

Silent deprecations have a much greater impact on open-weight and open-source models.

“silent deprecations have a much greater impact on openweight and open-source models.”

Cross-month duplication rates are substantial.

“the substantial cross-month duplication rates”

Using de-anonymization, the paper identified a total of 64 private models corresponding to 10 providers.

“Using this approach, we identified a total of 64 private models corresponding to 10 providers.”

• AssumptionRests on an assumption: Model self-identification reliably attributes private models to their respective providers.

Of the 243 public models, 205 participated in an average of 10 or fewer battles during March 3rd to April 23rd, 2025, based on leaderboard-stats.

“Of the 243 public models, 205 participated in an average of 10 or fewer battles during this period, based on leaderboard-stats.”

Large differences in final scores (1097 vs. 1059) are observed for two different model variants of Aya-Vision-32B.

“We observe large differences in final scores (1097 vs. 1059) for the two different model variants.”

Arena Scores for Aya-Vision-8B yield different Arena scores (1069 vs. 1052) for identical checkpoints.

“Arena Scores for Aya-Vision-8B yield different Arena scores (1069 vs. 1052).”

The actual number of deprecated models far exceeds the official count provided by Chatbot Arena.

“the actual number of deprecated models far exceeds the official count provided by Chatbot Arena.”

There are large discrepancies across providers in maximum observed sampling rates, with substantially higher sampling rates for OpenAI, Google, xAI, and Meta compared to others.

“We observe large discrepancies across providers, with substantially higher sampling rates for OpenAI, Google, xAI, and Meta compared to others.”

• AssumptionRests on an assumption: A model's sampling rate determines the amount of data its provider receives.
• AssumptionRests on an assumption: The scraped random sample is representative of all Chatbot Arena battles during the period studied.

The majority of the historical-battles dataset does not contain prompts.

“Majority of historical-battles dataset does not contain prompts as shown in Table 1.”

Use of Chatbot Arena dataset significantly improves win-rates on ArenaHard.

“Use of Chatbot Arena dataset significantly improves win-rates on ArenaHard.”

Providers like Google, OpenAI, Anthropic, Amazon, Meta, and DeepSeek AI have the most actively sampled public models, ranging from 3 to 10.

“providers like Google, OpenAI, Anthropic, Amazon, Meta, and DeepSeek AI have the most actively sampled public models, ranging from 3 to 10.”

• AssumptionRests on an assumption: A model's sampling rate determines the amount of data its provider receives.
• AssumptionRests on an assumption: The scraped random sample is representative of all Chatbot Arena battles during the period studied.

Open-weight and fully open-source models receive significantly less data, in some cases less than half the amount of data as proprietary developers.

“Open-weight and fully open-source models receive significantly less data, in some cases receiving less than half the amount of data as proprietary developers.”

Proprietary models consistently received the largest share of data, ranging from 54.3% to 70.1%, from January 2024 to March 2025.

“Proprietary models consistently received the largest share of data-ranging from 54.3% to 70.1%.”

From Table 9, all models achieve very similar scores on MMLU.

“From Table 9, we observe that all models achieve very similar scores.”

Deprecations disproportionately affect open-weight and open-source models compared to proprietary ones.

“deprecations disproportionately affect open-weight and open-source models compared to proprietary ones.”

Meta had an additional 16 private models active on the Vision leaderboard along with its 27 models on the Overall leaderboard, bringing its total count to 43.

“We find that Meta had an additional 16 private models active on the Vision leaderboard along with its 27 models on the Overall leaderboard, bringing its total count to 43.”

• AssumptionRests on an assumption: Model self-identification reliably attributes private models to their respective providers.

User queries in Chatbot Arena are often highly similar or duplicated.

“user queries in Chatbot Arena are often highly similar or duplicated.”

Proprietary providers collectively access a considerably greater volume of data compared to the broader research community, which receives only a fraction.

“proprietary providers collectively access a considerably greater volume of data compared to the broader research community, which receives only a fraction.”

• AssumptionRests on an assumption: The number of model API calls is a valid measure of the volume of data available to model providers.

Each of the 3 million votes resulted in twice the number of model API calls, i.e., 6 million, because each battle features two models.

“Each of these 3M votes resulted in twice the number of model API calls i.e. 6M since each battle features two models.”

Overall, open-weight and fully open-source models are more likely to become deprecated or inactive compared to proprietary models.

“Overall, open-weight and fully open-source models are more likely to become deprecated or inactive compared to proprietary models.”

According to exact string match and text embedding similarity, within-month duplication rates are generally high.

“according to two metrics (exact string match and text embedding similarity) within-month duplication rates are generally high”

The scraped-random-sample contains 5,864 battles collected by crawling Chatbot Arena between January 2025 and March 2025.

“we collected 5864 battles by crawling Chatbot Arena between January 2025 -March 2025”

• AssumptionRests on an assumption: The scraped-random-sample is representative of battles over time on Chatbot Arena.

Certain practices such as multiple submissions during the anonymous testing period and best-of-N submissions tend to favor large, proprietary players with disproportionate access to data.

“we report in this paper that certain practices-such as multiple submissions during the anonymous testing period and best-of-N submissions-tend to favor large, proprietary players with disproportionate access to data.”

The leaderboard-stats dataset contains 14.3K records corresponding to 243 unique models evaluated on Chatbot Arena.

“we obtained 14.3K records corresponding to 243 unique models evaluated on Chatbot Arena.”

These dynamics result in overfitting to Arena-specific dynamics rather than general model quality.

“Together, these dynamics result in overfitting to Arena-specific dynamics rather than general model quality.”

Willful engagement from a handful of providers along with preferential policies from Chatbot Arena towards the same small group have amplified the potential for gamification, in place of innovative progress.

“In this work, we show that willful engagement from a handful of providers along with preferential policies from Chatbot Arena towards the same small group have amplified the potential for gamification, in place of innovative progress.”

Win-rates are measured on ArenaHard, which has a high correlation of 98.6% to Chatbot Arena.

“The win-rates are measured on ArenaHard (Li et al., 2024c), which has a high correlation of 98.6% to Chatbot Arena.”

• cites Li et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Chatbot Arena has systematic issues that have resulted in a distorted playing field.

“Yet, in this work we identify systematic issues that have resulted in a distorted playing field.”

Even with identical variants, it is very feasible to obtain a biased positive advantage.

“This suggests that even with identical variants, it is very feasible to obtain a biased positive advantage.”

The ability to select the best-scoring variant from N models enables systematic gaming of the Arena rating.

“We show with real-world experiments and simulations that the ability to select the best-scoring variant from N models enables systematic gaming of the Arena rating.”

The estimate that training on Chatbot Arena data improves model ranking significantly is conservative because a subset of providers have disproportionate access to private API data, which if used can potentially yield even greater performance gains.

“We believe this is a conservative estimate, as a subset of providers have disproportionate access to private API data, which, if used, can potentially yield even greater performance gains.”

Disparities in access to Chatbot Arena data stem from differences in the number of private variants tested, sampling rate, and silent deprecation.

“These disparities stem from difference in the number of private variants tested, sampling rate, and silent deprecation.”

The ability of providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results.

“We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results.”

These results represent a conservative estimate of gains given the limited access to arena-style data.

“These results represent a conservative estimate of gains given the limited access to arena-style data.”

Developers who deploy a model and enable Chatbot Arena testing via an API have a default advantage.

“Developers who deploy a model and enable Chatbot Arena testing via an API have a default advantage.”

These policies appear to severely disadvantage open-weight and open-source providers.

“All these policies appear to severely disadvantage open-weight and open-source providers.”

Deprecation can violate key assumptions of the Bradley-Terry model, thereby reducing the reliability of the leaderboard rankings.

“We show that deprecation can violate key assumptions of the Bradley-Terry model (Bradley & Terry, 1952), which underpins Arena scoring, thereby reducing the reliability of the leaderboard rankings.”

• cites Bradley & Terry, 1952 sourceNot checked — no full text of the source

Access to Chatbot Arena data has an outsized impact on performance.

“Access to Chatbot Arena data has an outsized impact on performance.”

Chatbot Arena may be forced to deprecate a model when a provider no longer supports it via its API.

“Chatbot Arena may be forced to deprecate a model when a provider no longer supports it via its API.”

The active sampling rule selects model pairs based on the expected reduction in the confidence interval of the win-rate estimate.

“This rule selects model pairs based on the expected reduction in the confidence interval of the win-rate estimate”

• cites Chiang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Proprietary models may be trained on 5 to 10 times more data than the paper uses.

“This disparity suggests that proprietary models may be trained on 5 to 10 times more data than we use”

Having access to a large sample of last month's data will enable a developer to perform significantly better on next month's test set.

“Together, these suggest that having access to a large sample of last month's data will enable a developer to perform significantly better on next month's test set.”

This issue is particularly problematic in real-world settings where user prompt distributions shift over time.

“This issue is particularly problematic in real-world settings where user prompt distributions shift over time.”

For any possible partition of models, there must be at least one win going in each direction across the partition.

“For any possible partition of models, there must be at least one win going in each direction across the partition.”

Without access to original and comprehensive raw data, it is hard to investigate patterns related to adversarial voting.

“Without access to original and comprehensive raw data, it is hard to investigate patterns related to adversarial voting”

Transitivity allows the BT model to infer missing outcomes if two models share common opponents, their relative ranking can be deduced even without a direct comparison.

“Transitivity allows the BT model to infer missing outcomes-if two models share common opponents, their relative ranking can be deduced even without a direct comparison (Bradley & Terry, 1952).”

• cites Bradley & Terry, 1952 sourceNot checked — no full text of the source

This distributional shift contrasts with the static environments typically assumed in Elo and Bradley-Terry systems, such as chess, where rules and game format remain fixed.

“This distributional shift contrasts with the static environments typically assumed in Elo and Bradley-Terry systems, such as chess, where the rules and game format remain fixed, ensuring a consistent set of evaluation conditions.”

For a global technology provider, real-world commercial applications may differ significantly from this distribution.

“For a global technology provider, real-world commercial applications may differ significantly from this distribution.”

The disparity in training data potentially increases the risk of overfitting to patterns not present in the smaller subset.

“potentially increasing the risk of overfitting to patterns not present in our smaller subset.”

This indicates increased language diversity in submitted prompts.

“This indicates increased language diversity in submitted prompts.”

This ensures that no subset of models is entirely isolated in the win/loss structure.

“This ensures that no subset of models is entirely isolated in the win/loss structure.”

The paper's estimate of overfitting is likely conservative.

“Our estimate of overfitting is likely conservative”

Deprecations may result in fragmented graphs.

“In Section 5.2, we show that deprecations may result in these fragmented graphs.”

The sampling rate disparities result in unstable Arena scores.

“resulting in unstable Arena scores (Section 5)”

When a model's battle history is limited to outdated conditions, its comparisons no longer accurately reflect performance in the new context.

“We show in Section 5.1 that when a model's battle history is limited to outdated conditions, its comparisons no longer accurately reflect performance in the new context.”

Some test prompts are very similar to or the same as prompts seen previously.

“This means that some test prompts are very similar to (or even the same as) prompts seen previously.”

Model deprecations under a changing task distribution lead to unreliable rankings.

“Model deprecations under a changing task distribution lead to unreliable rankings.”

Some misattributions may occur due to limited data and inconsistent or ambiguous model responses.

“Due to limited data and the potential for models to respond inconsistently or ambiguously, some misattributions may occur.”

• AssumptionRests on an assumption: Model self-identification reliably attributes private models to their respective providers.

There is sufficient signal to suggest that Chatbot Arena distribution differs substantially from other evaluation settings.

“There is sufficient signal to suggest this is the case.”

The distribution of Chatbot Arena is unique since long-term shifts occur in categories and use cases.

“As we have shown in Section 4.1, the distribution of Chatbot Arena is unique since long-term shifts occur in categories and use cases.”

At present, free human feedback on Chatbot Arena primarily benefits proprietary models.

“where at present free human feedback is primarily benefiting proprietary models”

Model deprecation may result in disconnected subgraphs, which violates BT reliability.

“Model deprecation may result in disconnected subgraphs, which violates BT reliability.”

Reliable rankings under the BT model require a connected comparison graph.

“These results highlight that reliable rankings under the BT model require a connected comparison graph.”

The absence of updated comparisons causes a skew in the rankings of remaining models.

“While its matchups from Phase 1 still influence BT scores, the absence of updated comparisons causes a skew in the rankings of remaining models.”

It is often not necessary to train explicitly on the data to gain from data access.

“We also note that often it is not necessary to train explicitly on the data to gain from data access.”

If all models were continuously sampled across all points in time, the BT model would likely remain robust because every model would be evaluated on the evolving distribution of tasks.

“If all models were continuously sampled across all points in time, the BT model would likely remain robust because every model would be evaluated on the evolving distribution of tasks.”

When models are no longer sampled under current task distributions, historical pairwise comparisons cease to represent present-day performance.

“When models are no longer sampled under current task distributions, historical pairwise comparisons cease to represent present-day performance.”

A win-rate of 50% indicates that Llama 3.1 8B Instruct and 70_arena are on par on the ArenaHard test set.

“A win-rate of 50% indicates that the compared models (Llama 3.1 8B Instruct and 70_arena) are on par on the ArenaHard test set.”

When models are deprecated, future models that enter the Arena will not have direct comparisons with them.

“When models are deprecated, future models that enter the Arena will not have direct comparisons with them.”

Comparing Scenarios I and II reveals a clear divergence, with the scenario involving deprecation yielding unreliable rankings that no longer reflect true relative performance under the evolving task distribution.

“Comparing Scenarios I and II in Figure 14 reveals a clear divergence, with the scenario involving deprecation yielding unreliable rankings that no longer reflect true relative performance under the evolving task distribution.”

The study attributes private models to providers using model self-identification as a proxy.

“Since the identity of anonymous models is not publicly disclosed, we use model self-identification as a proxy-prompting each model directly and observing how it responds.”

• AssumptionRests on an assumption: Model self-identification reliably attributes private models to their respective providers.

When models are deprecated unevenly or sampling strategies fail to ensure robust overlap in comparisons, the resulting history matrix can become fragmented.

“When models are deprecated unevenly or when sampling strategies fail to ensure robust overlap in comparisons, the resulting history matrix can become fragmented.”

This lack of precision makes it challenging to apply the rule consistently or verify retirement decisions.

“This lack of precision makes it challenging to apply the rule consistently or verify retirement decisions.”

To ensure a unique and finite estimation, the comparison graph must be connected.

“Therefore, to ensure a unique and finite estimation, the comparison graph must be connected.”

The counts for providers with fewer model launches during January-March 2025 may be underestimated.

“We believe we might be underestimating the counts for providers having fewer model launches during this period.”

The gains are striking in part because the variants were not heavily optimized (3 variants trained, no hyperparameter optimization or tuning of training steps).

“The gains we observe are striking in part because we do not heavily optimize these variants (we trained 3 variants in total and did no hyperparameter optimization or tuning of training steps).”

The sampling rate disparities create large asymmetries in data access over time.

“creating large asymmetries in data access over time”

Model self-identification provides a reasonable but inherently approximate signal for provider attribution.

“While this method provides a reasonable signal, it is inherently approximate.”

• AssumptionRests on an assumption: Model self-identification reliably attributes private models to their respective providers.

Chatbot Arena's pre-processing often involves de-duplication and removal of battles corresponding to suspicious voting patterns.

“which as stated by Chatbot Arena often involves de-duplication, removal of battles corresponding to suspicious voting patterns, etc (Chiang et al., 2024).”

• cites Chiang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

If deprecated, a multilingual model's BT ranking will likely understate its true performance.

“However, if deprecated, its BT ranking will likely understate its true performance.”

Simulating human preferences using LM-as-a-judge allows measuring a reasonable estimate of gains in a controlled setting.

“To measure improvements, we simulate human preferences using LMas-a-judge, which allows us to measure a reasonable estimate of the gains in a controlled setting.”

A model tuned for multilingual prompts may improve ranking as non-English tasks become more common.

“For instance, a model tuned for multilingual prompts may improve ranking as non-English tasks become more common.”

The Chatbot Arena comparison matrix can potentially become disconnected because of the extremely high levels of model removals over time.

“The Chatbot Arena comparison matrix can potentially become disconnected because of the extremely high levels of model removals over time (as discussed in Figure 13).”

The Chatbot Arena authors introduced an active sampling rule designed to enhance the efficiency and statistical robustness of the leaderboard's evaluation process.

“the Chatbot Arena authors introduce an active sampling rule designed to enhance the efficiency and statistical robustness of the leaderboard's evaluation process.”

• cites Chiang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

As the number of private variants being tested on the Arena increases, the sampling of public models can be significantly reduced.

“as the number of private variants (see Figure 6) being tested on the Arena increases, the sampling of public models can be significantly reduced.”

As a result, model development may be deliberately optimized for performance on Chatbot Arena.

“As a result, model development may be deliberately optimized for performance on Chatbot Arena.”

The simulation shows how providers can gain leaderboard advantage by evaluating several variants privately and publishing only the top-scoring one.

“This simulation shows how providers can gain leaderboard advantage by evaluating several variants privately and publishing only the top-scoring one.”

Training on data from Arena Battles provides little to no effect on a non-arena style benchmark.

“provides little to no effect on a non-arena style benchmark.”

Chatbot Arena assigns higher sampling weights to top-10 models.

“Chatbot Arena assigns higher sampling weights to top-10 models”

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

This result formalizes the selection bias arising when one reports the best out of N noisy skill estimates: statistical fluctuations ensure that the maximum overestimates the expected performance of a typical sample.

“This result formalizes the selection bias arising when one reports the best out of N noisy skill estimates: statistical fluctuations ensure that the maximum overestimates the expected performance of a typical sample.”

When confidence intervals of two models overlap, it becomes difficult to determine which one is truly better.

“When the confidence intervals of two models overlap, it becomes difficult to determine which one is truly better.”

Training on data from Arena Battles helps boost scores specific to the Arena evaluation.

“data from Arena Battles helps boost scores specific to the Arena evaluation”

This selection bias is especially relevant in leaderboard scenarios where multiple submissions are made and only the top-performing one is reported.

“This is especially relevant in leaderboard scenarios where multiple submissions are made and only the top-performing one is reported.”

The lack of standardization in benchmarks complicates meaningful comparisons due to inconsistent metrics and task definitions.

“The lack of standardization in benchmarks complicates meaningful comparisons due to inconsistent metrics and task definitions.”

Allowing retraction of scores allows providers to skew Arena scores upwards.

“Allowing retraction of scores allows providers to skew Arena scores upwards.”

ArenaHard has a high correlation of 98.6% to Chatbot Arena.

“which has a high correlation of 98.6% to Chatbot Arena.”

• cites Li et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The substantial cross-month duplication rates suggest recurring patterns or frequently asked questions among Chatbot Arena users.

“the substantial cross-month duplication rates suggest recurring patterns or frequently asked questions among Chatbot Arena users”