Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models

Existing accuracy matching frameworks fail to confirm that the LLM is uncontaminated before manual contamination, so their claimed 'clean' performance may be inaccurate, introducing noise into their conclusions.

“In contrast, existing accuracy matching frameworks (Zhu et al., 2023b; 2024b; Ying et al., 2024) fail to confirm that the LLM is uncontaminated before manual contamination. As a result, their claimed 'clean' performance may be inaccurate, introducing noise into their conclusions.”

Previous assessment methods, such as accuracy drop and accuracy matching, focus solely on aggregate accuracy, often leading to incomplete or misleading conclusions.

“Previous assessment methods, such as accuracy drop and accuracy matching, focus solely on aggregate accuracy, often leading to incomplete or misleading conclusions.”

Some methods modify evaluation samples while preserving their semantics; others generate new samples with altered semantics based on original questions, using advanced LLMs.

“Some methods modify evaluation samples while preserving their semantics (Zhu et al., 2024b;a; 2023b; Li et al., 2024c; Wang et al., 2021; Xia et al., 2024; Haimes et al., 2024; Zheng et al., 2024). Others generate new samples with altered semantics based on original questions, using advanced LLMs (Ying et al., 2024).”

Researchers have proposed various mitigation strategies to update existing benchmarks, including modifying original questions or generating new ones based on them.

“To address this, researchers have proposed various mitigation strategies to update existing benchmarks, including modifying original questions or generating new ones based on them.”