A framework for few-shot language model evaluation
Open accessThe DOI matches its OpenAlex record.
Abstract
v0.4.13 Release Notes A fix-focused release. The main fixes are for few-shot leakage, a multiple-choice filter bug, and group stderr, alongside two new ONNX backends and eight new benchmark suites. Also updated most configs for datasets>=4, which accounts for much of the diff by volume. Highlights Bug Fixes Fixes that may shift previously reported numbers: Eval documents leaked into few-shot prompts. The sampler could draw the document under test into its own demonstrations by @SiavashShams in #3978, and gen_prefix was resolved against the eval doc rather than the few-shot doc — splicing the evaluated question into every shot for RULER niah_single_1, humaneval_instruct, and humaneval_64_instruct by @adityasingh2400 in #3979 MultiChoiceRegexFilter prefix-shadowing. Regex alternation is leftmost-wins, so a choice that prefixed a longer choice ("Guilty" vs "Guilty of Romance") matched inside it and scored correct answers as wrong — visible on BBH movie_recommendation by @iamsharduld in #3884 Group stderr with weight_by_size: false. Groups reported the size-weighted pooled stderr even when the point estimate was an unweighted mean, giving error bars up to ~3x too narrow. Only unequal-sized subtasks change by @iamsharduld in #3882 minerva_math answer normalization. sqrt shorthand no longer corrupts indexed roots (#4037), the thousands-separator strip no longer fuses digit tuples (0,1 → 01, #4039), an answer identical to the gold now scores correct (#4034), and the few-shot prompt LaTeX is corrected (#4045) by @feiiiiii5 and @nata2627. putnam_axiom shares these helpers and picks up the same fixes. New Model Backends onnxruntime — raw onnxruntime backend for Model Builder ONNX exports by @amd-sourjya in #3984 onnxruntime-genai — cross-platform ONNX Runtime GenAI backend, with winml refactored on top of it by @thiagocrepaldi in #3960 Megatron-LM v0.18 compatibility by @oberpierre in #3991 Install with pip install lm_eval[onnxruntime] or lm_eval[onnxruntime-genai]. For non-CPU execution providers install the matching wheel instead — onnxruntime-gpu / onnxruntime-rocm, or onnxruntime-genai-cuda / onnxruntime-genai-directml — these are mutually exclusive. New Tasks LongProc — long-context procedural reasoning, 6 task types across 16 configs by @xiye17 in #3544 Uncheatable Eval — contamination-resistant evaluation: rolling log-likelihood over recently-published documents from Wikipedia, GitHub, BBC News, arXiv, bioRxiv, and AO3, as 15 category tasks plus a size-weighted group by @ziqing-huang in #3442 TyDiQA Gold Passage — 9-language multilingual QA by @bongho in #4044 LegalBench — new suite, 19 tasks: the HELM-lite LegalBench subset (5 tasks, #3860) and the Contract NLI suite (14 NDA entailment tasks, #3954) by @bongho IndicParam — MCQ benchmark covering 12 low-resource Indic languages by @hjoshifonteva in #3826 IndicXNLI Gujarati (indicxnli_gu) by @bhaumik611 in #4056 GreekMMLU — official native-sourced configuration by @mersinkonomi in #3581 Physics GRE — InflectionAI multiple-choice physics benchmark by @bongho in #3853 Putnam Axiom — competition-level mathematical reasoning by @baberabb in #3998 Portuguese Bench — ASSIN2 RTE and STS by @bongho in #3812 Catalanbench — cieaCOVA (#4043) and Terretaqa (#3784) by @baberabb and @ivanmartinezmurillo MLQA — a tag to run all variants at once by @ivanbaldo in #2977 Task Changes Correctness & Prompts global_piqa restructured into parallel/non-parallel × cloze/generation variants; the global_piqa_completions and global_piqa_prompted groups are now global_piqa_cloze and global_piqa_generation by @baberabb in #3816 IrokoBench — repaired afrimmlu / afrimgsm / afrixnli task registration by @discobot in #3841, fixed the broken afrisenti / mafand prompt_2 group references by @DaoyuanLi2816 in #3847, and switched afrixnli prompt_1 doc_to_text to Jinja braces by @Solaris-star in #3944 KorMedMCQA — answer-choice extraction now follows the paper's Appendix B by @discobot in #3842 JSONSchema Bench — per-sample validation timeout so a pathological schema no longer hangs the whole eval by @hancheolcho in #3923 RULER — every task crashed with TypeError: unhashable type: 'dict' before inference when the tokenizer arrived as anything but a plain string (e.g. under local-chat-completions); the name is now resolved before the cached lookup by @nata2627 in #4048 NIAH — no longer loops forever when max_seq_lengths < 4096 by @vnayakde in #3372 LongBench — code_sim_score skips a blank leading line before extraction by @cameronshinn in #3921 FDA / SWDE / SQuAD_completion — whitespace stripped from input and target by @EphraiemSarabamoun in #3795 med_prescriptions — requires both keys before combining complaints and diagnosis by @Anai-Guo in #4041 HumanEval — humaneval_random_span_infilling_light was registered under the wrong task name, causing a KeyError by @jaydeepborkar in #3768 INCLUDE — stray space removed from North Macedonian task identifiers by @DaoyuanLi2816 in #3848 DarijaBench — trasnlation typo corrected in translation task names by @DaoyuanLi2816 in #3824 Unitxt — dataset_kwargs is now passed through (#3230) and init sets the task name (#3225) by @mprahl format_span filter — normalizes labels only, leaving entity text untouched by @k-dickinson in #3887 ContextSampler — no longer crashes when the few-shot pool contains duplicate eval-doc rows by @Kymi808 in #3790 fewshot_config.split — a nested fewshot_config.split now takes precedence over the inherited top-level fewshot_split, as documented by @chuenchen309 in #3937 Model Backends & APIs Chat completions now support think_end_token by @yaodong-shen in #3959 auto:N batch size is parsed correctly in API models by @AbdullahRasheed45 in #3970, and batch_size="auto" no longer raises on the neuronx backend by @AbdullahRasheed45 in #3971 Synchronous API requests honor the configured timeout by @SiavashShams in #3995 Anthropic stop sequences are kept non-empty by @he-yufeng in #3822 HF: max_length is detected from a nested text_config (Gemma3 multimodal) by @OrionArchitekton in #3916, max_cpu_memory is passed through to accelerate's max_memory by @Anai-Guo in #4016, and tokenizers that decode to an empty string fall back to tokenizer.eos_token by @ganeshr10 in #3657 vLLM warns when it ignores the device argument by @Apeironics in #3803; sglang argument passing fixed by @wm901115nwpu in #3817 Core, CLI & Config --metadata accepts key=value pairs in addition to JSON by @baberabb in #4054 --use_cache keys are hashable for multimodal image/byte requests by @feiiiiii5 in #4040 Cache parent directories are created when missing by @sdivyanshu90 in #4047; delete_cache is a no-op when the directory is absent by @feiiiiii5 in #4035 A local directory sharing a task's name no longer shadows the registered task by @nloughl in #3670 Function resolving works for custom tasks by @SkyR0ver in #3992; string dictionary arguments parse correctly by @tandede in #4020 A scalar seed from a config file is normalized the same way the CLI does by @winklemad in #4003 Sample counts no longer depend on metric ordering by @arthi-arumugam-git in #4068 scripts/requests_caching.py entrypoint repaired — bad kwargs and a helper defined after its use by @Anai-Guo in #4059 chrF++ aggregation and metric (word_order=2) by @KrishVenky in #3780; TER metric direction corrected and the chrF docstring fixed by @borgr in #3993 All numpy scalar types serialize to JSON, not just int64/int32, by @iamsharduld in #3885 Migration Notes Few-shot prompts changed for tasks using a document-specific gen_prefix (RULER niah_single_1, humaneval_instruct, humaneval_64_instruct) and anywhere the sampler previously drew the eval document into its own shots. Prior numbers on those tasks may not be comparable. minerva_math and leaderboard_math_* now diverge. lm_eval/tasks/leaderboard/math/utils.py is deliberately frozen to reproduce Open LLM Leaderboard v2 scoring; the normalization fixes landed in minerva_math only. global_piqa task names changed — see Task Changes above. If you pinned datasets<4 to keep script-based tasks working, you can unpin. Out-of-tree task YAMLs pointing at script-based datasets still need the same treatment. winml now sits on top of onnxruntime-genai and pulls in that extra. <details> <summary> What's Changed </summary> feat: 0.4.13.dev0 by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3764 update global piqa by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3816 fix for keyerror while running humaneval_infilling by @jaydeepborkar in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768 fix: strip whitespace from input/target for FDA, SWDE, and SQuAD_completion tasks by @EphraiemSarabamoun in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795 Fix "trasnlation" typo in DarijaBench translation task names by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824 fix(ruler): replace defunct hotpot dataset host with pinned HF mirror by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3806 fix: keep Anthropic stop sequences nonempty by @he-yufeng in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822 fix(vllm): warn when device argument is ignored by @Apeironics in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803 fix sglang args by @wm901115nwpu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817 feat(legalbench): add HELM-lite LegalBench subset (5 tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860 Fix space in North Macedonian task identifiers (INCLUDE suite) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3848 fix(kormedmcqa): extract answer choice per paper Appendix B by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842 Fix dataset paths for xnli, xcopa, paws-x, and xquad by @Chessing234 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3870 Add IndicParam: MCQ benchmark for 12 low-resource Indic languages by @hjoshifonteva in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826 feat(portuguese_bench): add ASSIN2 RTE and STS tasks by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3812 fix(prost): load PROST without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977 fix(spanish_bench): load wnli_es without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3982 fix(tests): pass dataset_kwargs in test_download by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3980 fix(tasks): use namespaced dataset path for wsc273 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3946 fix(tasks): use namespaced dataset paths for webqs, medmcqa, and wmt16 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3942 fix(arithmetic): load arithmetic tasks without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3976 fix(mmlusr): restore dataset loading for all 171 MMLU-SR tasks by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3972 fix(scrolls): load SCROLLS without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3975 fix(tasks): use script-less parquet mirrors for mathqa, siqa, and moral_stories by @shubhangithub in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943 fix(tmmluplus): use ikala/tmmluplus so the task loads on datasets>=4 by @gowtham-sai-yadav in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950 Fix broken afrobench group task references (afrisenti/mafand prompt_2) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3847 fix(longbench): skip blank leading line in code_sim_score extraction by @cameronshinn in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921 fix(afrixnli): use Jinja braces in prompt_1 doc_to_text by @Solaris-star in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944 fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs by @hancheolcho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923 Fix fewshot_config.split precedence in TaskConfig by @chuenchen309 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937 Exclude eval docs from first-n few-shot samples by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978 fix: prevent ValueError when batch_size="auto" is passed to neuronx model by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971 fix: prevent ValueError when batch_size="auto:N" is passed to API models by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3970 fix: resolve fewshot gen_prefix against the fewshot doc, not the eval doc by @adityasingh2400 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979 Putnam Axiom by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3998 feat(legalbench): add Contract NLI suite (14 NDA entailment tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3954 Fix megatron_lm backend against Megatron-LM core_v0.18+: argument parsing moved out of initialize_megatron() by @oberpierre in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991 chore(ci): update pre-commit hooks and workflow packages by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4023 fix(hf): pass max_cpu_memory through to accelerate's max_memory by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4016 chore(megatron): run linter by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4024 fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal) by @OrionArchitekton in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916 Honor configured timeout for synchronous API requests by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3995 fix(api): support think_end_token for chat completions by @yaodong-shen in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959 feat(models): add cross-platform onnxruntime-genai backend + refactor winml by @thiagocrepaldi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960 Enable function resolving for custom tasks by @SkyR0ver in https://github.com/EleutherAI/lm-evaluation-harness/pull/3992 fix(config): parse dictionary strings from YAML configs by @tandede in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020 fix: normalize scalar seed from a config file the way the CLI does by @winklemad in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003 fix(filters): format_span only normalizes labels, not entity text by @k-dickinson in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887 fix: make delete_cache a no-op when the cache directory is absent by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035 Add Terretaqa task into Catalanbench by @ivanmartinezmurillo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784 Add make_table regression coverage by @aryanputta in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788 Add cieaCOVA task into catalan bench by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4043 fix(irokobench): repair afrimmlu/afrimgsm/afrixnli task registration and update READMEs by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3841 fix: fall back to tokenizer.eos_token when decode returns empty string by @ganeshr10 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657 fix(med_prescriptions): require both keys before combining complaints and diagnosis by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4041 fix: local directory with task name no longer shadows registered task by @nloughl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670 feat(models): add raw onnxruntime backend for Model Builder ONNX exports by @amd-sourjya in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984 Align pubmedqa task name in README with yaml by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/4025 Serialize all numpy scalar types in JSON output, not just int64/int32 by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885 Fix TER metric direction (higher_is_better) and correct chrF docstring by @borgr in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993 test(registry): add tests for higher_is_better directions by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4050 Fix ContextSampler crash when few-shot pool contains duplicate eval_doc rows by @Kymi808 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790 Fix group stderr to match weight_by_size=False (unweighted) aggregation by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3882 Fix MultiChoiceRegexFilter prefix-shadowing of choice text by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3884 MMLU Task Name by @Teddygat0r in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660 Add mlqa tag to run all variants. by @ivanbaldo in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977 Create missing request-cache parent directories by @sdivyanshu90 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047 Fix the Unitxt init method to set the task name by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225 Pass dataset_kwargs for Unitxt tasks by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3230 feat: add chrf++ aggregation and metric (word_order=2) by @KrishVenky in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780 feat(tasks): add LongProc benchmark (6 task types, 16 configs) by @xiye17 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544 feat(tydiqa): add TyDiQA Gold Passage tasks (9 languages) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/4044 add GreekMMLU (official native-sourced benchmark) task configuration by @mersinkonomi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581 feat(cli): allow key=value to be passed to --metadata; linting by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4054 Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py by @vnayakde in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372 fix(minerva_math): correct few-shot prompt LaTeX by @Ben3892 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045 fix(minerva_math): score an answer identical to the gold as correct by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034 fix(minerva_math): thousands-separator comma strip fuses bare digit tuples ("0,1" -> "01") by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4039 fix(minerva_math): stop sqrt shorthand normalization from corrupting indexed roots by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4037 fix(ruler): resolve the tokenizer name before the cached lookup by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4048 chore(pre-commit) by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4060 fix(scripts): repair requests_caching.py entrypoint (bad kwargs + helper defined after use) by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4059 Add Uncheatable Eval by @ziqing-huang in https://github.com/EleutherAI/lm-evaluation-harness/pull/3442 Add IndicXNLI Gujarati task (indicxnli_gu) by @bhaumik611 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056 feat(physics_gre): add InflectionAI Physics GRE multiple-choice task by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3853 fix(evaluator): report a sample count that does not depend on metric order by @arthi-arumugam-git in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068 Fix/indicparam citation by @viveks-codes in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066 fix: make --use_cache keys hashable for multimodal image/byte requests by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4040 </details> New Contributors @jaydeepborkar made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768 @EphraiemSarabamoun made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795 @DaoyuanLi2816 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824 @he-yufeng made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822 @Apeironics made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803 @wm901115nwpu made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817 @bongho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860 @discobot made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842 @hjoshifonteva made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826 @ayaangazali made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977 @shubhangithub made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943 @gowtham-sai-yadav made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950 @cameronshinn made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921 @Solaris-star made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944 @hancheolcho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923 @chuenchen309 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937 @SiavashShams made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978 @AbdullahRasheed45 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971 @adityasingh2400 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979 @oberpierre made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991 @OrionArchitekton made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916 @yaodong-shen made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959 @thiagocrepaldi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960 @tandede made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020 @winklemad made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003 @k-dickinson made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887 @feiiiiii5 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035 @ivanmartinezmurillo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784 @aryanputta made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788 @ganeshr10 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657 @nloughl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670 @amd-sourjya made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984 @iamsharduld made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885 @borgr made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993 @Kymi808 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790 @Teddygat0r made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660 @ivanbaldo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977 @sdivyanshu90 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047 @mprahl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225 @KrishVenky made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780 @xiye17 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544 @mersinkonomi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581 @vnayakde made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372 @Ben3892 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045 @nata2627 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034 @bhaumik611 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056 @arthi-arumugam-git made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068 @viveks-codes made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066 Full Changelog: https://github.com/EleutherAI/lm-evaluation-harness/compare/v0.4.12...v0.4.13 What's Changed feat: 0.4.13.dev0 by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3764 update global piqa by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3816 fix for keyerror while running humaneval_infilling by @jaydeepborkar in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768 fix: strip whitespace from input/target for FDA, SWDE, and SQuAD_completion tasks by @EphraiemSarabamoun in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795 Fix "trasnlation" typo in DarijaBench translation task names by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824 fix(ruler): replace defunct hotpot dataset host with pinned HF mirror by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3806 fix: keep Anthropic stop sequences nonempty by @he-yufeng in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822 fix(vllm): warn when device argument is ignored by @Apeironics in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803 fix sglang args by @wm901115nwpu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817 feat(legalbench): add HELM-lite LegalBench subset (5 tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860 Fix space in North Macedonian task identifiers (INCLUDE suite) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3848 fix(kormedmcqa): extract answer choice per paper Appendix B by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842 Fix dataset paths for xnli, xcopa, paws-x, and xquad by @Chessing234 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3870 Add IndicParam: MCQ benchmark for 12 low-resource Indic languages by @hjoshifonteva in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826 feat(portuguese_bench): add ASSIN2 RTE and STS tasks by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3812 fix(prost): load PROST without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977 fix(spanish_bench): load wnli_es without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3982 fix(tests): pass dataset_kwargs in test_download by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3980 fix(tasks): use namespaced dataset path for wsc273 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3946 fix(tasks): use namespaced dataset paths for webqs, medmcqa, and wmt16 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3942 fix(arithmetic): load arithmetic tasks without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3976 fix(mmlusr): restore dataset loading for all 171 MMLU-SR tasks by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3972 fix(scrolls): load SCROLLS without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3975 fix(tasks): use script-less parquet mirrors for mathqa, siqa, and moral_stories by @shubhangithub in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943 fix(tmmluplus): use ikala/tmmluplus so the task loads on datasets>=4 by @gowtham-sai-yadav in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950 Fix broken afrobench group task references (afrisenti/mafand prompt_2) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3847 fix(longbench): skip blank leading line in code_sim_score extraction by @cameronshinn in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921 fix(afrixnli): use Jinja braces in prompt_1 doc_to_text by @Solaris-star in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944 fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs by @hancheolcho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923 Fix fewshot_config.split precedence in TaskConfig by @chuenchen309 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937 Exclude eval docs from first-n few-shot samples by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978 fix: prevent ValueError when batch_size="auto" is passed to neuronx model by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971 fix: prevent ValueError when batch_size="auto:N" is passed to API models by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3970 fix: resolve fewshot gen_prefix against the fewshot doc, not the eval doc by @adityasingh2400 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979 Putnam Axiom by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3998 feat(legalbench): add Contract NLI suite (14 NDA entailment tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3954 Fix megatron_lm backend against Megatron-LM core_v0.18+: argument parsing moved out of initialize_megatron() by @oberpierre in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991 chore(ci): update pre-commit hooks and workflow packages by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4023 fix(hf): pass max_cpu_memory through to accelerate's max_memory by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4016 chore(megatron): run linter by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4024 fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal) by @OrionArchitekton in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916 Honor configured timeout for synchronous API requests by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3995 fix(api): support think_end_token for chat completions by @yaodong-shen in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959 feat(models): add cross-platform onnxruntime-genai backend + refactor winml by @thiagocrepaldi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960 Enable function resolving for custom tasks by @SkyR0ver in https://github.com/EleutherAI/lm-evaluation-harness/pull/3992 fix(config): parse dictionary strings from YAML configs by @tandede in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020 fix: normalize scalar seed from a config file the way the CLI does by @winklemad in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003 fix(filters): format_span only normalizes labels, not entity text by @k-dickinson in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887 fix: make delete_cache a no-op when the cache directory is absent by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035 Add Terretaqa task into Catalanbench by @ivanmartinezmurillo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784 Add make_table regression coverage by @aryanputta in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788 Add cieaCOVA task into catalan bench by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4043 fix(irokobench): repair afrimmlu/afrimgsm/afrixnli task registration and update READMEs by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3841 fix: fall back to tokenizer.eos_token when decode returns empty string by @ganeshr10 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657 fix(med_prescriptions): require both keys before combining complaints and diagnosis by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4041 fix: local directory with task name no longer shadows registered task by @nloughl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670 feat(models): add raw onnxruntime backend for Model Builder ONNX exports by @amd-sourjya in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984 Align pubmedqa task name in README with yaml by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/4025 Serialize all numpy scalar types in JSON output, not just int64/int32 by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885 Fix TER metric direction (higher_is_better) and correct chrF docstring by @borgr in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993 test(registry): add tests for higher_is_better directions by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4050 Fix ContextSampler crash when few-shot pool contains duplicate eval_doc rows by @Kymi808 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790 Fix group stderr to match weight_by_size=False (unweighted) aggregation by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3882 Fix MultiChoiceRegexFilter prefix-shadowing of choice text by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3884 MMLU Task Name by @Teddygat0r in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660 Add mlqa tag to run all variants. by @ivanbaldo in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977 Create missing request-cache parent directories by @sdivyanshu90 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047 Fix the Unitxt init method to set the task name by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225 Pass dataset_kwargs for Unitxt tasks by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3230 feat: add chrf++ aggregation and metric (word_order=2) by @KrishVenky in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780 feat(tasks): add LongProc benchmark (6 task types, 16 configs) by @xiye17 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544 feat(tydiqa): add TyDiQA Gold Passage tasks (9 languages) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/4044 add GreekMMLU (official native-sourced benchmark) task configuration by @mersinkonomi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581 feat(cli): allow key=value to be passed to --metadata; linting by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4054 Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py by @vnayakde in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372 fix(minerva_math): correct few-shot prompt LaTeX by @Ben3892 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045 fix(minerva_math): score an answer identical to the gold as correct by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034 fix(minerva_math): thousands-separator comma strip fuses bare digit tuples ("0,1" -> "01") by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4039 fix(minerva_math): stop sqrt shorthand normalization from corrupting indexed roots by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4037 fix(ruler): resolve the tokenizer name before the cached lookup by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4048 chore(pre-commit) by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4060 fix(scripts): repair requests_caching.py entrypoint (bad kwargs + helper defined after use) by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4059 Add Uncheatable Eval by @ziqing-huang in https://github.com/EleutherAI/lm-evaluation-harness/pull/3442 Add IndicXNLI Gujarati task (indicxnli_gu) by @bhaumik611 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056 feat(physics_gre): add InflectionAI Physics GRE multiple-choice task by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3853 fix(evaluator): report a sample count that does not depend on metric order by @arthi-arumugam-git in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068 Fix/indicparam citation by @viveks-codes in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066 fix: make --use_cache keys hashable for multimodal image/byte requests by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4040 New Contributors @jaydeepborkar made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768 @EphraiemSarabamoun made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795 @DaoyuanLi2816 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824 @he-yufeng made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822 @Apeironics made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803 @wm901115nwpu made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817 @bongho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860 @discobot made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842 @hjoshifonteva made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826 @ayaangazali made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977 @shubhangithub made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943 @gowtham-sai-yadav made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950 @cameronshinn made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921 @Solaris-star made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944 @hancheolcho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923 @chuenchen309 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937 @SiavashShams made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978 @AbdullahRasheed45 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971 @adityasingh2400 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979 @oberpierre made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991 @OrionArchitekton made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916 @yaodong-shen made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959 @thiagocrepaldi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960 @tandede made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020 @winklemad made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003 @k-dickinson made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887 @feiiiiii5 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035 @ivanmartinezmurillo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784 @aryanputta made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788 @ganeshr10 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657 @nloughl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670 @amd-sourjya made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984 @iamsharduld made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885 @borgr made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993 @Kymi808 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790 @Teddygat0r made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660 @ivanbaldo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977 @sdivyanshu90 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047 @mprahl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225 @KrishVenky made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780 @xiye17 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544 @mersinkonomi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581 @vnayakde made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372 @Ben3892 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045 @nata2627 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034 @bhaumik611 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056 @arthi-arumugam-git made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068 @viveks-codes made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066 Full Changelog: https://github.com/EleutherAI/lm-evaluation-harness/compare/v0.4.12...v0.4.13
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Self-citationMamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
Self-citationC-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2023) · arXiv · doi:10.48550/arxiv.2309.07597 · Cited by 514
Self-citation
Metadata
- Authors
- Sutawika, L., Gao, L., Schoelkopf, H., Biderman, S., Tow, J., Abbasi, B., fattori, B., Lovering, C., farzanehnakhaee70, Phang, J., Thite, A., Fazz, Aflah, Muennighoff, N., Wang, T., sdtblck, nopperl, gakada, tttyuntian, researcher2, Chris, Etxaniz, J., Kasner, Z., Khalid, Hsu, J., AndyZwei, Ammanamanchi, P. S., Groeneveld, D., Smith, E., & Tang, E.
- Venue
- Zenodo (CERN European Organization for Nuclear Research)
- Publisher
- Zenodo
- Published
- 2026-08-31
- OpenAlex
- W6893809404
- Cited by
- 121
- References
- 0
- Type
- software
- Language
- en
- Open access
- green
- Licence
- opensource.org/licenses/MIT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A framework for few-shot language model evaluation. https://app.cochat.ai/papers/10.5281/zenodo.5371628
- The paper
- Sutawika, L., Gao, L., Schoelkopf, H., Biderman, S., Tow, J., Abbasi, B., fattori, B., Lovering, C., farzanehnakhaee70, Phang, J., Thite, A., Fazz, Aflah, Muennighoff, N., Wang, T., sdtblck, nopperl, gakada, tttyuntian, . . . Tang, E. (2021). A framework for few-shot language model evaluation. Zenodo (CERN European Organization for Nuclear Research). https://doi.org/10.5281/zenodo.5371628
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.