CoChat
Sign Up
PreprintCited by 63Artificial Intelligence ApplicationsScientific Computing and Data ManagementAI-based Problem Solving and Planning

The DOI matches its DataCite record; 45 of the 45 references it cites were found in a registry or an index; 4 of the 50 sources its claims cite were checked for support.

AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

Open access
Bragg, Jonathan, D'Arcy, Mike, Balepur, Nishant, Bareket, Dan, Dalvi, Bhavana, Feldman, Sergey, Haddad, Dany, Hwang, Jena D., Jansen, Peter, Kishore, Varsha, Majumder, Bodhisattwa Prasad, Naik, Aakanksha, Rahamimov, Sigal, Richardson, Kyle, Singh, Amanpreet, Surana, Harshit, Tiktinsky, Aryeh, Vasu, Rosni, Wiener, Guy, Anastasiades, Chloe, Candra, Stefan, Dunkelberger, Jason, Emery, Dan, Evans, Rob, Hamada, Malachi, Huff, Regan, Kinney, Rodney, Latzke, Matt, Lochner, Jaron, Lozano-Aguilera, Ruben, Nguyen, Cecile, Rao, Smita, Tanaka, Amber, Vlahos, Brooke, Clark, Peter, Downey, Doug, Goldberg, Yoav, Sabharwal, Ashish, Weld, Daniel S.·2025·arXiv
DOI 10.48550/arxiv.2510.21652arXiv 2510.21652
Open this paper in CoChat — free accountOpen PDF

Abstract

AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose "deep research" systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

AstaBench is a new holistic benchmark suite for evaluating AI agents on scientific research assistance, comprising 2,400+ problems across 11 benchmarks, a controlled environment with production-grade literature search tools, and a cost-aware leaderboard. Evaluating 57 agents across 22 agent classes, the authors find that while literature understanding is reasonably strong, coding, experiment execution, data analysis, and end-to-end discovery remain largely unsolved.

Key findings

  • The best overall agent, Asta v0 (a mixture of models), scores 53.0% overall versus 44.0% for ReAct with gpt-5; the best open-source agent with open-weight LLMs scores only 11.1% (Smolagents Coder with Llama-4-Scout-17B-16E-Instruct).
  • Cost matters: ReAct with gpt-5-mini scores 32% overall at $0.04 per problem, "within 21% (absolute) of the best performing models" while costing over an order of magnitude less.
  • Coding and execution are far from solved: all but two agents scored below 25% on SUPER-Expert (ReAct with gpt-5 scored 41%, gpt-5-mini 37%). Data analysis maxed out at 34% on DiscoveryBench.
  • End-to-end discovery is near-zero: although individual research-step completion reaches ~70%, the probability of completing all (~10–15) steps is ≈0.7^10 ≈ 3%, with actual numbers reaching a maximum of 5%.
  • Literature understanding is comparatively strong: best models score around 80% on QA, with Asta Scholar QA, Elicit, and SciSpace Deep Review all at ~85% or higher on ScholarQA-CS2; literature review table generation remains weak, with recall scores around 43%.
  • gpt-5's effect is unpredictable: it boosts ReAct substantially (+13.4% on ScholarQA-CS2, +24.8% on SUPER-Expert, +25.3% on LitQA2-FullText-Search, +21.1% on E2E-Bench-Hard) but hurts several specialized agents (Asta Scholar QA, Asta DataVoyager, Asta Code).

Methods

  • Design: 11 benchmarks spanning four categories — Literature Understanding (PaperFindingBench, LitQA2-FullText, LitQA2-FullText-Search, ScholarQA-CS2, ArxivDIGESTables-Clean), Code & Execution (SUPER-Expert, CORE-Bench-Hard-, DS-1000), Data Analysis (DiscoveryBench), and End-to-End Discovery (E2E-Bench, E2E-Bench-Hard) — totaling 2,400+ problems, many inspired by real user requests to deployed Asta agents.
  • Infrastructure: the Asta Environment provides standardized, decoupled tools (Asta Scientific Corpus with date-restricted search, a computational notebook sandbox), and the agent-eval toolkit built on Inspect provides time-invariant cost accounting and leaderboards.
  • Evaluation: 57 agents across 22 agent classes were run over several months; scoring uses LLM-as-judge against rubrics for some benchmarks (PaperFindingBench, ScholarQA-CS2, ArxivDIGESTables-Clean, DiscoveryBench, E2E-Bench, E2E-Bench-Hard) and programmatic evaluation for others (LitQA2-FullText, LitQA2-FullText-Search, SUPER-Expert, CORE-Bench-Hard-, DS-1000); 95% CIs are reported (±1.96 × SE).

Limitations

  • Stated: E2E-Bench problems are specified in considerable detail, so they "only weakly test the ideation and planning steps"; rubric ingredients for ScholarQA-CS2 are sourced from evaluated systems, and hold-out experiments showed significant drops for three systems (average 2.5 points, p ≤ 0.01); semantic-query recall in PaperFindingBench uses an estimated set size rather than a definitive gold set; category-level standard errors are propagated assuming independence between tasks, "which could slightly underestimate uncertainty."
  • Stated: Asta v0 was not evaluated with gpt-5 (experiments began before its release), and Asta Code was selected based on early experiments with older models despite ReAct with o3 performing better on SUPER-Expert.
  • Evident from the text: several agents (Elicit, SciSpace, OpenAI Deep Research, OpenSciLM, Asta Panda, Asta CodeScientist) were graded from cached results rather than live runs, and some commercial agents could not be evaluated on all benchmarks (e.g., no API access for LitQA2-FullText), limiting comparability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.21652

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Using AstaBench we evaluated 22 agent classes on a diverse set of science tasks while controlling the set of available tools, e.g., to ensure each agent has access to the same set of scientific papers. AstaBench leaderboards record not just agents accuracy but also how much computation is required to achieve that performance.
Figure 1 Using AstaBench we evaluated 22 agent classes on a diverse set of science tasks while controlling the set of available tools, e.g., to ensure each agent has access to the same set of scientific papers. AstaBench leaderboards record not just agents accuracy but also how much computation is required to achieve that performance.CC-BY · From the PDF
Figure 2 Score vs. cost analysis for overall and category results (from Tables 4, 11, 16 and 17). Points indicate means. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each category (Literature Understanding, Code & Execution, Data Analysis, End-to-End Discovery). † denotes models not pinned to a date-stamped version. Note: the x-axis (cost per answer in dollars) uses a log scale. For more detailed plots for individual categories and benchmarks, see Appendix D.
Figure 2 Score vs. cost analysis for overall and category results (from Tables 4, 11, 16 and 17). Points indicate means. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each category (Literature Understanding, Code & Execution, Data Analysis, End-to-End Discovery). † denotes models not pinned to a date-stamped version. Note: the x-axis (cost per answer in dollars) uses a log scale. For more detailed plots for individual categories and benchmarks, see Appendix D.CC-BY · From the PDF
Figure 3 Score vs. cost analysis for Literature Understanding search benchmarks (Table 12). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( PaperFindingBench , LitQA2-FullText-Search ). Note: the x-axis (cost) uses a log scale.
Figure 3 Score vs. cost analysis for Literature Understanding search benchmarks (Table 12). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( PaperFindingBench , LitQA2-FullText-Search ). Note: the x-axis (cost) uses a log scale.CC-BY · From the PDF
Figure 4 Score vs. cost analysis for Literature Understanding QA benchmarks (Table 13). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( ScholarQACS2 , LitQA2-FullText ). Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.
Figure 4 Score vs. cost analysis for Literature Understanding QA benchmarks (Table 13). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( ScholarQACS2 , LitQA2-FullText ). Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.CC-BY · From the PDF
Figure 5 Score vs. cost analysis for the Literature Understanding ArxivDIGESTables-Clean benchmark (Table 14). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval. Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.
Figure 5 Score vs. cost analysis for the Literature Understanding ArxivDIGESTables-Clean benchmark (Table 14). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval. Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.CC-BY · From the PDF
Figure 6 Score vs. cost analysis for Code & Execution benchmarks (Table 15). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( CORE-Bench-Hard -, SUPER-Expert , DS-1000 ). Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.
Figure 6 Score vs. cost analysis for Code & Execution benchmarks (Table 15). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( CORE-Bench-Hard -, SUPER-Expert , DS-1000 ). Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.CC-BY · From the PDF
Figure 7 Score vs. cost analysis for Data Analysis sub-benchmarks. Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are denoted with red triangle markers, representing optimal quality-cost trade-offs for each eval ( DiscoveryBench ). † denotes models not pinned to a date-stamped version.
Figure 7 Score vs. cost analysis for Data Analysis sub-benchmarks. Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are denoted with red triangle markers, representing optimal quality-cost trade-offs for each eval ( DiscoveryBench ). † denotes models not pinned to a date-stamped version.CC-BY · From the PDF
Figure 8 Score vs. cost analysis for End-to-End Discovery benchmarks (Table 17). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( E2E-Bench , E2E-BenchHard ). Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.
Figure 8 Score vs. cost analysis for End-to-End Discovery benchmarks (Table 17). Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval ( E2E-Bench , E2E-BenchHard ). Note: the x-axis (cost) uses a log scale. † denotes models not pinned to a date-stamped version.CC-BY · From the PDF
Figure 9 PaperFinder semantic query workflow
Figure 9 PaperFinder semantic query workflowCC-BY · From the PDF
Figure 10 Graphical presentation of scoring a single end-to-end answer. Each row is a different rubric item, columns 3, 4, and 5 show whether that rubric item was met (green), not met (red), or unknown (yellow) by the generated paper, code, and artifacts respectively. Column 6 indicates whether 3-5 are consistent (green) or not (red), with the overall verdict in the last column 7. The overall score is the average of the final column cells (green = 1, red = 0).
Figure 10 Graphical presentation of scoring a single end-to-end answer. Each row is a different rubric item, columns 3, 4, and 5 show whether that rubric item was met (green), not met (red), or unknown (yellow) by the generated paper, code, and artifacts respectively. Column 6 indicates whether 3-5 are consistent (green) or not (red), with the overall verdict in the last column 7. The overall score is the average of the final column cells (green = 1, red = 0).CC-BY · From the PDF
Image 11
CC-BY · From the PDF

Verification

Claims the paper makes

49 No basis given · 47 Own result · 37 Own interpretation · 28 Sources not found · 28 Own method · 3 Check pending · 2 Supported · 2 Partly supported · 2 Not checked yet · accuracy not yet measured

Verify all 5 references — free account
  • It may state two separate claims as one. 66
  • Probably not a substantial claim. 81
  • The quote only partly states this claim. 2
128 of 128

  • The paper relies on this without support.

    Real-world product usage data is typically guarded by product companies and unavailable to academic benchmark creators.

    “Real-world product usage provides an informative basis for determining appropriate tasks, but unfortunately such data is typically guarded by product companies (who use it to create private evaluations) and unavailable to academic benchmark creators.”

  • The paper relies on this without support.

    Large-scale, optimized search indexes are costly to create.

    “yet large-scale, optimized search indexes are costly to create and public search tools are not reproducible”

  • The paper relies on this without support.

    Public search tools are not reproducible.

    “yet large-scale, optimized search indexes are costly to create and public search tools are not reproducible”

  • The paper relies on this without support.

    The Asta user requests used to inspire benchmark problems are representative of real-world scientific research use cases.

    “including many problems inspired by actual user requests to deployed Asta agents”

  • The paper relies on this without support.

    The success rate per step is independent across steps in the end-to-end discovery calculation.

    “given ∼10 steps per experiment, and a success rate of 70% per step, the success rate to complete all steps in the experiment will be ≈ 0.7 10 ≈ 3%”

  • The paper relies on this without support.

    Most previous benchmark suites require general agent developers to adapt agents for individual tasks, introducing developer bias and hindering development.

    “Unfortunately, most previous benchmark suites require general agent developers to adapt agents for individual tasks, introducing developer bias and hindering development.”

  • The paper relies on this without support.

    A large integrated suite of agent baselines is lacking from current agent suites, resulting in most evaluations comparing only to a small number of other agents or ablations on the evaluator's own agent.

    “A large integrated suite of agent baselines must be available to identify which agents are truly state-of-the-art agents and to provide high-quality starting points for future development, yet is lacking from current agent suites resulting in most evaluations comparing only to a small number of other agents or ablations on the evaluator's own agent.”

  • The authors state this as an assumption.

    The propagation of standard errors assumes independence between tasks, which could slightly underestimate uncertainty.

    “This propagation assumes independence between tasks, which could slightly underestimate uncertainty.”

  • The authors state this as an assumption.

    The set contains a mix of queries for which there are many relevant results, and queries for which only a handful of results exist.

    “The set contains a mix of queries for which we assume there are many relevant results, and queries for which we assume only a handful of results exist.”

  • The paper relies on this without support.

    Setting up and running code from sparsely documented repositories accompanying published papers is a common, non-trivial, and time-consuming task.

    “It targets the common yet often non-trivial and time-consuming task of setting up and running code from sparsely documented repositories accompanying published papers.”

  • The authors state this as an assumption.

    The published workflows used as gold trajectories in DiscoveryBench are scientifically valid.

    “As most of the papers are highly cited, peer-reviewed, and from top venues in the domains, it is reasonable to assume the published workflows are scientifically valid.”

  • The paper relies on this without support.

    Data-driven research is the cornerstone of scientific progress in the six selected domains: sociology, biology, humanities, economics, engineering, and meta-science.

    “We focus on six scientific domains where data-driven research is the cornerstone of scientific progress: sociology, biology, humanities, economics, engineering, and meta-science.”

  • The paper relies on this without support.

    Papers in academic scientific-literature retrieval benchmarks focus almost exclusively on semantic criteria.

    “Papers in academic scientific-literature retrieval benchmarks focus almost exclusively on this criteria.”

  • The paper relies on this without support.

    Real-world queries may include additional details such as metadata constraints or other properties.

    “However, real-world queries may include additional details such as metadata constraints or other properties, as discussed below.”

  • The paper relies on this without support.

    The order in which we go over the results matters for efficiency.

    “The order in which we go over the results matters for efficiency.”

  • The paper relies on this without support.

    Asta Paper Finder is actively used by a growing number of users.

    “For Asta Paper Finder, an expanded and continuously developed version of the agent-including a user interface and additional infrastructure-is actively used by a growing number of users.”

  • The paper relies on this without support.

    At the time, PaperQA2 used gpt-4-turbo as the backbone LLM, while FutureHouse Crow, which is based on PaperQA2, uses gpt-4.1-mini.

    “Further, at the time, PaperQA2 used gpt-4-turbo as the backbone LLM, while FutureHouse Crow, which is based on PaperQA2 uses gpt-4.1-mini.”

  • The paper relies on this without support.

    The Perplexity API provides a 'search_mode' parameter that can be set to 'academic' to only retrieve academic sources.

    “The Perplexity API also provides a”

  • The paper relies on this without support.

    The AstaBench tasks are a valid measure of science research assistance.

    “Science research assistance is still far from solved, as evidenced by the generally low overall scores for the full gamut of agents, from fully open to fully closed.”

  • The paper relies on this without support.

    The set of agents evaluated is representative of the full gamut of agents, from fully open to fully closed.

    “the full gamut of agents, from fully open to fully closed”
20 of 20

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 23 references

Research works printed but not linked: 14

  1. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mádry. MLE- bench: Evaluating machine learning agents on machine learning engineering. In ICLR, 2025. URL https://openreview.net/forum?id=6s5uXNWGIh.
  2. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. In ICLR, 2025b. URL https://openreview.net/forum?id=6z4YKr0GK6.
  3. Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, and Weinan E. PaSa: An LLM agent for comprehensive academic paper search. In ACL, 2025. URL https: //aclanthology.org/2025.acl-long.572/.
  4. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In ICML, 2023.
  5. Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste. The BrowserGym ecosystem for web agent research. TMLR, 2025. URL https://openreview.net/forum ?id=5298fKGmv3.
  6. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Data-driven discovery with large generative models. ICML, 2024.
  7. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench: Towards data-driven discovery with large language models. In ICLR, 2025. URL https://openreview.net/pdf?id=vyflgpwfJW.
  8. Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S Weld, Joseph Chee Chang, and Kyle Lo. ArxivDIGESTables: Synthesizing scientific literature into tables using language models. In EMNLP, 2024. URL https://aclantholo gy.org/2024.emnlp-main.538/.
  9. Pritika Ramu, Aparna Garimella, and Sambaran Bandyopadhyay. Is this a bad table? a closer look at the evaluation of table generation from text. In EMNLP, 2024. URL https://aclantholo gy.org/2024.emnlp-main.1239/.
  10. Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE- Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. TMLR, 2025-January:1-31, 2025. URL https://tmlr.org/papers/v2 025/01-2025paper.pdf.
  11. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
  12. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In ICML, 2024.
  13. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In ICLR, 2025. URL https://openreview.net/forum ?id=OJd3ayDDoF.
  14. Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. Scireplicate-bench: Bench- marking llms in agent-driven algorithmic reproduction from research papers. Proceedings of COLM, 2025.

Cited by

Sorted by citations

See all 63

Related

See all

Metadata

Authors
Bragg, J., D'Arcy, M., Balepur, N., Bareket, D., Dalvi, B., Feldman, S., Haddad, D., Hwang, J. D., Jansen, P., Kishore, V., Majumder, B. P., Naik, A., Rahamimov, S., Richardson, K., Singh, A., Surana, H., Tiktinsky, A., Vasu, R., Wiener, G., Anastasiades, C., Candra, S., Dunkelberger, J., Emery, D., Evans, R., Hamada, M., Huff, R., Kinney, R., Latzke, M., Lochner, J., Lozano-Aguilera, R., Nguyen, C., Rao, S., Tanaka, A., Vlahos, B., Clark, P., Downey, D., Goldberg, Y., Sabharwal, A., & Weld, D. S.
Affiliations
  • Allen Institute for Artificial Intelligence (US)
  • University of Maryland, College Park (US)
  • University of Arizona (US)
  • University of Washington (US)
  • University of Zurich (CH)
  • Bar-Ilan University (IL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2510.21652
arXiv
2510.21652
OpenAlex
W7101416451
Cited by
63
References
23
Citations by year
2026: 57 · 2025: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
AI agents, benchmarking, artificial intelligence in science, agent evaluation, reproducibility, search tools, baselines, AI scientists, literature review automation, experimental replication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review