DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation
Open accessAbstract
Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DeepResearchEval, ReportBench, DeepScholar-Bench, and DRBench rely on synthetic task generation.
“DeepResearchEval [Wang et al., 2026], ReportBench [Li et al., 2025b], DeepScholar-Bench [Patel et al., 2025], and DRBench [Abaskohi et al., 2025], in contrast, rely on synthetic task generation.”
From DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity (2026) · cites this paper as Wang et al., 2026
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 30
- Amirhossein Abaskohi, Tianyi Chen, Miguel Muñoz- Mármol, Curtis Fox, Amrutha Varshini Ramesh, Étienne Marcotte, Xing Han Lù, Nicolas Chapados, Spandana Gella, Christopher Pal, and 1 others. 2025. Drbench: A realistic benchmark for enterprise deep research. arXiv preprint arXiv:2510.00172. · arXiv
- Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, and 1 others. 2025. Browsecomp-plus: A more fair and transparent evalua- tion benchmark of deep-research agent. arXiv preprint arXiv:2508.06600. · arXiv
- Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. Deepresearch bench: A com- prehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763. · arXiv
- Tianyu Fan, Xinyao Niu, Yuxiang Zheng, Fengji Zhang, Chengen Huang, Bei Chen, Junyang Lin, and Chao Huang. 2025. Understanding deepresearch via reports. arXiv preprint arXiv:2510.07861. · arXiv
- Gemini. 2025. Gemini 3 pro model card. Gemini. 2025. Gemini deep research. https://gemini. google/overview/deep-research/. Google Gemini. 2025. Gemini 2.5: Pushing the fron- tier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities. Preprint, arXiv:2507.06261. · arXiv
- Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, and 1 others. 2025. Mind2web 2: Evaluating agentic search with agent-as-a-judge. arXiv preprint arXiv:2506.21506. · arXiv
- Janghoon Han, Heegyu Kim, Changho Lee, Dahm Lee, Min Hyung Park, Hosung Song, Stanley Jungkyu Choi, Moontae Lee, and Honglak Lee. 2025. Deer: A com- prehensive and reliable benchmark for deep-research expert reports. Preprint, arXiv:2512.17776. · arXiv
- Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534. · arXiv
- Fangyu Lei, Jinxiang Meng, Yiming Huang, Junjie Zhao, Yitong Zhang, Jianwen Luo, Xin Zou, Ruiyi Yang, Wenbo Shi, Yan Gao, Shizhu He, Zuo Wang, Qian Liu, Yang Wang, Ke Wang, Jun Zhao, and Kang Liu. 2025. Dacomp: Benchmarking data agents across the full data intelligence lifecycle. Preprint, arXiv:2512.04324. · arXiv
- Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, and Kai Jia. 2025a. Reportbench: Evaluating deep re- search agents via academic survey tasks. arXiv preprint arXiv:2508.15804. · arXiv
- Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yu- tao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. 2025b. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776. · arXiv
- Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. · arXiv
- Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chao- fan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. · arXiv
- Haotian Luo, Huaisong Zhang, Xuelin Zhang, Haoyu Wang, Zeyu Qin, Wenjie Lu, Guozheng Ma, Haiying He, Yingsha Xie, Qiyang Zhou, Zixuan Hu, Hongze Mi, Yibo Wang, Naiqiang Tan, Hong Chen, Yi R. Fung, Chun Yuan, and Li Shen. 2025. Ultrahorizon: Bench- marking agent capabilities in ultra long-horizon scenar- ios. Preprint, arXiv:2509.21766. · arXiv
- Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations.
- Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, and 1 oth- ers. 2025. Humanity's last exam. arXiv preprint arXiv:2501.14249. · arXiv
- Zile Qiao, Guoxin Chen, Xuanzhong Chen, Donglei Yu, Wenbiao Yin, Xinyu Wang, Zhen Zhang, Baixuan Li, Huifeng Yin, Kuan Li, and 1 others. 2025. Webre- searcher: Unleashing unbounded reasoning capability in long-horizon agents. arXiv preprint arXiv:2509.13309. · arXiv
- Tongyi DeepResearch Team, Baixuan Li, Bo Zhang, Dingchu Zhang, Fei Huang, Guangyu Li, Guoxin Chen, Huifeng Yin, Jialong Wu, Jingren Zhou, and 1 others. 2025. Tongyi deepresearch technical report. arXiv preprint arXiv:2510.24701. · arXiv
- Haiyuan Wan, Chen Yang, Junchi Yu, Meiqi Tu, Jiaxuan Lu, Di Yu, Jianbao Cao, Ben Gao, Jiaqing Xie, Aoran Wang, and 1 others. 2025. Deepresearch arena: The first exam of llms' research abilities via seminar-grounded tasks. arXiv preprint arXiv:2509.01396. · arXiv
- Jiayu Wang, Yifei Ming, Riya Dulepet, Qinglin Chen, Austin Xu, Zixuan Ke, Frederic Sala, Aws Albarghouthi, Caiming Xiong, and Shafiq Joty. 2025. Liveresearch- bench: A live benchmark for user-centric deep research in the wild. arXiv preprint arXiv:2510.14240. · arXiv
- Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKin- ney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet chal- lenging benchmark for browsing agents. Preprint, arXiv:2504.12516. · arXiv
- Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Zixia Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V Le. 2024. Long-form factuality in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Ha- jishirzi, Danqi Chen, and Luca Soldaini. 2025. Organize the web: Constructing domains enhances pre-training data curation. In Forty-second International Conference on Machine Learning.
- Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao, Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, and 1 others. 2025. Widesearch: Bench- marking agentic broad info-seeking. arXiv preprint arXiv:2508.07999. · arXiv
- Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. 2025. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv preprint arXiv:2507.16280. · arXiv
- An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. · arXiv
- Yang Yao, Yixu Wang, Yuxuan Zhang, Yi Lu, Tianle Gu, Lingyu Li, Dingyi Zhao, Keming Wu, Haozhe Wang, Ping Nie, and 1 others. 2025. A rigorous bench- mark with multidimensional evaluation for deep re- search agents: From answers to reports. arXiv preprint arXiv:2510.02190. · arXiv
- Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. Glm-4.5: Agen- tic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471. · arXiv
- Dingling Zhang, He Zhu, Jincheng Ren, Kangqi Song, Xinran Zhou, Boyu Feng, Shudong Liu, Jiabin Luo, Weihao Xie, Zhaohui Wang, and 1 others. 2025. How far are we from genuinely useful deep research agents? arXiv preprint arXiv:2512.01948. · arXiv
- Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. Deepresearcher: Scaling deep research via reinforce- ment learning in real-world environments. arXiv preprint arXiv:2504.03160. · arXiv
Cited by
Sorted by citations
DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity
Zhong, J., Zhang, H., Southern, C., Yang, J., Wang, T., Jung, K., Zhang, S., Yarats, D., Ho, J., & Ma, J. (2026) · arXiv · doi:10.48550/arxiv.2602.11685
“DeepResearchEval [Wang et al., 2026], ReportBench [Li et al., 2025b], DeepScholar-Bench [Patel et al., 2025], and DRBench [Abaskohi et al., 2025], in contrast, rely on synthetic task generation.”
LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity
Wang, Y., Jin, S., Ning, Y., Liang, X., Luo, H., Zhang, J., He, Y., Wu, W., & Clarenceai (2026) · arXiv · doi:10.48550/arxiv.2610.18092
“More recent work further automates this pipeline: DeepResearchEval (Wang et al., 2026b) combines adaptive quality assessment with active fact-checking, while LiveResearchBench (Wang et al., 2026a) and WideSearch (Wong et al., 2026) extend rubric-style grading to dynamic, user-centric reports and structured-table deliv…”
Self-citationMiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome
Ye, F., Hu, Y., Zhu, P., Li, Y., Jin, Z., Xiao, Y., Wang, Y., Wang, L., Zhang, Z., Wang, L., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.28407
Self-citation
- Authors
- Wang, Y., Wang, L., Deng, Y., Wu, K., Xiao, Y., Yao, H., Kang, L., Ye, H., Jing, Y., & Bing, L.
- Affiliations
- Nanyang Technological University (SG)
- Infinity (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2601.09688
- OpenAlex
- W7124300113
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- task construction, fact-checking, citation verification, evaluation benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
