DeepResearchGym: A Free, Transparent, and Reproducible Sandbox for Deep Research
Open accessAbstract
Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries, and/or answers to hard-to-locate factual questions. However, most existing systems rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges, in addition to high costs. To address these limitations, we introduce DeepResearchGym as a free and open-source search sandbox for reproducible research on deep research systems. The sandbox provides a search API that indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves comparable latency to popular commercial APIs while ensuring stable document rankings across runs. We demonstrate the sandbox's utility through two use cases. For training, we synthesize queries grounded in the indexed corpora and show that search agents trained within the sandbox generalize to commercial search at inference time, enabling cost-effective reinforcement learning. For evaluation, we extend the Researchy Questions benchmark with LLM-as-a-judge metrics to measure alignment with users' information needs, retrieval faithfulness, and report quality. Evaluation results show that system rankings remain consistent when switching from commercial APIs to ours.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Broder, A. (2002) · ACM SIGIR Forum · doi:10.1145/792550.792552 · Cited by 1,944 · see citing works
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · see citing works
Search-o1: Agentic Search-Enhanced Large Reasoning ModelsPreprint
Li, X., Dong, G.-T., Jin, J., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2501.05366 · Cited by 496 · see citing works
Self-citationFActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
Gao, J.-X., Fu, W., Xie, M., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2508.07976 · Cited by 166
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.20 · Cited by 148 · see citing works
Analysis of long queries in a large scale search log
Bendersky, M., & Croft, W. B. (2009) · WSCD '09 · doi:10.1145/1507509.1507511 · Cited by 122
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.03300 · Cited by 101 · see citing works
Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL
Li, W.-Z., Lin, J.-B., Jiang, Z.-S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2508.13167 · Cited by 77
Phan, L. (2025) · doi:10.21203/rs.3.rs-6615297/v1 · Cited by 65 · see citing works
ClueWeb22: 10 Billion Web Documents with Rich Information
Overwijk, A., Xiong, C., & Callan, J. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3536321 · Cited by 64
Self-citationOpen Deep Search: Democratizing Search with Open-source Reasoning Agents
Alzubi, S., Brooks, C., Chiniya, P., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2503.20201 · Cited by 54
(2024) · doi:10.52202/079017-0970 · Cited by 53 · see citing works
Long-form factuality in large language models
Wei, J., Yang, C., Song, X., Lu, Y., Hu, N., Huang, J., Tran, D., Peng, D., Liu, R., Huang, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.18802 · Cited by 36 · see citing works
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
Saha, S., Levy, O., Çelikyılmaz, A., Bansal, M., Weston, J., & Li, X. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.462 · Cited by 32 · see citing works
Deep Researcher with Test-Time Diffusion
Han, R.-J., Chen, Y.-F., CuiZhu, Z., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.16075 · Cited by 30
Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
Thakur, N., Pradeep, R., Upadhyay, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.15205 · Cited by 29
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments
Zheng, Y., Fu, D., Hu, X., Cai, X., Ye, L., Lu, P., & Liu, P. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.22 · Cited by 25 · see citing works
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments
Zheng, Y., Fu, D., Hu, X., Cai, X., Ye, L., Lu, P., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2504.03160 · Cited by 25 · see citing works
A benchmark of expert-level academic questions to assess AI capabilities
Center for AI Safety, Phan, L., Gatti, A., Li, N., Khoja, A., Kim, R., Ren, R., Hausenloy, J., Zhang, O., Mazeika, M., et al. (2026) · Nature · doi:10.1038/s41586-025-09962-4 · Cited by 23 · see citing works
Self-citationSearch-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., & Han, J. (2025) · arXiv · doi:10.48550/arxiv.2503.09516 · Cited by 13 · see citing works
WebThinker: Empowering Large Reasoning Models with Deep Research Capability
Li, X., Jin, J., Dong, G., Qian, H., Wu, Y., Wen, J.-R., Zhu, Y., & Dou, Z. (2025) · arXiv · doi:10.48550/arxiv.2504.21776 · Cited by 7
Self-citationResearchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents
Rosset, C., Chung, H.-L., Qin, G., Chau, E. C., Feng, Z., Awadallah, A., Neville, J., & Rao, N. (2024) · arXiv · doi:10.48550/arxiv.2402.17896 · Cited by 6 · see citing works
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Liu, A., Green, J., Patel, K., Meng, R., Su, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.06600 · Cited by 3 · see citing works
An Interactive Paradigm for Deep ResearchPreprint
Ai, L., Bursztyn, V. S., Chen, X., Hirschberg, J., & Mitra, S. (2026) · arXiv.org · doi:10.48550/arxiv.2605.24266 · Cited by 2 · see citing works
Beneficial Reasoning Behaviors in Agentic Search and Effective Training Methods to Obtain Them
Jin, J., Paladugu, A. S., & Xiong, C. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1400
Self-citationGu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594
Taxonomy-driven lumping for sequence mining
Bonchi, F., Castillo, C., Donato, D., et al. (2009) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-009-0141-6
Flagged first, then by citations
Research works printed but not linked: 22
- Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. In Conference on Language Modeling, COLM.
- Shengding Hu et al., 2024. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. In Conference on Language Modeling, COLM.
- Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2025. Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation. In Conference of the North American Chapter of the Association for Computational Linguistics, NAACL.
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Annual Conference on Neural Information Processing Systems, NeurIPS.
- Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A Benchmark for General AI Assistants. In International Conference on Learning Representations, ICLR.
- Niklas Muennighoff. 2022. SGPT: GPT Sentence Embeddings for Semantic Search. ArXiv abs/2202.08904. · arXiv
- Xuan-Phi Nguyen, Shrey Pandit, Revanth Gangi Reddy, Austin Xu, Silvio Savarese, Caiming Xiong, and Shafiq Joty. 2025. SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agent. ArXiv abs/2509.06283. · arXiv
- Jingjie Ning, João Coelho, Yibo Kong, Yunfan Long, Bruno Martins, João Magalhães, Jamie Callan, and Chenyan Xiong. 2026. Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M Real Search Requests. In International Conference on Research and Development in Information Retrieval, SIGIR.
- OpenAI. 2025. Introducing Deep Research. https://openai.com/index/introducing-deep-research/ OpenAI Blog. May 11, 2025.
- Zehan Qi Rongwu Xu Zhijiang Guo Cunxiang Wang Hao Zhang and Wei Xu. 2024. LONG(^2)RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall. In Findings of the Association for Computational Linguistics EMNLP.
- Rishiraj Saha Roy, Smith Agarwal, Niloy Ganguly, and Monojit Choudhury. 2016. Syntactic Complexity of Web Search Queries through the Lenses of Language Models, Networks and Users. Information Processing and Management, Vol. 52, 5 (2016).
- Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation. In Annual Conference on Neural Information Processing Systems, NeurIPS.
- Serper. 2025. Serper.dev Search API. https://serper.dev Accessed May 11, 2025.
- Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Liu, Jian Yang, Ge Zhang, Jiaheng Liu, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2025. TaskCraft: Automated Generation of Agentic Tasks. ArXiv abs/2506.10055. · arXiv
- Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. ArXiv abs/2503.05592. · arXiv
- Suhas Jayaram Subramanya, Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnaswamy, and Rohan Kadekodi. 2019. DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node. In Annual Conference on Neural Information Processing Systems, NeurIPS.
- Tavily. 2025. Tavily Search API. https://tavily.com Accessed May 11, 2025.
- Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Conference on Neural Information Processing System, NeurIPS.
- Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In Annual Meeting of the Association for Computational Linguistics, ACL.
- Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. 2025. WebWalker: Benchmarking LLMs in Web Traversal. In Annual Meeting of the Association for Computational Linguistics, ACL.
- Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A Critical Evaluation of Evaluations for Long-form Question Answering. In Annual Meeting of the Association for Computational Linguistics, ACL.
- Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung-Yi Ho, and Philip S. Yu. 2024. Trustworthiness in Retrieval-Augmented Generation Systems: A Survey. ArXiv abs/2409.10102. · arXiv
Cited by
Sorted by citations
ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Sharma, M., Zhang, C. B. C., Bandi, C., Wang, C., Aich, A., Nghiem, H., Rabbani, T., Htet, Y., Jang, B., Basu, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.07685 · Cited by 1 · see citing works
- Authors
- Coelho, J., Ning, J., He, J., Mao, K., Paladugu, A. S., Setlur, P., Jin, J., Callan, J., Magalhães, J., Martins, B., & Xiong, C.
- Affiliations
- Carnegie Mellon University (US)
- Laboratório para a Ciência da Computação e Informática (PT)
- Instituto de Engenharia de Sistemas e Computadores Investigação e Desenvolvimento (PT)
- ORCID
- Venue
- Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR)
- Publisher
- ACM
- Published
- 2026-07-24
- Pages
- 34-43
- OpenAlex
- W7168300957
- Semantic Scholar
- 026cef5c6d273aab6119191f3f7b116edfdd6298
- Cited by
- 0
- References
- 29
- FWCI
- 0.00
- Type
- conference paper
- Open access
- gold
- doi.org
- Funders
- Portuguese Recovery and Resilience Plan (645008882-00000055)
- Fundação para a Ciência e a Tecnologia, I.P. (FCT) (UID\/50021\/2025, UID\/PRR\/50021\/2025)
- CMU Portugal Program (PRT\/BD\/153683\/2021)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- reproducibility, dense retrieval, approximate nearest neighbor search, query synthesis, LLM-as-a-judge, reporting quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
