DORA Explorer: Improving the Exploration Ability of LLMs Without Training
Open accessAbstract
Large language model (LLM) agents for sequential decision-making struggle to produce diverse outputs. This leads to insufficient exploration, suboptimal solutions, and repeated actions. Actions are generated at the sequence level, but existing sampling strategies, such as temperature scaling, introduce diversity at the token level, not at the sequence level. We introduce DORA EXPLORER (Diversity-Oriented Ranking of Actions), a training-free, inference-time algorithm for improving exploration in LLM agents. DORA generates multiple candidate actions, scores them using sequence-level log-probability statistics, and samples an action via a tunable exploration parameter. We first study exploration in the classic Multi-Armed Bandit setting, where DORA substantially outperforms temperature-based sampling. Our main evaluation is on the Text Adventure Learning Environment Suite (TALES), where prompting strategies fail to explore but DORA delivers consistent gains across model families, e.g., 31.43% (ReAct) -> 45.5% (DORA) for Qwen-2.5 7B in TextWorld. Beyond exploration, DORA prevents common failures, such as getting stuck in loops. Our project is available at: https://dora-explore.github.io/.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multi-Agent Systems are designed to overcome limitations of single-agent systems in context capacity, insufficient exploration, and inefficient serial execution through specialized collaboration.
“Multi-Agent Systems (MAS) are designed to overcome the limitations of single-agent systems in context capacity (Yu et al., 2026), insufficient exploration (Gurjar et al., 2026), and inefficient serial execution (Masterman et al., 2024), through specialized collaboration (Guo et al., 2024).”
From MS-Eval: Benchmarking Sub-Agent Delegation and Orchestration in MCP-Intensive Working Scenarios (2026) · cites this paper as Gurjar et al., 2026
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033 · cited 4× · see citing works
“DORA achieves UCB-competitive performance on MAB and consistent gains across the TALES environments.”
InfluentialBuffel, C., van Aalst, J., Bangels, A., Toelen, J., Allegaert, K., Verschueren, S., & Stichele, G. V. (2018) · doi:10.2196/preprints.12431 · Cited by 5,871
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · see citing works
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2× · see citing works
“Chain-of-Thought (CoT) prompting Wei et al. (2022) encourages step-by-step reasoning, while Tree-of-Thought (ToT) Yao et al. (2023) extends this to structured search over multiple reasoning paths.”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · cited 1× · see citing works
“The strong reasoning capabilities of auto-regressive LLMs have led to their widespread adoption as decision-making agents across diverse domains, including recommendation systems (Zhu et al., 2025), scientific discovery (Ren et al., 2025), game-playing (Costarelli et al., 2024), healthcare (Gebreab…”
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · cited 2× · see citing works
“…decoding strategies for LLMs. Top-k sampling restricts generation to the k most probable tokens (Fan et al., 2018), top-p (nucleus) sampling (Holtzman et al., 2019) selects the smallest set of tokens whose cumulative probability exceeds a threshold p (Holtzman et al., 2019), and min-p…”
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 2× · see citing works
“The first category of methods is sampling-based decoding strategies for LLMs. Top-k sampling restricts generation to the k most probable tokens (Fan et al., 2018), top-p (nucleus) sampling (Holtzman et al., 2019) selects the smallest set of tokens whose cumulative probability exceeds a threshold p…”
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158 · cited 3× · see citing works
“While prior approaches have attempted to improve LLM agent exploration using pure sampling (Fan et al., 2018), pure prompting (Yao et al., 2023) or through controlled exploration (Troshin et al., 2025), our approach uses aspects of all three approaches to explicitly control the…”
Reinforcement Learning with Deep Energy-Based PoliciesPreprint
Haarnoja, T., Tang, H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08165 · Cited by 638 · cited 1× · see citing works
“The final action a ∗ t is sampled via a ∗ t ∼ Categorical ( p λ ) , where the likelihood of each action a corresponds to λ -probabilities P λ ( a ) from Eq.”
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 1× · see citing works
“We compare DORA against standard prompting baselines, including zero-shot, chain-of-thought (CoT) (Wei et al., 2022), tree-of-thought (ToT) (Yao et al., 2023), ReAct (Yao et al., 2022), and exploration-based prompt (§C.2).”
Unpacking the exploration–exploitation tradeoff: A synthesis of human and animal literatures.
Mehlhorn, K., Newell, B. R., Todd, P. M., Lee, M. D., Morgan, K. V., Braithwaite, V. A., Hausmann, D., Fiedler, K., & González, C. (2015) · Decision · doi:10.1037/dec0000033 · Cited by 383 · cited 1× · see citing works
“Multi-Armed Bandits (MAB) provide a classical testbed for studying the exploration-exploitation trade-off (Mehlhorn et al., 2015; Sutton et al., 1998).”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.00909 · Cited by 366 · cited 1× · see citing works
“The final action a ∗ t is sampled via a ∗ t ∼ Categorical ( p λ ) , where the likelihood of each action a corresponds to λ -probabilities P λ ( a ) from Eq.”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304 · cited 2× · see citing works
“While this translation allows for LLMs to work zero-shot (Huang et al., 2022), the text-based nature of this action space means that to properly explore, LLM agents must sample diverse sequences of tokens which comprise the action.”
The effect of sampling temperature on problem solving in large language models
Renze, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.432 · Cited by 212 · see citing works
TextWorld: A Learning Environment for Text-Based Games
Côté, M.-A., Kádár, Á., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J. A. Y., Hausknecht, M., Asri, L. E., Adada, M., et al. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-24337-1_3 · Cited by 210 · cited 1× · see citing works
“TextWorld uses CookingWorld instances (Cˆot´e et al., 2018), in which the agent follows recipes by locating and processing ingredients as task complexity increases.”
A Thorough Examination of Decoding Methods in the Era of LLMs
Shi, C., Yang, H., Cai, D., Zhang, Z., Wang, Y., Yang, Y.-J., & Lam, W. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2402.06925 · Cited by 164 · cited 1×
“Furthermore, Shi et al. (2024) found that increasing sampling temperature also reduces task accuracy compared to deterministic decoding.”
Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM DiversityPreprint
Zhang, J.-Y., Yu, S., Chong, D., Sicilia, A. B., Tomz, M., Manning, C. D., & Shi, W.-Y. (2025) · arXiv.org · doi:10.48550/arxiv.2510.01171 · Cited by 144 · see citing works
Interactive Fiction Games: A Colossal Adventure
Hausknecht, M., Ammanabrolu, P., Côté, M.-A., & Yuan, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6297 · Cited by 122 · cited 1× · see citing works
“Jericho (Hausknecht et al., 2020) includes long-horizon, human-authored interactive fiction and represents the most challenging setting in the suite.”
Can large language models explore in-context?
Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., & Slivkins, A. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2403.15371 · Cited by 104 · cited 5×
“Krishnamurthy et al. (2024) evaluate 32 prompt designs to study LLM exploration in a Multi-Armed Bandit setting.”
InfluentialGameBench: Evaluating Strategic Reasoning Abilities of LLM AgentsPreprint
Costarelli, A., Allen, M., Hauksson, R., Sodunke, G., Hariharan, S., Cheng, C., Li, W. (., Clymer, J., & Yadav, A. (2024) · arXiv.org · doi:10.48550/arxiv.2406.06613 · Cited by 101 · cited 1× · see citing works
“…LLMs have led to their widespread adoption as decision-making agents across diverse domains, including recommendation systems (Zhu et al., 2025), scientific discovery (Ren et al., 2025), game-playing (Costarelli et al., 2024), healthcare (Gebreab et al., 2024), and robotics (Kim et al., 2024).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020) · arXiv · doi:10.48550/arxiv.2010.03768 · Cited by 80 · cited 1× · see citing works
“AlfWorld (Shridhar et al., 2020) consists of text-based household tasks with sparse feedback, no intermediate rewards, and the constraint that the agent can hold only one object at a time.”
OpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62 · cited 1× · see citing works
“…LLMs have led to their widespread adoption as decision-making agents across diverse domains, including recommendation systems (Zhu et al., 2025), scientific discovery (Ren et al., 2025), game-playing (Costarelli et al., 2024), healthcare (Gebreab et al., 2024), and robotics (Kim et al., 2024).”
LLM-Based Framework for Administrative Task Automation in Healthcare
Gebreab, S. A., Salah, K., Jayaraman, R., Rehman, M. H. U., & Ellaham, S. (2024) · International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs60797.2024.10527275 · Cited by 56 · cited 1× · see citing works
“…LLMs have led to their widespread adoption as decision-making agents across diverse domains, including recommendation systems (Zhu et al., 2025), scientific discovery (Ren et al., 2025), game-playing (Costarelli et al., 2024), healthcare (Gebreab et al., 2024), and robotics (Kim et al., 2024).”
A Thorough Examination of Decoding Methods in the Era of LLMs
Shi, C., Yang, H., Cai, D., Zhang, Z., Wang, Y., Yang, Y., & Lam, W. P. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.489 · Cited by 55 · see citing works
Is Temperature the Creativity Parameter of Large Language Models?
Peeperkorn, M., Kouwenhoven, T., Brown, D., & Jordanous, A. (2024) · arXiv · doi:10.48550/arxiv.2405.00492 · Cited by 52 · cited 3×
“Unfortunately, as seen in narrative generation (Peeperkorn et al., 2024), sampling diverse sequences is extremely challenging and LLMs continue to generate high-likelihood and uncreative sequences.”
InfluentialUncertainty Estimation in Autoregressive Structured Prediction
Gales, M., & Malinin, A. (2021) · Apollo - University of Cambridge Repository · doi:10.17863/cam.63497 · Cited by 48 · cited 1× · see citing works
“Classic reinforcement learning problems generally require higher initial exploration to build epistemic knowledge, followed by exploitation based on that knowledge.”
LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
Schmied, T., Bornschein, J., Grau-Moya, J., Wulfmeier, M., & Pascanu, R. (2025) · arXiv.org · doi:10.48550/arxiv.2504.16078 · Cited by 47
ScienceWorld: Is your Agent Smarter than a 5th Grader?
Wang, R., Jansen, P., Côté, M.-A., & Ammanabrolu, P. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.775 · Cited by 45 · see citing works
ScienceWorld: Is your Agent Smarter than a 5th Grader?
Wang, R., Jansen, P., Côté, M.-A., & Ammanabrolu, P. (2022) · arXiv · doi:10.48550/arxiv.2203.07540 · Cited by 45 · cited 1×
“ScienceWorld (Wang et al., 2022) contains open-ended scientific tasks with multiple valid solution paths and occasional irreversible states.”
EVOLvE: Evaluating and Optimizing LLMs For Exploration
Nie, A., Su, Y., Chang, B., Lee, J. N., Chi, E. H., Le, Q. V., & Chen, M.-M. (2024) · arXiv.org · doi:10.48550/arxiv.2410.06238 · Cited by 37 · cited 1×
“Nie et al. (2025) , acting under the history-conditioned autoregressive policy π ( a | h t , τ ; θ ) with parameters θ and temperature τ .”
Controlling Large Language Model Agents with Entropic Activation Steering
Rahn, N., D'Oro, P., & Bellemare, M. G. (2024) · arXiv.org · doi:10.48550/arxiv.2406.00244 · Cited by 32 · cited 1×
“Our approach separates the processes of generating candidates and sampling them to ensure that we have highly diverse but also correct actions, which can also be explicitly controlled through our exploration parameter λ .”
Recommender Systems Meet Large Language Model Agents: A Survey
Zhu, X., Wang, Y., Gao, H., Xu, W., Wang, C., Liu, Z., Wang, K., Jin, M., Pang, L. X., Weng, Q., et al. (2025) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000050 · Cited by 21 · see citing works
TextWorldExpress: Simulating Text Games at One Million Steps Per Second
Jansen, P. A., & Côté, M.-A. (2022) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2208.01174 · Cited by 16
Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators
Mahaut, M., Aina, L., Czarnowska, P., Hardalov, M., Müller, T., & Màrquez, L. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.250 · Cited by 15 · cited 2× · see citing works
“A higher mean log-probability naturally indicates that the model assigns greater likelihood to each token in the sequence (Mahaut et al., 2024), while a lower variance shows better internal consistency across tokens (Gong et al., 2019).”
TALES: Text Adventure Learning Environment SuitePreprint
Cui, C., Yuan, X.-D., Xiao, Z., Ammanabrolu, P., & Côté, M.-A. (2025) · arXiv.org · doi:10.48550/arxiv.2504.14128 · Cited by 14 · cited 3× · see citing works
“LLMs tend to favor a narrow set of familiar responses over other plausible alternatives (Zhang et al., 2025); however, when explicitly prompted to generate multiple candidates, they can help recover some of this latent diversity (Zhang et al., 2025).”
InfluentialCan large language models explore in-context?
Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., & Slivkins, A. (2024) · neural information processing systems · doi:10.52202/079017-3818 · Cited by 10
Sentence-wise Smooth Regularization for Sequence to Sequence Learning
Gong, C., Tan, X., He, D., & Qin, T. (2018) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016449 · Cited by 9 · cited 2×
“A higher mean log-probability naturally indicates that the model assigns greater likelihood to each token in the sequence (Mahaut et al., 2024), while a lower variance shows better internal consistency across tokens (Gong et al., 2019).”
Expanding LLM Agent Boundaries with Strategy-Guided Exploration
Szot, A., Kirchhof, M., Attia, O., & Toshev, A. (2026) · arXiv.org · doi:10.48550/arxiv.2603.02045 · Cited by 8 · cited 1×
“Despite the importance, exploration is an area where LLM agents struggle (Krishnamurthy et al., 2024; Szot et al., 2026).”
Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
Ren, S., Xie, C., Jian, P., Ren, Z., Leng, C., & Zhang, J. (2025) · arXiv · doi:10.48550/arxiv.2503.24047 · Cited by 7 · cited 1× · see citing works
“…LLMs have led to their widespread adoption as decision-making agents across diverse domains, including recommendation systems (Zhu et al., 2025), scientific discovery (Ren et al., 2025), game-playing (Costarelli et al., 2024), healthcare (Gebreab et al., 2024), and robotics (Kim et al., 2024).”
OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents
Yu, S., Wu, J.-D., Li, X.-T., Kuang, N. L., Zhou, S., Yu, T., Han, J.-W., Shang, J.-B., & McAuley, J. J. (2026) · arXiv.org · doi:10.48550/arxiv.2605.11169 · Cited by 6
TextWorldExpress: Simulating Text Games at One Million Steps Per Second
Jansen, P., & Côté, M.-A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.20 · Cited by 5 · see citing works
Turning up the heat: Min-p sampling for creative and coherent llm outputs
Nguyen, M. N., Baker, A., Neo, C., Roush, A., Kirsch, A., & Shwartz-Ziv, R. (2024) · arXiv · doi:10.48550/arxiv.2407.01082 · Cited by 1 · cited 1× · see citing works
“…2018), top-p (nucleus) sampling (Holtzman et al., 2019) selects the smallest set of tokens whose cumulative probability exceeds a threshold p (Holtzman et al., 2019), and min-p sampling truncates low-probability continuations relative to the model’s highest-probability token (Nguyen et al., 2024).”
Training a Generally Curious Agent
Tajwar, F., Jiang, Y., Thankaraj, A., Rahman, S. S., Kolter, J. Z., Schneider, J., & Salakhutdinov, R. (2025) · arXiv · doi:10.48550/arxiv.2502.17543
Control the Temperature: Selective Sampling for Diverse and High-Quality LLM OutputsPreprint
Troshin, S., Mohammed, W. A.-A., Meng, Y., Monz, C., Fokkens, A. S., & Niculae, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.01218 · cited 4×
“…to improve LLM agent exploration using pure sampling (Fan et al., 2018), pure prompting (Yao et al., 2023) or through controlled exploration (Troshin et al., 2025), our approach uses aspects of all three approaches to explicitly control the exploration/exploitation tradeoff, generate a…”
Influential
Flagged first, then by citations
Research works printed but not linked: 2
- Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In COLT, 2012.
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygGQyrFvH.
Cited by
Sorted by citations
MS-Eval: Benchmarking Sub-Agent Delegation and Orchestration in MCP-Intensive Working Scenarios
Ding, B., Wang, Q., Ruan, L., Lu, W., Ge, Q., Fang, R., & Lin, T. (2026) · arXiv · doi:10.48550/arxiv.2610.22869
- Authors
- Gurjar, P., Ishmam, M. F., & Marino, K.
- Affiliations
- University of Utah (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2604.17244
- OpenAlex
- W7155030667
- Cited by
- 0
- References
- 42
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- LLM agents, exploration, multi-armed bandits, React, loop prevention, temperature scaling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
