Simple statistical gradient-following algorithms for connectionist reinforcement learning
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The REINFORCE algorithm is a policy gradient method that adjusts parameters to favor actions that yield higher rewards and improve segmentation.
“The agent's policy is updated using the REINFORCE algorithm [20], a policy gradient method that adjusts its parameters to favor actions that yield higher rewards and improve segmentation.”
From Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback (2025) · cites this paper as [20]
Section 4 defines REINFORCE as a parameter-update algorithm and shows that its average update follows a direction of increasing expected reward. The paper does not establish that REINFORCE improves segmentation; that is a claim about the citing paper’s application, not a result shown by this source. (LLM, conf=0.88)
Experimental — accuracy not yet measuredSuch methods commonly use online policy optimization algorithms such as PPO, GRPO, REINFORCE, and DAPO.
“These methods often employ online policy optimization algorithms, including Proximal Policy Optimization (PPO) (Schulman et al., 2017), Group Relative Policy Optimization (GRPO) (Shao et al., 2024a), REINFORCE (Williams, 1992), and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) (Yu et al., 2025).”
From Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback (2025) · cites this paper as Williams, 1992
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 22
- Barto, A.G. & Anandan, P. (1985). Pattern recognizing stochastic learning automata. IEEE Transactions on Systems, Man, and Cybernetics, 15, 360-374.
- Barto, A.G. & Anderson, C.W. (1985). Structural learning in connectionist systems. Proceedings of the Seventh Annual Conference of the Cognitive Science Society, (pp. 43-53).
- Barto, A.G., Sutton, R.S., & Anderson, C.W. (1983). Neuronlike elements that can solve difficult learning con- trol problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 835-846.
- Barto, A.G., & Jordan, M.I. (1987). Gradient following without back-propagation in layered networks. Proceedings of the First Annual International Conference on Neural Networks, Vol. II (pp. 629-636). San Diego, CA.
- Barto, A.G., Sutton, R.S., &Watkins, C.J.C.H. (1990). Learning and sequential decision making. In: M. Gabriel & J.W. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press.
- Dayan, P. (1990). Reinforcement comparison. In D.S. Touretzky, J.L. Elman, T.J. Sejnowski, & G.E. Hinton (Eds.), Proceedings of the 1990 Connectionist Models Summer School (pp. 45-51). San Mateo, CA: Morgan Kaufmann.
- Hinton, G.E. & Sejnowski, T.J. (1986). Learning and relearning in Boltzmann machines. In: D.E. Rumelhart & J.L. McClelland, (Eds.), Parallel distributed processing: Explorations in the microstructure of cognition. Vol. 1: Foundations. Cambridge, MA: MIT Press.
- Jordan, M.I. & Rumelhart, D.E. (1990). Forward models: supervised learning with a distal teacher. (Occasional Paper -40). Cambridge, MA: Massachusetts Institute of Technology, Center for Cognitive Science.
- leCun, Y. (1985). Une procedure d'apprentissage pour resau a sequil assymetrique [A learning procedure for asymmetric threshold networks]. Proceedings of Cognitiva, 85, 599-604.
- Munro, P. (1987). A dual back-propagation scheme for scalar reward learning. Proceedings of the Ninth Annual Conference of the Cognitive Science Society (pp. 165-176). Seattle, WA.
- Narendra, K.S. & Thathatchar, M.A.L. (1989). Learning Automata: An introduction. Englewood Cliffs, NJ: Prentice Hall.
- Narendra, K.S. & Wheeler, R.M., Jr. (1983). An N-player sequential stochastic game with identical payoffs. IEEE Transactions on Systems, Man, and Cybernetics, 13, 1154-1158.
- Parker, D.B. (1985). Learning-logic. (Technical Report TR-47). Cambridge, MA: Massachusetts Institute of Tech- nology, Center for Computational Research in Economics and Management Science.
- Rohatgi, V.K. (1976) An introduction to probability theory and mathematical statistics. New York: Wiley.
- Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986). Learning internal representations by error propaga- tion. In: D.E. Rumelhart & J.L. McClelland, (Eds.), Parallel distributed processing: Explorations in the micro- structure of cognition. Vol. 1: Foundations. Cambridge: MIT Press.
- Sutton, R.S. (1984). Temporal credit assignment in reinforcement learning. Ph.D. Dissertation, Dept. of Com- puter and Information Science, University of Massachusetts, Amherst, MA.
- Thathatchar, M.A.L. & Sastry, P.S. (1985). A new approach to the design of reinforcement schemes for learning automata. IEEE Transactions on Systems, Man, and Cybernetics, 15, 168-175.
- Wheeler, R.M., Jr. & Narendra K.S. (1986). Decentralized learning in finite Markov chains. IEEE Transactions on Automatic Control, 31, 519-526.
- Watkins, C.J.C.H. (1989). Learning from delayed rewards. Ph.D. Dissertation, Cambridge University, Cambridge, England.
- Werbos, P.J. (1974). Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. Dissertation, Harvard University, Cambridge, MA.
- Williams, R.J. (1987b). A class of gradient-estimating algorithms for reinforcement learning in neural networks. Proceedings of the First Annual International Conference on Neural Networks, Vol. II (pp. 601-608). San Diego, CA.
- Williams, R.J. (1988a). On the use of backpropagation in associative reinforcement learning. Proceedings of the Second Annual International Conference on Neural Networks, Vol. I (pp. 263-270). San Diego, CA.
Cited by
Sorted by citations
An Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609 · see citing works
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124 · see citing works
“Return normalization with a denominator limit overcomes these challenges, exploring rapidly under sparse rewards and converging to high performance across diverse domains.”
BackgroundDevice-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Accelerators
Jiang, W., Lou, Q., Yan, Z., Yang, L., Hu, J., Hu, X. S., & Shi, Y. (2021) · IEEE Transactions on Computers · doi:10.1109/tc.2020.2991575 · Cited by 96 · see citing works
Noise-resilient variational hybrid quantum-classical optimization
Gentini, L., Cuccoli, A., Pirandola, S., Verrucchi, P., & Banchi, L. (2020) · Physical Review A · doi:10.1103/physreva.102.052414 · Cited by 46 · see citing works
DayDreamer: World Models for Physical Robot Learning
Wu, P., Escontrela, A., Hafner, D., Goldberg, K., & Abbeel, P. (2022) · arXiv · doi:10.48550/arxiv.2206.14176 · Cited by 76 · see citing works
“Different gradient estimators are available for computing the policy gradient for optimizing the actor, such as Reinforce (Williams, 1992) and the reparameterization trick (Kingma and Welling, 2013; Rezende et al., 2014) that directly backpropagates return gradients through the differentiable…”
BackgroundMethodInfluentialIn-context Reinforcement Learning with Algorithm Distillation
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14215 · Cited by 14 · see citing works
“RL algorithms broadly fall into two categories: on-policy algorithms (Williams, 1992) where the agent directly maximizes a Monte-Carlo estimate of the total returns or off-policy (Mnih et al.”
BackgroundLearning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 12 · see citing works
“We then use the REINFORCE algorithm [39] to optimize the VLA policy, treating the rollouts from the simulator as the on-policy rollouts from the real environment and use the learned reward model to predict rewards from simulated rollouts.”
MethodWang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01939 · Cited by 14
“Online methods, such as PPO (Schulman et al., 2017), GRPO (Shao et al., 2024), and REIN-FORCE (Williams, 1992), generate responses during training and receive real-time feedback.”
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025) · arXiv · doi:10.48550/arxiv.2504.13837 · Cited by 3 · see citing works
“PPO and its variants belong to the policy gradient class of reinforcement learning (Williams, 1992; Sutton et al., 1998).”
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3 · see citing works
“RL algorithms such as Q-learning (Watkins, 1989) or policy gradient (Williams, 1992) account for feedback during training.”
MethodIn Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Tan, Z., Yan, J., Hsu, I.-H., Han, R., Wang, Z., Le, L. T., Song, Y., Chen, Y., Palangi, H., Lee, G., et al. (2025) · arXiv
“The reranker is fine-tuned using the REIN-FORCE algorithm (Williams, 1992) to optimize its relevance predictions based on these binary rewards with the following formulation: where 𝑅 is the reward ( + 1 or − 1), 𝑏 is a baseline value set as a hyperparameter, and 𝜙 denotes the weights of the…”
LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
Zhang, G., Meng, F., Wan, G., Li, Z., Wang, K., Yin, Z., Bai, L., & Yan, S. (2025) · arXiv · doi:10.48550/arxiv.2509.24771
“…and CoT (Wei et al., 2023); • Reinforcement Learning : (1) self-rewarding methods, including Self-Rewarding (Yuan et al., 2025) and Genius (Xu et al., 2025a), and (2) verifiable reward methods, including GRPO (DeepSeek-AI et al., 2025), Reinforce (Williams, 1992), and Reinforce++ (Hu et al., 2025).”
Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates
Li, Y., Lin, Z., Deng, A., Zhang, X., He, Y., Ji, S., Cao, T., & Hooi, B. (2026) · arXiv · doi:10.48550/arxiv.2601.18510
“A fundamental approach to optimize this objective is the Policy Gradient method (Williams, 1992; Sutton et al., 1999).”
Improving Controller Generalization with Dimensionless Markov Decision Processes
Charvet, V., Stein, S., & Murray-Smith, R. (2025) · arXiv · doi:10.48550/arxiv.2504.10006
“…optimize the parameters of the policy, different algorithms use different gradient of return estimation schemes like reparameterization trick (Kingma and Welling 2014; Xu et al. 2019) or likelihood ratio (Williams 1992) to backprop-agate derivatives through sampling the model (Mohamed et al. 2020).”
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Zhang, X., Zhang, Y., Sun, H., Feng, K., Lu, C., Yang, C., & Meng, H. (2025) · arXiv · doi:10.48550/arxiv.2506.03106
“( iii ) CoT Critique : A model-generated critique providing step-by-step reasoning to justify correctness or incorrectness, concluding with a binary correctness indication.”
Some Considerations on Learning to Explore via Meta-Reinforcement Learning
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y., Wu, Y., Abbeel, P., & Sutskever, I. (2018) · arXiv · doi:10.48550/arxiv.1803.01118 · Cited by 74 · see citing works
- Authors
- Williams, R. J.
- Affiliations
- Northeastern University (US)
- Corresponding author
- Williams, Ronald J.
- Venue
- Machine Learning
- Published
- 1992-05
- Volume
- 8
- Issue
- 3-4
- Pages
- 229-256
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W4241811150
- Cited by
- 817
- References
- 0
- Citations by year
- 2026: 296 · 2025: 264 · 2024: 83 · 2023: 30 · 2022: 29 · 2021: 34 · 2020: 26 · 2019: 29 · 2018: 8 · 2017: 8 · 2015: 2 · 2012: 1
- FWCI
- 0.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Machine Learning (publishedVersion)
- Funders
- National Science Foundation (8921275)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- REINFORCE algorithm, delayed reinforcement, backpropagation, weight adjustment, asymptotic behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
