Temporal Difference Learning and TD-Gammon

Abstract

We provide an abstract, selectively u §ing the author's formulations: "The article presents a game-learning program called TD-GAMMON.TD-GAMMON is a neural network that trains itself to be an evaluation function for the game of backgammon by playing against itself and learning from the outcome.It was not developed to surpass all previous computer programs in backgammon; rather, its purpose was to explore some new ideas and approaches to traditional problems in reinforcement learning.The basic paradigm of reinforcement learning is as follows: the learning agent observes an input state or input pattern, it produces an output signal (most commonly thought of as an "action" or "control signal"), and then it receives a scalar "reward" or "reinforcement" feedback signal from the environment indicating how good or bad its output was.The goal of learning is to generate the optimal actions leading to maximal reward, which often is delayed (i.e., is given at the end of a long sequence of inputs and outputs).The learner then has to solve the temporal credit-assignment problem, i.e., it must figure out how to apportion credit and blarne to each of the inputs and outputs leading to the final reward signal.