DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS

In the paper

“Including these covariates accounted for quadratic and cubic deviations in the data, but only increased the average percentage of variance explained by 1.8% (range, 0.31 to 3.81%) and 1.5% (range, 0.13 to 2.63%) for the reinforcer frequency and reinforcer magnitude conditions, respectively.”

In the paper

“We have found that local linear weightings of past reinforcers predict the choice behavior of monkeys performing a matching task, with weights similar to those suggested by reinforcer rate estimation models, but we extend this to show that accounting for recent choices also is required for accurate predictions by models of this type.”

In the paper

“There was a rapid acquisition period following the transition, and the curves diverged for the two possible posttransition ratios.”

In the paper

“For both monkeys, the coefficients for reinforcer history are similar in shape for both the probability and magnitude conditions, although the coefficients are larger in the probability conditions.”

In the paper

“Positive coefficients for the trials further in the past than the last trial mean that choosing a particular alternative some time in the past increases the likelihood of choosing it again, independent of the effects due to past reinforcers. Alone, these coefficients would produce a tendency to persist on an alternative.”

In the paper

“Monkeys adjusted their choice behavior in response to abrupt reversals in the prevailing reinforcer conditions, reaching an average steady-state of responding after obtaining relatively few reinforcers under the new schedule conditions.”

In the paper

“Almost as soon as one of the alternatives was preferred, the runlengths for the nonpreferred alternative became very short, typically one trial (mean across all preferences was 1.05 trials with a standard deviation of 0.09).”

In the paper

“For both monkeys, the autocorrelations stayed mostly within these confidence intervals, indicating that the residuals were largely unstructured.”

In the paper

“The model predictions captured the steady-state average as well as the dynamics of the transitions between blocks.”

In the paper

“By accounting for the effects of past choices we showed that our monkeys have a short-term tendency to alternate as well as a longer-term tendency to persist on the more frequently chosen alternative.”

In the paper

“The pattern of results for reinforcer history is similar for both animals; the most recent reinforcers strongly increased the odds of choosing the reinforced alternative again, and this influence decayed as reinforcers receded in time.”

In the paper

“The actual runlength data systematically deviate from and hence reject this model”

In the paper

“There was little or no spatial bias towards either alternative, which was consistent with our steady-state analysis using the generalized matching law.”

In the paper

“The negative coefficient for the last trial means that choosing a particular alternative decreased the odds of choosing that alternative again on the next trial, independent of any effects due to reinforcer history. Alone, this coefficient would produce a tendency to alternate.”

In the paper

“Instead, they show that the monkeys tend to stay at the richer alternative, switching briefly to sample the leaner alternative.”

In the paper

“The data analyzed were 67,827 completed trials from Monkey H and 47,160 completed trials from Monkey B.”

In the paper

“The strict matching law equates these two quantities; the relative number of choices matches the relative reinforcer frequency (Herrnstein, 1961).”

What the sources say

• Not checked yetcites Herrnstein, 1961 source

In the paper

“The mean number of trials per block observed was 121 trials for Monkey H and 125 trials for Monkey B, after excluding aborted trials.”

In the paper

“The generalized matching law for reinforcer frequency and magnitude assumes that these attributes have independent effects (Baum & Rachlin, 1969), but allows for differential levels of control by each variable;”

What the sources say

• Not checked yetcites Baum & Rachlin, 1969

In the paper

“Single neurons in a number of cortical areas have activity correlated with reinforcer frequency (Barraclough, Conroy, & Lee, 2004; Platt & Glimcher, 1999; Sugrue et al., 2004) and magnitude (Platt & Glimcher, 1999).”

What the sources say

• Not checked yetcites Barraclough, Conroy, & Lee, 2004 source
• Not checked yetcites Platt & Glimcher, 1999 source
• Not checked yetcites Sugrue et al., 2004 source

In the paper

“All the data reported were collected after at least 3 months of training on the choice task.”

In the paper

“If the monkeys chose in a purely probabilistic manner by flipping a weighted coin on each trial, with the weight dictated by the degree of preference (e.g., a 2∶1 preference gives a two-thirds probability of choosing the preferred alternative on each trial), the runlengths would be distributed geometrically with a mean equal to the reciprocal of the coin weight.”

What the sources say

• Not checked yetcites Houston & Sumida, 1987

In the paper

“This indicates that the generalized matching law is a reasonable description of our data, even though it typically is used to describe data after many sessions, rather than trials, of training under the same reinforcer conditions.”

In the paper

“The present results suggest that behavioral models incorporating dependence on both past reinforcers and choices may provide more accurate predictions, which will be essential for correlation with neural activity.”

In the paper

“The generalized matching law describes an enormous amount of data collected from many species (de Villiers, 1977), and allows one to predict average choice behavior under different conditions, such as for arbitrary combinations of reinforcer frequency and magnitude.”

What the sources say

• Not checked yetcites de Villiers, 1977

In the paper

“The monkeys rarely aborted trials (7% and 6% of trials for Monkey H and Monkey B, respectively), and these trials were excluded from our analyses.”

In the paper

“Although the generalized matching law fits the steady-state behavior that we measured, it only describes average choice. It is silent regarding the sequences of choices underlying these averages.”

In the paper

“Many of the more plausible—in the sense that animals may actually implement them—theoretical models incorporate some form of reinforcer rate estimation using linear weightings of past reinforcers (e.g., Killeen, 1981; Luce, 1959).”

What the sources say

• Not checked yetcites Killeen, 1981
• Not checked yetcites Luce, 1959

In the paper

“The differences in AIC values in Table 3 represent the degree of evidence in favor of the best-fitting model, and the simpler models were selected to give a sense of the contribution of each model component. The larger the difference the less plausible a model is compared to the best model; values greater than 10 on this scale provide strong support for the model with the smallest AIC value (Burnh”

What the sources say

• Not checked yetcites Burnham & Anderson, 1998 source

In the paper

“In the first stage, the subjective value of each alternative is estimated from past experience, and in the second stage a decision is formed by comparing these estimates.”

In the paper

“This model is inadequate, and clearly indicates the need to incorporate the effects of past choices to account for the response-by-response structure in our task.”

In the paper

“Because the model is linear in the parameters, the coefficients weighting past reinforcers can be rescaled so they sum to 1.”

In the paper

“As we have sketched it, the first stage includes the effects of past reinforcers, which we think of as providing estimates of the rate of reinforcement.”

In the paper

“This pattern comes about because larger or more frequent reinforcers increase the likelihood of repeating choices to the alternative that yields them. Momentary decreases in rich reinforcers eventually cause a switch to the lean alternative, followed by a tendency to switch immediately back to the rich alternative due to longer-term choice history effects.”

In the paper

“This estimate has units of reinforcer magnitude per trial, and is passed to the second stage to form a choice probability.”

In the paper

“Baum, Schwendiman, and Bell (1999) observed this pattern of behavior in pigeons under concurrent VI VI schedules, terming it “fix-and-sample.””

What the sources say

• Not checked yetcites Baum, Schwendiman, and Bell (1999)

In the paper

“In addition, we may have observed stronger undermatching because we reversed the reinforcement contingencies many times during a single session.”

In the paper

“Moreover, Baum et al. showed that the sequential structure varied depending on presence or absence of a changeover delay; pigeons adjust their patterns of choice behavior to respect the fluctuations in local probability of reinforcement.”

What the sources say

• Not checked yetcites Baum et al.

In the paper

“Many theoretical models of foraging and repeated choice behavior assume that the rate of prey encounter or reinforcement is a fundamental variable that animals use to achieve the computational goals of maximizing fitness or reinforcement (Stephens & Krebs, 1986).”

What the sources say

• Not checked yetcites Stephens & Krebs, 1986

In the paper

“These response-by-response dependencies could have arisen from the effects of past reinforcers and/or past choices, but how this occurs cannot be revealed through sequential statistics like runlengths because they do not take into account when reinforcers were delivered.”

In the paper

“It is common to split the decision process in this manner (e.g., Davis, Staddon, Machado, & Palmer, 1993; Houston, Kacelnik, & McNamara, 1982; Luce, 1959).”

What the sources say

• Not checked yetcites Davis, Staddon, Machado, & Palmer, 1993
• Not checked yetcites Houston, Kacelnik, & McNamara, 1982
• Not checked yetcites Luce, 1959

In the paper

“Relative to Monkey H, the coefficients for Monkey B are initially larger, but also decay more quickly, indicating that recent reinforcers would shift preference more strongly, but that their effects are more local in time.”

In the paper

“Rate estimation is usually implemented by a weighted average with exponential decay (Killeen, 1981, 1994), which captures the intuitive notion that because natural environments are nonstationary, recent events should weigh more heavily in memory than long-past events (Cowie, 1977).”

What the sources say

• Not checked yetcites Killeen, 1981, 1994
• Not checked yetcites Cowie, 1977 source

In the paper

“Houston and McNamara (1981) prove that reinforcer rate is maximized by repeating a deterministic sequence; an optimal subject would choose the rich alternative a fixed number of times (dictated by the particular schedules) followed by a single choice of the lean alternative (see also Staddon et al., 1981).”

What the sources say

• Not checked yetcites Houston and McNamara (1981)
• Not checked yetcites Staddon et al., 1981 source

In the paper

“Because the probability of reinforcement grows geometrically with time or responses spent away from the lean alternative in tasks like the one we used, it eventually pays to choose that alternative, but because its probability is reset to its initial low value once chosen it is never worth staying more than one trial.”

In the paper

“For example, the dopaminergic neurons in the substantia nigra pars compacta (SNC) and the ventral tegmental area (VTA) appear to be important for estimating reinforcement rate.”

In the paper

“characterizing the effects of past reinforcers and choices allows a finer separation of these effects at the neural level.”

In the paper

“They showed that the number of action potentials discharged by individual SNC neurons is proportional to the magnitude of the difference between the reinforcer expected (vi−1) and the reinforcer actually obtained (ri), thereby lending strong support to the hypothesis that dopamine neurons underlie the computation of prediction error.”

What the sources say

• Not checked yetcites Bayer and Glimcher (2005) source

In the paper

“This idea is consistent with evidence that frontal cortex and the basal ganglia—which are densely interconnected—are important for the expression and learning of motor sequences (Hikosaka et al., 1999; Tanji, 2001; Tanji & Hoshi, 2001).”

What the sources say

• Not checked yetcites Hikosaka et al., 1999 source
• Not checked yetcites Tanji, 2001
• Not checked yetcites Tanji & Hoshi, 2001

In the paper

“These neurons send dense projections to the prefrontal cortex and the striatum, and current evidence suggests that these areas may use a prediction error to estimate reinforcer rate (Barraclough et al., 2004; Haruno et al., 2004; O'Doherty et al., 2004).”

What the sources say

• Not checked yetcites Barraclough et al., 2004 source
• Not checked yetcites Haruno et al., 2004 source
• Not checked yetcites O'Doherty et al., 2004 source

In the paper

“Experiments with monkeys and humans have shown that these neurons encode a prediction error, the difference between the reinforcer expected and the reinforcer obtained within a trial (O'Doherty et al., 2004; Schultz, 1998).”

What the sources say

• Not checked yetcites O'Doherty et al., 2004 source
• Not checked yetcites Schultz, 1998 source

In the paper

“Appropriate use of recent memory for past responses can generate tendencies to shift away from, or to persist on, recently chosen alternatives (negative recency and positive recency), and is necessary for producing patterned sequences of responses.”

In the paper

“The covariation of neural activity in the parietal cortex with behaviorally relevant variables likely reflects computations carried out in other brain areas.”

In the paper

“Bayer and Glimcher (2005) linked SNC activity to reinforcement-rate estimation using the exponentially weighted average described above.”

What the sources say

• Not checked yetcites Bayer and Glimcher (2005) source

In the paper

“For example, Devenport and Devenport (1994) propose a temporal weighting rule that weights past reinforcers hyperbolically (with respect to units of time).”

What the sources say

• Not checked yetcites Devenport and Devenport (1994)

In the paper

“The decision stage of many choice models uses rules including simply selecting the most valuable alternative (Herrnstein & Vaughan, 1980), matching choice proportions to the local rates of reinforcement (Kacelnik, Krebs, & Ens, 1987), and the logistic function we have used, which is commonly referred to as the softmax (Sutton & Barto, 1998) or logit (Camerer & Ho, 1998) decision rule.”

What the sources say

• Not checked yetcites Herrnstein & Vaughan, 1980
• Not checked yetcites Kacelnik, Krebs, & Ens, 1987
• Not checked yetcites Sutton & Barto, 1998

In the paper

“This captures the idea that one can only remember so much information, but it is distinct from what one does with that memory, which presumably depends on reinforcement contingencies and other factors (Glimcher, 2005; Neuringer, 2002).”

What the sources say

• Not checked yetcites Glimcher, 2005 source
• Not checked yetcites Neuringer, 2002 source

In the paper

“These variables must be represented in the nervous system if they are to affect choice”

In the paper

“Because the probability of reinforcement grows geometrically with time or responses spent away from the lean alternative in tasks like the one we used, it eventually pays to choose that alternative, but because its probability is reset to its initial low value once chosen it is never worth staying more than one trial.”

In the paper

“For an animal foraging for food, evaluating which patch to forage at is a complex combination of estimating the reinforcement rate at each patch, the likelihood that an alternative patch has recently become richer, the risk of predation, and so on.”