Changes of mind in decision-making
Open accessAbstract
A decision is a commitment to a proposition or plan of action based on evidence and the expected costs and benefits associated with the outcome. Progress in a variety of fields has led to a quantitative understanding of the mechanisms that evaluate evidence and reach a decision. Several formalisms propose that a representation of noisy evidence is evaluated against a criterion to produce a decision. Without additional evidence, however, these formalisms fail to explain why a decision-maker would change their mind. Here we extend a model, developed to account for both the timing and the accuracy of the initial decision, to explain subsequent changes of mind. Subjects made decisions about a noisy visual stimulus, which they indicated by moving a handle. Although they received no additional information after initiating their movement, their hand trajectories betrayed a change of mind in some trials. We propose that noisy evidence is accumulated over time until it reaches a criterion level, or bound, which determines the initial decision, and that the brain exploits information that is in the processing pipeline when the initial decision is made to subsequently either reverse or reaffirm the initial decision. The model explains both the frequency of changes of mind as well as their dependence on both task difficulty and whether the initial decision was accurate or erroneous. The theoretical and experimental findings advance the understanding of decision-making to the highly flexible and cognitive acts of vacillation and self-correction.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper shows that subjects can change their mind after initiating a perceptual decision even though no new visual evidence is available, because late-arriving evidence in the processing pipeline can revise the initial choice. It extends bounded drift-diffusion models with a post-initiation change-of-mind bound and deadline to explain when and how often these reversals occur.
Key findings
- Changes of mind were reflected in curved hand trajectories and reliably improved accuracy for all three subjects by improving sensitivity to motion (p<0.006 for each subject).
- The initial choice depended on early stimulus fluctuations (average over first 150 ms positive, p<0.0001), while later fluctuations had little bearing on the initial choice; on change-of-mind trials, later motion energy supported the final, changed decision (p<0.0001).
- Non-decision time was estimated at 300–400 ms, and the model’s change bound required reversal of the accumulated evidence; the amount of evidence needed to change mind (BΔ) differed by ~30% across subjects.
- A model with a post-initiation change bound fit better than using all available information (p<0.003 for all subjects), capturing changes to correct and incorrect choices with R² = 0.63–0.85 and 0.76–0.99, respectively; changes were most frequent at intermediate motion strengths when the initial choice was erroneous.
Methods
- Three naïve participants judged left/right motion of a dynamic random dot stimulus and indicated choices by moving a handle to left/right targets; the motion stimulus was extinguished at movement initiation, so no new visual evidence was available during the movement.
- Subjects completed at least 500 training trials followed by 1500 test trials; hand trajectories were recorded at 1000 Hz, and a change of mind was detected if the area carved out by the hand opposite the final chosen target exceeded 0.1 cm².
- Analyses used bounded drift-diffusion modeling, motion-energy residuals to test which stimulus epochs supported initial versus changed choices, and fits of post-initiation change-bound/deadline parameters; a control experiment used an auditory beep to time movement initiation, with stimulus motion beginning 200–2000 ms before the beep (mean 440 ms).
Limitations
- Stated in the paper: the proposed mechanism cannot explain all changes of mind, such as corrections arising from confusion about stimulus-response associations, or changes requiring memory retrieval or incorporation of a new decision policy.
- Stated in the paper: the model is specific to situations with time pressure and an arm movement; ballistic actions such as eye movements preclude this kind of change-of-mind mechanism.
- Evident from the text: the main experiment used only three naïve subjects, and some supporting results (e.g., slower-instruction effects) are reported as “data not shown,” limiting generalizability and independent verification.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 24 cited sources checked · 0 supported · 0 flagged
- Findingown data
Although subjects received no additional information after initiating their movement, their hand trajectories betrayed a change of mind on some trials.
“Although they received no additional information after initiating their movement, their hand trajectories betrayed a change of mind on some trials.”
Quote in the paperSubstantial claim - Findingown data
Although no further visual information was available after movement initiation, the hand trajectories gave a clear indication that on some trials observers changed their mind.
“Although no further visual information was available after movement initiation, the hand trajectories (Fig. 1c) gave a clear indication that on some trials observers changed their mind.”
Quote in the paperSubstantial claim - Findingown data
The choice at initiation and reaction times as a function of task difficulty (coherence of dot motion) were explained by a bounded drift-diffusion model consistent with previous studies in humans and monkeys.
“The choice at initiation (initial hand trajectory) and reaction times as a function of task difficulty (coherence of dot motion) were explained by a bounded drift-diffusion model (Fig. 2, black curves) consistent with previous studies in humans and monkeys1,9,14.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [1] Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007) source
Not checked in the free run.
- cites [9] Palmer, J., Huk, A. C. & Shadlen, M. N. The effect of stimulus strength on the speed and accuracy of a perceptual decision. J. Vis. 5, 376–404 (2005) source
Not checked in the free run.
- cites [14] Smith, P. L. & Ratcliff, R. Psychology and neurobiology of simple decisions. Trends Neurosci. 27, 161–168 (2004) source
Not checked in the free run.
- Findingown data
Normally hand movements for easy trials (high coherence) were straight to the target.
“Normally hand movements for easy trials (high coherence) were straight to the target.”
Quote in the paperSubstantial claim - Findingown data
For all three subjects, the model fits imply that upon termination of the initial decision, the subjects set a new bound at a level that would necessitate a reversal of the sign of the accumulated evidence.
“For all three subjects, the model fits imply that upon termination of the initial decision, the subjects set a new bound at a level that would necessitate a reversal of the sign of the accumulated evidence.”
Quote in the paperSubstantial claim
- Findingown data
For each subject, the time point when the average came within 1 s.e. of zero provided an empirical estimate of non-decision time.
“For each subject, we identified the time point when the average came within 1 s.e. of zero (arrows), thus providing an empirical estimate of non-decision time.”
Quote in the paperSubstantial claim - Findingreasoning
Changes of mind are most frequent at intermediate motion strengths when the initial choice was erroneous.
“Notice that changes of mind are most frequent at intermediate motion strengths when the initial choice was erroneous.”
Quote in the paperSubstantial claim - Findingown data
When these processes act on the same data stream, they can lead to a change in a decision.
“What we have shown here is that when these processes act on the same data stream, they can lead to a change in a decision.”
Quote in the paperSubstantial claim - Findingown data
The motion energy in the later epoch was significantly more negative compared to the motion energy on the remaining trials (p<0.0001).
“The motion energy in this later epoch was significantly more negative compared to the motion energy on the remaining trials (p<0.0001).”
Quote in the paperSubstantial claim - Findingown data
In all cases, the existence of this change-bound led to a significant improvement in the fits, compared to using all the available information (i.e., no bound and choice based on sign of decision variable after tnd, p<0.003 for all subjects, likelihood ratio test).
“In all cases, the existence of this change-bound led to a significant improvement in the fits, compared to using all the available information (i.e., no bound and choice based on sign of decision variable after tnd, p<0.003 for all subjects, likelihood ratio test).”
Quote in the paperSubstantial claim - Findingown data
The sum of sensory and motor processing latencies, termed the non-decision time (tnd), is estimated to be 300-400 ms in our experiments.
“The sum of these latencies, termed the non-decision time (tnd), is estimated to be 300-400 ms in our experiments (Supplementary Table 1; Methods).”
Quote in the paperSubstantial claim - Findingown data
An unexpected virtue of bounded diffusion models demonstrated by our experiment is that a part of the data stream that is not used to make the decision can nonetheless support revision after a response is initiated.
“An unexpected virtue of such models demonstrated by our experiment is that a part of the data stream that is not used to make the decision can nonetheless support revision after a response is initiated.”
Quote in the paperSubstantial claim - Findingown data
The motion energy filtering induces a delay of 50-150 ms.
“Notice that the motion energy filtering induces a delay of 50-150 ms (Fig 3a, insert).”
Quote in the paperSubstantial claim - Findingcitation
The pattern of slower performance was explained by the same model with higher initiation bounds.
“The pattern was explained by the same model with higher initiation bounds9.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [9] Palmer, J., Huk, A. C. & Shadlen, M. N. The effect of stimulus strength on the speed and accuracy of a perceptual decision. J. Vis. 5, 376–404 (2005) source
Not checked in the free run.
- Findingown data
The model captures the complex dependence of post-initiation changes on both the motion strength and the initial decision (R2 = 0.63-0.85 and 0.76-0.99 for changes to correct and incorrect, respectively).
“The model captures the complex dependence of post-initiation changes on both the motion strength and the initial decision (R2 = 0.63-0.85 and 0.76-0.99 for changes to correct and incorrect, respectively).”
Quote in the paperSubstantial claim - Findingown data
The same data stream may be sampled at different moments to support different decisions, hence a change of mind.
“Our central finding is that the same data stream may be sampled at different moments to support different decisions, hence a change of mind.”
Quote in the paperSubstantial claim - Findingown data
These data also conform to a variant of the bounded accumulation mechanism with post initiation processing.
“These data also conform to a variant of the bounded accumulation mechanism with post initiation processing (see Methods and Supplementary Figures 2 & 3).”
Quote in the paperSubstantial claim - Findingown data
Subjects generated a curved hand path that initially was on course to reach one target, but changed during the movement to finish on the other target.
“That is subjects generated a curved hand path that initially was on course to reach one target, but changed during the movement to finish on the other target.”
Quote in the paperSubstantial claim - Findingown data
The pattern of motion evidence is different for the subset of trials in which there is a change of mind.
“The pattern is different for the subset of trials in which there is a change of mind.”
Quote in the paperSubstantial claim - Premisenone given
Without additional evidence, these formalisms fail to explain why a decision-maker would change her mind.
“Without additional evidence, however, these formalisms fail to explain why a decision-maker would change her mind.”
Quote in the paperSubstantial claim - Premisecitation
Psychophysical and neural data support a model termed drift diffusion, random walk or race in which a decision is made when the accumulated noisy evidence reaches a criterion level, termed a decision bound.
“Psychophysical and neural data1 support a model termed drift diffusion6, random walk5,7 or race8 in which a decision is made when the accumulated noisy evidence (decision variable) reaches a criterion level, termed a decision bound.”
Quote in the paperSubstantial claim0 of 4 sources checked- cites [1] Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007) source
Not checked in the free run.
- cites [6] Ratcliff, R. & Rouder, J. N. Modelling response times for two-choice decisions. Psychol. Sci. 9, 347–356 (1998) source
Not checked in the free run.
- cites [5] Laming, D. R. J. Information Theory of Choice-Reaction Times (Wiley, 1968)
Not checked in the free run.
- cites [7] Link, S. W. The relative judgment theory of two choice response time. J. Math. Psychol. 12, 114–135 (1975) source
Not checked in the free run.
- Premisecitation
In some instances, changes of mind can lead to the correction of an initial error.
“In some instances, such changes can lead to the correction of an initial error12,13.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [12] Rabbitt, P. & Vyas, S. Processing a display even after you make a response to it. How perceptual errors can be corrected. Q. J. Exp. Psychol. A 33, 223–239 (1981) source
Not checked in the free run.
- cites [13] Rabbitt, P. M. Error correction time without external error signals. Nature 212, 438 (1966) source
Not checked in the free run.
- Interpretationreasoning
Although the stimulus vanishes upon movement initiation, there is information in the processing pipeline that is potentially available to the decision-maker after movement initiation.
“Although the stimulus vanishes upon movement initiation, there is information in the processing pipeline that is potentially available to the decision-maker after movement initiation.”
Quote in the paperSubstantial claim - Interpretationreasoning
Because the subject must complete a hand movement, the optimal solution is likely to incorporate motor costs (energy) associated with larger corrections nearer the end of the movement.
“Because the subject must complete a hand movement, the optimal solution is likely to incorporate motor costs (energy) associated with larger corrections nearer the end of the movement.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The proposed mechanism cannot explain all changes of mind.
“Our proposed mechanism cannot explain all changes of mind.”
Quote in the paperNot a substantial claim - Interpretationnone given
In the most parsimonious model, once the initial bound has been reached and a decision made, evidence continues to accumulate until it either reaches a new 'change of mind' bound or a time-deadline terminates post-initiation processing.
“In this model, once the initial bound has been reached and a decision made, evidence continues to accumulate until it either reaches a new “change of mind” bound or a time-deadline terminates post-initiation processing.”
Quote in the paperSubstantial claim - Premisereasoning
Sensory and motor processing latencies ensure that not all of the information available from stimulus onset to movement initiation contributes to the decision.
“Sensory and motor processing latencies ensure that not all of the information available from stimulus onset to movement initiation contributes to the decision.”
Quote in the paperSubstantial claim - Interpretationcitation
The proposed mechanism would not explain corrections of initial errors that arise from confusion about stimulus-response associations.
“For example, it would not explain corrections of initial errors that arise from confusion about stimulus-response associations12.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [12] Rabbitt, P. & Vyas, S. Processing a display even after you make a response to it. How perceptual errors can be corrected. Q. J. Exp. Psychol. A 33, 223–239 (1981) source
Not checked in the free run.
- Interpretationown data
The deadline produced by the fit suggests that subjects avail themselves of most of the information in the processing pipeline.
“The deadline produced by the fit suggests that subjects avail themselves of most of the information in the processing pipeline.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Viewed as a decision process beginning at the initial decision bound, there is a higher probability of reaffirming the initial choice, because the accumulated evidence is far from the change of mind bound.
“Viewed as a decision process beginning at the initial decision bound, there is a higher probability of reaffirming the initial choice, because the accumulated evidence is far from the change of mind bound.”
Quote in the paperSubstantial claim - Interpretationown data
The analysis supports a non-decision time in which information from the stimulus arrives too late to affect an initial decision but is present to refine it after the brain has committed to a particular response and action.
“The analysis instead supports a non-decision time in which information from the stimulus arrives too late to affect an initial decision but is present to refine it after the brain has committed to a particular response and action.”
Quote in the paperSubstantial claim - Study design and conductreasoning
We were able to determine for each trial the choice at both initiation and termination of the movement.
“We were, therefore, able to determine for each trial the choice at both initiation and termination of the movement.”
Quote in the paperSubstantial claim - Premisecitation
Single unit recordings from the lateral intraparietal area (LIP) of the macaque in eye movement versions of this task suggest the non-decision time includes sensory and motor delays of around 220 ms and 80 ms, respectively.
“Single unit recordings from the lateral intraparietal area (LIP) of the macaque in eye movement versions of this task, suggest the non-decision time includes sensory and motor delays of around 220 ms and 80 ms, respectively15,16.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [15] Roitman, J. D. & Shadlen, M. N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 22, 9475–9489 (2002) source
Not checked in the free run.
- cites [16] Kiani, R., Hanks, T. D. & Shadlen, M. N. Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment. J. Neurosci. 28, 3017–3029 (2008) source
Not checked in the free run.
- Interpretationreasoning
A change of mind requires strong evidence in the short time available for post-initiation processing to move the accumulated evidence to the change-of-mind bound.
“A change of mind therefore requires strong evidence in the short time available for post-initiation processing to move the accumulated evidence to the change-of-mind bound.”
Quote in the paperSubstantial claim - Premisenone given
If the motion is very strong, initial errors are rare.
“However, if the motion is very strong, initial errors are rare.”
Quote in the paperSubstantial claim - Interpretationreasoning
The idea of motor costs can be captured by incorporating new bounds in the post-decision period to change or reaffirm an initial decision based on some criterion, thereby allowing changes to occur earlier in the movement.
“This idea can be captured by incorporating new bounds in the post-decision period to change or reaffirm an initial decision based on some criterion, thereby allowing changes to occur earlier in the movement.”
Quote in the paperSubstantial claim - Interpretationreasoning
The paradox of why information available to make a better decision fails to influence the initial decision is resolved if the decision-maker does not use all of the available evidence to make the initial choice but can tap into further information in the period between commitment to the initial response and termination of the movement.
“The paradox is resolved if the decision-maker does not use all of the available evidence to make the initial choice but can tap into further information in the period between commitment to the initial response and termination of the movement.”
Quote in the paperSubstantial claim
- May combine two claims. 15
- The quote only partly states this claim. 1
- Classified as probably not a substantial claim. 9
May combine two claims.
Findingown dataChanges of mind reliably improved accuracy for all three subjects by improving sensitivity to motion (p<0.006 for each subject).
“Changes of mind reliably improved accuracy (Fig. 2 top row, black and red circles for the initial and final choice, respectively) for all three subjects by improving sensitivity to motion (p<0.006 for each subject).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe amount of evidence required for a subject to change their mind (Table 1, BΔ) differed by ~30% across subjects, which explains the variation in the pattern of their changes.
“The amount of evidence required for a subject to change their mind (Table 1, BΔ) differed by ~30% across subjects, which explains the variation in the pattern of their changes.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThis optimal solution fails to explain the findings: it predicts too many changes and would defer them to the end of the evidence stream.
“This idea fails to explain our findings: it predicts too many changes and it would defer them to the end of the evidence stream, which is clearly not the case (Fig 1c).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe results of the experiment with an exogenous cue confirm that subjects base their initial choice on early evidence but can avail themselves of additional evidence in the processing pipeline to revise this choice.
“The results of this experiment, which are summarized in Supplementary Figures 1-3, confirm the finding that subjects base their initial choice on early evidence but can avail themselves of additional evidence in the processing pipeline to revise this choice.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn a control experiment with 100% coherent motion, the criterion detected no change-of-mind trials, suggesting that the detection strategy is conservative.
“In a control experiment with 100% coherent motion, this criterion detected no change-of-mind trials, suggesting that our detection strategy is conservative.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe stimulus fluctuations immediately after stimulus onset supported the initial choice (average over first 150 ms is positive, p<0.0001), whereas the fluctuations in the final few hundred ms had little bearing on the choice.
“The stimulus fluctuations immediately after stimulus onset supported the initial choice (Fig. 3a, left blue curve; average over first 150 ms is positive, p<0.0001), whereas the fluctuations in the final few hundred ms had little bearing on the choice.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown dataAn analysis of the motion evidence leading to the subjects' choices supports the hypothesis that unused information could be processed after the brain has committed to an initial choice.
“An analysis of the motion evidence leading to the subjects’ choices supports this hypothesis.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAlthough some changes of mind resulted in errors, the majority corrected an initial error.
“Although some changes of mind resulted in errors, the majority corrected an initial error.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataWhen two subjects were instructed to perform the reaction time experiment more slowly, their initial decisions were more accurate and there were fewer changes of mind.
“When two of our subjects were instructed to perform the reaction time experiment more slowly, their initial decisions were more accurate and there were fewer changes of mind (data not shown).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe early information from the stimulus provided weaker support for the initial choice and exhibited a negative trend near the time of initiation, in support of the final, changed decision.
“The early information from the stimulus provided weaker support for the initial choice (left red trace) and exhibited a negative trend near the time of initiation (right red trace), in support of the final, changed decision.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationSuch an accumulation process explains both the choice and accuracy of decisions over a range of difficulty levels as well as the time required to make the decision.
“Such an accumulation process explains both the choice and accuracy of decisions over a range of difficulty levels as well as the time required to make the decision9.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [9] Palmer, J., Huk, A. C. & Shadlen, M. N. The effect of stimulus strength on the speed and accuracy of a perceptual decision. J. Vis. 5, 376–404 (2005) source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThese models are naturally viewed as an extension of signal detection theory and Bayesian inference to streams of data over time.
“They are naturally viewed as an extension of signal detection theory and Bayesian inference to streams of data over time4,11.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [4] Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966)
Not checked in the free run.
- cites [11] Beck, J. M. et al. Probabilistic population codes for Bayesian decision making. Neuron 60, 1142–1152 (2008) source
Not checked in the free run.
May combine two claims.
Study design and conductown dataThe moving dots were extinguished as soon as the subjects initiated their movement, and hence subjects could not acquire new evidence during their movement.
“Critically, the moving dots were extinguished as soon as the subjects initiated their movement (Fig. 1b) and hence subjects could not acquire new evidence during their movement.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationSeveral formalisms propose that a representation of noisy evidence is evaluated against a criterion to produce a decision.
“Several formalisms propose that a representation of noisy evidence is evaluated against a criterion to produce a decision4-8.”
Quote in the paperSubstantial claim0 of 5 sources checked- cites [4] Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966)
Not checked in the free run.
- cites [5] Laming, D. R. J. Information Theory of Choice-Reaction Times (Wiley, 1968)
Not checked in the free run.
- cites [6] Ratcliff, R. & Rouder, J. N. Modelling response times for two-choice decisions. Psychol. Sci. 9, 347–356 (1998) source
Not checked in the free run.
- cites [7] Link, S. W. The relative judgment theory of two choice response time. J. Math. Psychol. 12, 114–135 (1975) source
Not checked in the free run.
- cites [8] Smith, P. L. & Vickers, D. The accumulator model of two-choice discrimination. J. Math. Psychol. 32, 135–168 (1988) source
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenOne important limitation of these models is that they fail to explain why a decision-maker might change their mind after an initial decision has been taken.
“One important limitation of these models is that they fail to explain why a decision-maker might change their mind after an initial decision has been taken.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe observation provides evidence against two main alternatives to post-initiation processing: (1) change of decision based on recall and/or reconsideration of evidence acquired before initiation, and (2) a correction of an initial motor error perhaps owing to confusion about the stimulus-response mapping.
“The observation provides evidence against two main alternatives to post-initiation processing: (1) change of decision based on recall and/or reconsideration of evidence acquired before initiation18, and (2) a correction of an initial motor error perhaps owing to confusion about the stimulus-response mapping12.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [18] Romo, R., Hernandez, A., Zainos, A., Lemus, L. & Brody, C. D. Neuronal correlates of decision-making in secondary somatosensory cortex. Nature Neurosci. 5, 1217–1225 (2002) source
Not checked in the free run.
- cites [12] Rabbitt, P. & Vyas, S. Processing a display even after you make a response to it. How perceptual errors can be corrected. Q. J. Exp. Psychol. A 33, 223–239 (1981) source
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
InterpretationreasoningSuch strong evidence ought to arrive when the initial choice is an error and when the motion is strong.
“Such strong evidence ought to arrive when the initial choice is an error and when the motion is strong.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataTaking the motion energy filtering delay into account, the initial choices depend on the earliest information in the stimulus, but ignore an epoch on the order of tnd.
“Taking this into account, the initial choices depend on the earliest information in the stimulus, but ignore an epoch on the order of tnd.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningBecause in our study the subject must complete an arm movement, the optimal solution is likely to trade-off accuracy against motor costs (energy) associated with larger corrections nearer the end of the movement.
“Also, because in our study the subject must complete an arm movement, the optimal solution is likely to trade-off accuracy against motor costs (energy) associated with larger corrections nearer the end of the movement.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
InterpretationreasoningA change that depends on retrieval of information from memory or incorporation of a new decision policy (e.g., values) would require elaboration of the model.
“Further, a change that depends on retrieval of information from memory or incorporation of a new decision policy (e.g., values) would require elaboration of the model.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationIt is well known that the initiation and final specification of a movement can be dissociated in time.
“It is well known that the initiation and final specification of a movement can be dissociated in time30.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [30] Ghez, C., Hening, W. & Favilla, M. Gradual specification of response amplitude in human tracking performance. Brain Behav. Evol. 33, 69–74 (1989) source
Not checked in the free run.
May combine two claims.
Interpretationnone givenThe decision rule is to change only if the accumulated evidence reaches the change bound and to reaffirm otherwise.
“The decision rule is to change only if the accumulated evidence reaches the change bound and to reaffirm otherwise.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S1 (asserted), S4 (based).
InterpretationreasoningAn optimal solution to maximize the percentage of correct final choices would be to continue to accumulate evidence about direction until there is no more to be had (i.e., tnd) and to decide in favor of the more likely direction.
“Suppose the subject wishes to use changes of mind to maximize the percentage of correct final choices. Then the subject ought to continue to accumulate evidence about direction until there is no more to be had (i.e., tnd) and to decide in favor of the more likely direction.”
Quote in the paperNot a substantial claim
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Variation in post-decision confidence predicts how likely the decision is to be revised if an opportunity arises.
“Variation in post-decision confidence predicts how likely the decision is to be revised if an opportunity arises,9”
From Two prefrontal-subcortical circuits estimate controllability and reflect its impact on learning (2026) · cites this paper as [9]
The paper’s Abstract reports that changes of mind depend on task difficulty and whether the initial decision was correct, and its model explains revision using evidence accumulated after the initial decision. It does not report measuring post-decision confidence or show that variation in confidence predicts revision. (LLM, conf=0.78)
Experimental — accuracy not yet measuredEvidence accumulation may continue after a decision has been made, informing estimates of decision confidence and potentially leading to changes of mind.
“Pleskac & Busemeyer (2010) additionally proposed that this evidence accumulation process may continue after a decision has been made, informing estimates of decision confidence and potentially leading to changes of mind (Resulaj et al. 2009, van den Berg et al. 2016a).”
From Metacognition and Confidence: A Review and Synthesis (2024) · cites this paper as Resulaj et al. 2009
The passage describing the model (section labeled “Changing your mind: a computational mechanism of vacillation > Abstract”; page unavailable) says evidence continues to accumulate after the initial decision and can reach a change-of-mind bound. Our copy does not say that this post-decision accumulation informs estimates of decision confidence. (LLM, conf=0.85)
Experimental — accuracy not yet measuredEvidence for a process of post-decision evidence accumulation was adduced to explain reversals of perceptual decisions in humans and monkeys.
“Evidence for such a process was adduced to explain reversals of perceptual decisions in humans and monkeys (Rabbitt, 1966; Rabbitt and Vyas, 1981; McPeek et al., 2000; Caspi and Beutter, 2004; Van Zandt and Maldonado-Molina, 2004; Resulaj et al., 2009; Burk et al., 2014; Kiani et al., 2014b; Moher and Song, 2014).”
From A common mechanism underlies changes of mind about decisions and confidence (2016) · cites this paper as Resulaj et al., 2009
In previous studies, the motor system receives a continuous updating of the state of the evidence leading to a choice and can utilize late arriving information to revise an initial choice by continuing the process of deliberation.
“In previous studies, we showed that the motor system receives a continuous updating of the state of the evidence leading to a choice (Gold and Shadlen, 2000; Selen et al., 2012) and can utilize the late arriving information to revise an initial choice by continuing the process of deliberation (Resulaj et al., 2009; Burk et al., 2014).”
From A common mechanism underlies changes of mind about decisions and confidence (2016) · cites this paper as Resulaj et al., 2009
For three of the four subjects, changes of decision corrected an initial error more often than they spoiled an initially correct choice.
“For three of the four subjects, changes of decision corrected an initial error more often than they spoiled an initially correct choice (P<0.001, P<0.001, P = 0.084, P<0.001), consistent with previous reports (Resulaj et al., 2009; Burk et al., 2014).”
From A common mechanism underlies changes of mind about decisions and confidence (2016) · cites this paper as Resulaj et al., 2009
Cites
(1966) · Medical Entomology and Zoology · Cited by 10,293 · see citing works
Risken, H. (1984) · Springer series in synergetics · doi:10.1007/978-3-642-61544-3 · Cited by 9,128 · see citing works
Signal detection theory and psychophysics
Ingleby, D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,616 · see citing works
The Jackknife, the Bootstrap and Other Resampling Plans
Efron, B. (1982) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970319 · Cited by 8,120 · see citing works
The Neural Basis of Decision Making
Gold, J. I., & Shadlen, M. N. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.113038 · Cited by 3,764 · see citing works
Self-citationSpatiotemporal energy models for the perception of motion
Adelson, E. H., & Bergen, J. R. (1985) · Journal of the Optical Society of America A · doi:10.1364/josaa.2.000284 · Cited by 3,442 · see citing works
The Jackknife: The Bootstrap and Other Resampling Plans.
Low, L. Y., & Efron, B. (1983) · Journal of the American Statistical Association · doi:10.2307/2288218 · Cited by 2,728 · see citing works
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224 · see citing works
The time course of perceptual choice: The leaky, competing accumulator model.
Usher, M., & McClelland, J. L. (2001) · Psychological Review · doi:10.1037/0033-295x.108.3.550 · Cited by 2,152 · see citing works
Bogacz, R., Brown, E., Moehlis, J., Holmes, P. J., & Cohen, J. D. (2006) · Psychological Review · doi:10.1037/0033-295x.113.4.700 · Cited by 1,845 · see citing works
Optimality principles in sensorimotor control
Todorov, E. (2004) · Nature Neuroscience · doi:10.1038/nn1309 · Cited by 1,746 · see citing works
Roitman, J. D., & Shadlen, M. N. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-21-09475.2002 · Cited by 1,589 · cited 1× · see citing works
“It is precluded when an action is ballistic, for instance when a subject makes an eye movement to a choice target9,15.”
Self-citationModeling Response Times for Two-Choice Decisions
Ratcliff, R., & Rouder, J. N. (1998) · Psychological Science · doi:10.1111/1467-9280.00067 · Cited by 1,478 · see citing works
A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion
Kass, R. E., & Wasserman, L. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476592 · Cited by 1,254 · see citing works
Psychology and neurobiology of simple decisions
Smith, P. L., & Ratcliff, R. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.01.006 · Cited by 1,051 · see citing works
A Recurrent Network Mechanism of Time Integration in Perceptual Decisions
Wong, K.-F., & Wang, X.-J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.3733-05.2006 · Cited by 1,029 · see citing works
Information theory of choice-reaction times
Laming, D. (1968) · Medical Entomology and Zoology · Cited by 952 · see citing works
The neural systems that mediate human perceptual decision making
Heekeren, H. R., Marrett, S., & Ungerleider, L. G. (2008) · Nature reviews. Neuroscience · doi:10.1038/nrn2374 · Cited by 877 · see citing works
Bayesian brain : probabilistic approaches to neural coding
賢治, 銅., 信, 石., Pouget, A., & Rao, R. P. N. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/9780262042383.001.0001 · Cited by 823 · see citing works
SENSE AND THE SINGLE NEURON: Probing the Physiology of Perception
Parker, A. J., & Newsome, W. T. (1998) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.21.1.227 · Cited by 801 · see citing works
The effect of stimulus strength on the speed and accuracy of a perceptual decision
Palmer, J., Huk, A. C., & Shadlen, M. N. (2005) · Journal of Vision · doi:10.1167/5.5.1 · Cited by 691 · see citing works
Self-citationProbabilistic population codes for Bayesian decision making
Beck, J. M., Ma, W. J., Kiani, R., Hanks, T., Churchland, A. K., Roitman, J., Shadlen, M. N., Latham, P. E., & Pouget, A. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.021 · Cited by 645 · see citing works
Self-citationProbabilistic reasoning by neurons
Yang, T., & Shadlen, M. N. (2007) · Nature · doi:10.1038/nature05852 · Cited by 581 · see citing works
Self-citationGold, J. I., & Shadlen, M. N. (2002) · Neuron · doi:10.1016/s0896-6273(02)00971-6 · Cited by 547 · see citing works
Self-citationChoosing the greater of two goods: neural currencies for valuation and decision making
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1666 · Cited by 536 · see citing works
Neural basis of deciding, choosing and acting
Schall, J. D. (2001) · Nature reviews. Neuroscience · doi:10.1038/35049054 · Cited by 528 · see citing works
Kiani, R., Hanks, T. D., & Shadlen, M. N. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.4761-07.2008 · Cited by 500 · see citing works
Self-citationMonkeys Pay Per View: Adaptive Valuation of Social Images by Rhesus Macaques
Deaner, R. O., Khera, A. V., & Platt, M. L. (2005) · Current Biology · doi:10.1016/j.cub.2005.01.044 · Cited by 431 · see citing works
Performance monitoring by the supplementary eye field
Stuphorn, V., Taylor, T. L., & Schall, J. D. (2000) · Nature · doi:10.1038/35048576 · Cited by 392 · see citing works
Neuronal correlates of decision-making in secondary somatosensory cortex
Romo, R., Hernández, A., Zainos, A., Lemus, L., & Brody, C. D. (2002) · Nature Neuroscience · doi:10.1038/nn950 · Cited by 387 · see citing works
The relative judgment theory of two choice response time
Link, S. W. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90053-x · Cited by 251 · see citing works
Seeing at a glance, smelling in a whiff: rapid forms of perceptual decision making
Uchida, N., Kepecs, Á., & Mainen, Z. F. (2006) · Nature reviews. Neuroscience · doi:10.1038/nrn1933 · Cited by 231 · see citing works
The accumulator model of two-choice discrimination
Smith, P. L., & Vickers, D. (1988) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(88)90043-0 · Cited by 223 · see citing works
Error Correction Time without External Error Signals
RABBITT, P. M. A. (1966) · Nature · doi:10.1038/212438a0 · Cited by 213 · see citing works
A modular planar robotic manipulandum with end-point torque control
Howard, I. S., Ingram, J. N., & Wolpert, D. M. (2009) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2009.05.005 · Cited by 211 · see citing works
Self-citationProcessing a Display Even after you Make a Response to it. How Perceptual Errors can be Corrected
Rabbitt, P., & Vyas, S. (1981) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748108400790 · Cited by 97 · see citing works
Gradual Specification of Response Amplitude in Human Tracking Performance
Ghez, C., Hening, W., & Favilla, M. (2008) · Brain, Behavior and Evolution · doi:10.1159/000115902 · Cited by 62 · see citing works
Fiducial Intervals: What are They?
Wang, Y. H. (2000) · The American Statistician · doi:10.1080/00031305.2000.10474520 · Cited by 35 · see citing works
The Speed and Accuracy of a Simple Perceptual Decision: A Mathematical Primer
Shadlen, M. N., Hanks, T. D., Churchland, A. K., Kiani, R., & Yang, T. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/1535.003.0016 · Cited by 24 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Neural Mechanisms for Interacting with a World Full of Action Choices
Cisek, P., & Kalaska, J. F. (2010) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.051508.135409 · Cited by 1,543 · see citing works
Principles of sensorimotor learning
Wolpert, D. M., Diedrichsen, J., & Flanagan, J. R. (2011) · Nature Reviews Neuroscience · doi:10.1038/nrn3112 · Cited by 1,391 · see citing works
Self-citationWhat is consciousness, and could machines have it?
Dehaene, S., Lau, H., & Kouider, S. (2017) · Science · doi:10.1126/science.aan8871 · Cited by 855 · see citing works
Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.
Pleskac, T. J., & Busemeyer, J. R. (2010) · Psychological Review · doi:10.1037/a0019737 · Cited by 722 · see citing works
A synaptic threshold mechanism for computing escape decisions
Evans, D. A., Stempel, A. V., Vale, R., Ruehle, S., Lefler, Y., & Branco, T. (2018) · Nature · doi:10.1038/s41586-018-0244-6 · Cited by 449 · see citing works
Choice Certainty Is Informed by Both Evidence and Decision Time
Kiani, R., Corthell, L., & Shadlen, M. N. (2014) · Neuron · doi:10.1016/j.neuron.2014.12.015 · Cited by 470 · see citing works
Self-citationThe classic P300 encodes a build‐to‐threshold decision variable
Twomey, D. M., Murphy, P. R., Kelly, S. P., & O'Connell, R. G. (2015) · European Journal of Neuroscience · doi:10.1111/ejn.12936 · Cited by 376 · see citing works
Confidence as Bayesian probability: From neural origins to behavior
Meyniel, F., Sigman, M., & Mainen, Z. F. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.039 · Cited by 349 · see citing works
A common mechanism underlies changes of mind about decisions and confidence
van den Berg, R., Anandalingam, K., Zylberberg, A., Kiani, R., Shadlen, M. N., & Wolpert, D. M. (2016) · eLife · doi:10.7554/elife.12192 · Cited by 270 · see citing works
Self-citationMetacognition and Confidence: A Review and Synthesis
Fleming, S. M. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-022423-032425 · Cited by 256 · see citing works
Hutcherson, C. A., Plassmann, H., Gross, J. J., & Rangel, A. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.6387-11.2012 · Cited by 227 · see citing works
Distinct brain mechanisms for conscious versus subliminal error detection
Charles, L., Van Opstal, F., Marti, S., & Dehaene, S. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.01.054 · Cited by 208 · see citing works
Embodied choice: how action influences perceptual decision making
Lepora, N. F., & Pezzulo, G. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004110 · Cited by 209 · see citing works
Neural evidence accumulation persists after choice to inform metacognitive judgments
Murphy, P. R., Robertson, I. H., Harty, S., & O'Connell, R. G. (2015) · eLife · doi:10.7554/elife.11946 · Cited by 170 · see citing works
Representation of accumulating evidence for a decision in two parietal areas
de Lafuente, V., Jazayeri, M., & Shadlen, M. N. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2451-14.2015 · Cited by 162 · see citing works
“…of these fluctuations on the monkeys’ choices, we extracted the motion energy profile (Adelson and Bergen, 1985; Neri, 2004; Kiani et al., 2008; Resulaj et al., 2009) as a function of time from the dis- play of each trial, applying a sign convention such that positive motion energy represents…”
BackgroundMethodSelf-citationCompulsivity reveals a novel dissociation between action and confidence
Vaghi, M. M., Luyckx, F., Sule, A., Fineberg, N. A., Robbins, T. W., & De Martino, B. (2017) · Neuron · doi:10.1016/j.neuron.2017.09.006 · Cited by 145 · see citing works
Confidence Is the Bridge between Multi-stage Decisions
van den Berg, R., Zylberberg, A., Kiani, R., Shadlen, M. N., & Wolpert, D. M. (2016) · Current Biology · doi:10.1016/j.cub.2016.10.021 · Cited by 125 · see citing works
Confidence controls perceptual evidence accumulation
Balsdon, T., Wyart, V., & Mamassian, P. (2020) · Nature Communications · doi:10.1038/s41467-020-15561-w · Cited by 137 · see citing works
Forming global estimates of self-performance from local confidence
Rouault, M., Dayan, P., & Fleming, S. M. (2019) · Nature Communications · doi:10.1038/s41467-019-09075-3 · Cited by 134 · see citing works
The neural system of metacognition accompanying decision-making in the prefrontal cortex
Qiu, L., Su, J., Ni, Y., Bai, Y., Zhang, X., Li, X., & Wan, X. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004037 · Cited by 125 · see citing works
Motor effort alters changes of mind in sensorimotor decision making
Burk, D., Ingram, J. N., Franklin, D. W., Shadlen, M. N., & Wolpert, D. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0092681 · Cited by 117 · see citing works
Berwian, I. M., Wenzel, J. G., Collins, A. G. E., Seifritz, E., Stephan, K. E., Walter, H., & Huys, Q. J. M. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.4971 · Cited by 113 · see citing works
Human VMPFC encodes early signatures of confidence in perceptual decisions
Gherman, S., & Philiastides, M. G. (2018) · eLife · doi:10.7554/elife.38293 · Cited by 107 · see citing works
Decoding the Dynamics of Action, Intention, and Error Detection for Conscious and Subliminal Stimuli
Charles, L., King, J.-R., & Dehaene, S. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.2465-13.2014 · Cited by 84 · see citing works
Understanding neural signals of post-decisional performance monitoring: An integrative review
Desender, K., Ridderinkhof, K. R., & Murphy, P. R. (2021) · eLife · doi:10.7554/elife.67556 · Cited by 77 · see citing works
Unconscious information changes decision accuracy but not confidence
Vlassova, A., Donkin, C., & Pearson, J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403619111 · Cited by 70 · see citing works
Modeling perceptual confidence and the confidence forced-choice paradigm.
Mamassian, P., & de Gardelle, V. (2022) · Psychological Review · doi:10.1037/rev0000312 · Cited by 70 · see citing works
Khalvati, K., Kiani, R., & Rao, R. P. N. (2021) · Nature Communications · doi:10.1038/s41467-021-25419-4 · Cited by 64 · see citing works
Self-citationConsensus Goals in the Field of Visual Metacognition
Rahnev, D., Balsdon, T., Charles, L., de Gardelle, V., Denison, R., Desender, K., Faivre, N., Filevich, E., Fleming, S. M., Jehee, J., et al. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221075615 · Cited by 63 · see citing works
Nakayama, K., Moher, J., & Song, J.-H. (2023) · Annual Review of Psychology · doi:10.1146/annurev-psych-021422-043229 · Cited by 21 · see citing works
Two prefrontal-subcortical circuits estimate controllability and reflect its impact on learning
Liu, Y., Spiering, L., Luo, S., Kingston, N., Grohn, J., & Rushworth, M. F. S. (2026) · Neuron · doi:10.1016/j.neuron.2026.07.029 · Cited by 1 · see citing works
Sensorimotor Mechanisms of Decisions and Actions
James, T. W. (2026) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.a.2484
- Authors
- Resulaj, A., Kiani, R., Wolpert, D. M., & Shadlen, M. N.
- Affiliations
- University of Cambridge (GB)
- Howard Hughes Medical Institute (US)
- Corresponding author
- Shadlen, Michael N.
- Venue
- Nature
- Published
- 2009-08-19
- Volume
- 461
- Issue
- 7261
- Pages
- 263-266
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC2875179
- OpenAlex
- W2070029806
- Cited by
- 678
- References
- 39
- Citations by year
- 2026: 27 · 2025: 46 · 2024: 65 · 2023: 33 · 2022: 45 · 2021: 67 · 2020: 58 · 2019: 44 · 2018: 42 · 2017: 31 · 2016: 43 · 2015: 46 · 2014: 41 · 2013: 26 · 2012: 35
- FWCI
- 10.05
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Funders
- Howard Hughes Medical Institute
- Wellcome Trust (077730, 028056)
- National Eye Institute (EY11378, R01EY011378)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
- Keywords
- decision-making, change of mind, vacillation, self-correction, evidence accumulation, drift-diffusion model, decision reversal, decision accuracy, task difficulty, hand trajectory
- MeSH terms
- Computers, Cues, Decision Making, Female, Hand, Humans, Male, Models, Neurological, Models, Psychological, Motion, Movement, Photic Stimulation, Psychomotor Performance, Reaction Time, Time Factors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review