The anatomy of choice: active inference and agency

Value of a policy can be decomposed into the entropy of the distribution over final states given the current state and policy, and the expected utility of the final state.

“Equation 4 shows that value can be decomposed into terms. The first is the entropy (intrinsic reward) of the distribution over final states, given the current state and policy. The second is the expected utility of the final state, where utility (extrinsic reward) or negative cost is the log probability of the final state, under the prior goals c(sT|m) = ln P(sT|m).”

The expectations of hidden states and policies are softmax functions of mixtures of the other expectations.

“In terms of the updates themselves, the expectations of hidden states and policies are softmax functions of mixtures of the other expectations.”

The first variational step updates beliefs about the current state of the world using observed outcomes and representations of the preceding state.

“The first variational step updates beliefs about the current state of the world using observed outcomes and representations of the preceding state.”

There is a third term that corresponds to expected value of each state, averaged over policies.

“However, there is a third term that corresponds to expected value of each state, averaged over policies.”

The marginal distributions Q(ψi |μi) that minimize free energy can be expressed in terms of the variational energies V(ψi) of each subset.

“It is straightforward to show (Beal, 2003) that the marginal distributions Q(ψi |μi) that minimize free energy can be expressed in terms of the variational energies V(ψi) of each subset:”

• cites Beal, 2003Not checked yet

  Not checked in the free run.

This restriction reduces the number of sufficient statistics for policies from KT to (K − 1) × T.

“This allows one to reduce the number of sufficient statistics for policies from KT to (K − 1) × T by only allowing policies in which a choice uτ > 1 is made at t = τ and ut = 1 otherwise.”

The softmax functions are of linear mixtures of sufficient statistics (neuronal activity) with one key exception; namely, the modulation by precision when updating beliefs about the current state of the world and selecting the next action.

“Furthermore, the softmax functions are of linear mixtures of sufficient statistics (neuronal activity) with one key exception; namely, the modulation by precision when updating beliefs about the current state of the world and selecting the next action.”

There are two expected values, one for action Q · s⌢ and one for perception QT · π⌢.

“In other words, there are two expected values, one for action Q · s⌢ and one for perception QT · π⌢.”

The second variational update is a softmax function of the expected value of competing choices under the current state.

“The second variational update is a softmax function of the expected value of competing choices under the current state.”

The final variational step estimates the precision of prior beliefs about policies, using posterior expectations about hidden states and choices.

“The final variational step estimates the precision of prior beliefs about policies, using posterior expectations about hidden states and choices.”

Value can never be greater than zero (because the KL divergence is always non-negative).

“Mathematically, value can never be greater than zero (because the KL divergence is always non-negative).”

Prior to the high offer, the agent believes that it will accept the low offer three or four trials in the future.

“The interesting thing here is that prior to the high offer, the agent believes that it will accept the low offer three or four trials in the future.”

The update for expected precision is an increasing asymptotic function of value, expected under current beliefs about states and choices.

“Inspection of the update for expected precision shows that it is an increasing asymptotic function of value, expected under current beliefs about states and choices (see Figure 5).”

The resulting updates show phasic responses to the arrival of new sensory information that converge to tonic values, which minimize free energy.

“The resulting updates are quite revealing and show phasic responses to the arrival of new sensory information that converge to tonic values, which minimize free energy.”

Precision increases to its upper bound of α, when value increases to zero.

“This means that precision increases to its upper bound of α, when value increases to zero (see Figure 5).”

There is still an increasing propensity to accept, even though the agent knows that accepting is futile.

“Notice that there is still an increasing propensity to accept, even though the agent knows that accepting is futile.”

In another realization where the low offer was withdrawn after the fourth trial, the expected control states and precision are exactly the same as in the previous simulation until the offer is withdrawn.

“Figure 7 shows the results of this simulation, where the expected control states and precision are exactly the same as in the previous simulation, until the offer is withdrawn.”

The expected precision of these posterior beliefs declines gently until the high offer is made.

“The expected precision of these posterior beliefs is shown in the lower left panel and declines gently until the high offer is made.”

The propensity to accept (in the future) increases as time goes on.

“Furthermore, the propensity to accept (in the future) increases as time goes on (see dotted lines).”

At the point the offer is withdrawn, the agent moves to the no-offer state and remains there until the end of the game.

“At this point, the agent moves to the no-offer state and remains there until the end of the game.”

In this instance, precision falls at the point the offer is withdrawn and remains low until the last trial.

“In this instance, precision falls at the point the offer is withdrawn and remains low until the last trial.”

Uncertainty about the policy increases as the propensity to accept increases.

“One can see that this uncertainty increases as the propensity to accept increases.”

The marginal utility of the low offer decreases more slowly as the utility of the low offer falls.

“Conversely, the marginal utility of the low offer decreases more slowly as the utility of the low offer falls.”

Including the entropy term in value slightly delays the time to choice (increases latency).

“Note that including the entropy in value slightly delays the time to choice—to ensure a greater latitude of options.”

• Also stated in: § Temporal discounting and marginal utility · plain

  “Here, one sees a slight increase in the latency when value includes the entropy term.”

When the agent has in mind a 50-50 probability of accepting, the entropy peaks shortly before the last offer.

“When the agent has in mind a 50–50 probability of accepting, the entropy peaks, shortly before the last offer.”

The expected utility of the final state is greater when there are fewer intervening trials.

“The key result here is the time-dependent change in expected utility, which corresponds to temporal discounting of the expected utility: i.e., the expected utility of the final state is greater when there are fewer intervening trials.”

The delay from including entropy in value is particularly noticeable in the ambiguous situation when the low offer has the same utility as the high offer (of four).

“This is particularly noticeable in the ambiguous situation when the low offer has the same utility as the high offer (of four).”

Entropy and expected utility demonstrate a complementary dependency on time: as time progresses, expected utility first falls and then increases, while entropy does the converse.

“The entropy (red) and expected utility (blue) underlying these choices are shown in the right panel and demonstrate—in this example—a complementary dependency on time.”

• Also stated in: § Temporal discounting and marginal utility · plain

  “As time progresses, the expected utility first falls and then increases, while the entropy does the converse.”

In active inference, parameters like sensitivity or inverse temperature must themselves minimize free energy.

“Crucially, in active inference, parameters like sensitivity or inverse temperature must themselves minimize free energy.”

This behavior is equivalent to minimizing the Shannon entropy of the distribution over the outcomes agents experience.

“This behavior is equivalent to minimizing the Shannon entropy of the distribution over the outcomes they experience.”

The free-energy principle explains how agents restrict themselves to a small number of attracting states.

“The free-energy principle (Friston et al., 2006) tries to explain how agents restrict themselves to a small number of attracting states.”

• cites Friston et al., 2006Not checked yet

  Not checked in the free run.

Under ergodic assumptions, this entropy is almost surely the long-term time average of self-information or surprise.

“Under ergodic assumptions, this entropy is (almost surely) the long-term time average of self-information or surprise (Birkhoff, 1931).”

• cites Birkhoff, 1931Not checked yet

  Not checked in the free run.

Prior beliefs that action minimizes the Kullback-Leibler divergence between desired states and attainable states in the future allow bounded rationality to be formulated as approximate Bayesian inference that optimizes a free energy bound on model evidence.

“In particular, we consider prior beliefs that action minimizes the Kullback–Leibler (KL) divergence between desired states and attainable states in the future. This allows one to formulate bounded rationality as approximate Bayesian inference that optimizes a free energy bound on model evidence.”

• AssumptionThe authors state this as an assumption: Policies are selected under the prior belief that they minimize the divergence, or relative entropy, between a probability distribution over states that can be reached and states agents believe they should occupy, which are states or goals believed a priori to have high utility.

QRE, logit choice models and softmax rules can be derived as formal consequences of free energy minimization, using variational Bayes.

“Furthermore, QRE, logit choice models and softmax rules can be derived as formal consequences of free energy minimization, using variational Bayes.”

Variational Bayes provides formal constraints on the computational anatomy of inference and action that are remarkably consistent with neuroanatomy.

“Here, we consider variational Bayes as an alternative scheme that provides formal constraints on the computational anatomy of inference and action—constraints that are remarkably consistent with neuroanatomy.”

Optimal behavior entails a representation of confidence about outcomes that are under an agent's control.

“In turn, this means that optimal behavior entails a representation of confidence about outcomes that are under an agent's control.”

Expected utility, exploration bonuses, softmax choice rules and optimism bias emerge as natural consequences of the active inference formulation.

“We show that constructs like expected utility, exploration bonuses, softmax choice rules and optimism bias emerge as natural consequences of this formulation.”

Sensitivity ceases to be a free parameter adjusted to describe observed behavior and becomes diagnostic of the underlying approximate Bayesian inference.

“This means that sensitivity ceases to be a free parameter that is adjusted to describe observed behavior and becomes diagnostic of the underlying (approximate) Bayesian inference (that can be disclosed by observed choices).”

Expected utility theory emerges as a special case of free energy minimization, where the sensitivity or inverse temperature of softmax functions and quantal response equilibria has a unique and Bayes-optimal solution that minimizes free energy.

“For example, expected utility theory emerges as a special case of free energy minimization, where the sensitivity or inverse temperature (of softmax functions and quantal response equilibria) has a unique and Bayes-optimal solution—that minimizes free energy.”

The free energy minimization considered in this paper can be regarded as a special case of the general utility-based free energy minimization formulation of Braun et al., where the utility function is the log-likelihood of outcomes and their causes under a generative model.

“The free energy minimization considered in this paper can be regarded as a special case of their general formulation, where the utility function is the log-likelihood of outcomes and their causes, under a generative model.”

• cites Braun et al., 2011Not checked yet

  Not checked in the free run.

• cites Ortega and Braun, 2011Not checked yet

  Not checked in the free run.

An intrinsically rewarding policy maximizes the opportunity to explore, or the entropy of, future states.

“In the present context, an intrinsically rewarding policy maximizes the opportunity to explore (or the entropy of) future states.”

This value has exactly the same form as the objective functions in KL control schemes that grandfather conventional utility-based schemes.

“This value has exactly the same form as the objective functions in KL control schemes that grandfather conventional utility-based schemes (Kappen et al., 2012; Ortega and Braun, 2011).”

• cites Kappen et al., 2012Not checked yet

  Not checked in the free run.

• cites Ortega and Braun, 2011Not checked yet

  Not checked in the free run.

Equipping agents with a sense of agency or prior beliefs that they are KL optimal, enacted through active inference, allows the precision of beliefs about control to be optimized because the optimal control problem is cast as an inference problem.

“All we have done here is to equip agents with a sense of agency or prior beliefs that they are KL optimal. These beliefs are then enacted through active inference. The advantage of this is that the precision of beliefs about control can now be optimized—because we have effectively cast the optimal control problem as an inference problem.”

• AssumptionThe authors state this as an assumption: Policies are selected under the prior belief that they minimize the divergence, or relative entropy, between a probability distribution over states that can be reached and states agents believe they should occupy, which are states or goals believed a priori to have high utility.

It is sufficient to minimize surprise at each point in time to minimize its time average or Shannon entropy.

“It is therefore sufficient to minimize surprise—at each point in time—to minimize its time average or Shannon entropy.”

Free energy is an upper bound on surprise because the first relative entropy or KL divergence term is non-negative by Gibbs inequality.

“The second equality shows that free energy is an upper bound on surprise, because the first relative entropy or Kullback–Leibler (KL) divergence term is non-negative by Gibbs inequality (Beal, 2003).”

• cites Beal, 2003Not checked yet

  Not checked in the free run.

Minimizing free energy corresponds to minimizing the divergence between the approximate and true posterior.

“This means minimizing free energy corresponds to minimizing the divergence between the approximate and true posterior.”

This requires agents to entertain beliefs about the control of state transitions producing outcomes.

“Crucially, this requires agents to entertain beliefs about the control of state transitions producing outcomes.”

This probabilistic prediction error is a difference between probability distributions over states, unlike classic reward prediction errors.

“Unlike classic reward prediction errors, this probabilistic prediction error is a difference between probability distributions over states.”

Prior beliefs about control sequences or policies determine which actions are selected.

“This describes prior beliefs about control sequences or policies that determine which actions are selected.”

Variational free energy is by construction an upper bound on surprise.

“The proxy is variational free energy that, by construction, is an upper bound on surprise (Feynman, 1972; Hinton and van Camp, 1993).”

• cites Feynman, 1972Not checked yet

  Not checked in the free run.

• cites Hinton and van Camp, 1993Not checked yet

  Not checked in the free run.

The entropy term implies that (beliefs about) choices are driven not just to maximize expected value but to explore all options in a way that confers an exploratory aspect on behavior.

“The entropy term implies that (beliefs about) choices are driven not just to maximize expected value but to explore all options in a way that confers an exploratory aspect on behavior.”

If agents occupy a limited set of attracting states, their generative models must be equipped with prior beliefs that controlled state transitions will minimize the divergence between a distribution over attainable states and a distribution that specifies states as attractive.

“In summary, if agents occupy a limited set of attracting states, it follows that their generative models must be equipped with prior beliefs that controlled state transitions will minimize the divergence between a distribution over attainable states and a distribution that specifies states as attractive.”

In the absence of (or change in) beliefs about ultimate states, there will be a bias toward visiting all (low cost) states with equal probability.

“In the absence of (or change in) beliefs about ultimate states, there will be a bias toward visiting all (low cost) states with equal probability.”

If agents resist a natural tendency to disorder, occupying a limited number of characteristic states, then they become implicit Bayesian modelers of their environment.

“In summary, if agents resist a natural tendency to disorder (occupy a limited number of characteristic states), then they become implicit Bayesian modelers of their environment.”

To evaluate surprise it is necessary to marginalize over the hidden causes of outcomes.

“However, to evaluate surprise it is necessary to marginalize over the hidden causes of outcomes.”

The relative entropy or divergence can be thought of as a prediction error that reports the mismatch between the final outcomes expected with and without considering the current state.

“The relative entropy or divergence can be thought of as a prediction error that is nuanced in an important way: it reports the mismatch—not between expected and observed outcomes—but between the final outcomes expected with and without considering the current state: in other words, the difference between what can be attained from the current state and the goals encoded by prior beliefs.”

The novelty bonus of a new state is, in this formulation, conferred by the opportunity to access states that were previously unavailable-thereby increasing the entropy over final states.

“Similarly, the novelty bonus (Kakade and Dayan, 2002) of a new state is, in this formulation, conferred by the opportunity to access states that were previously unavailable—thereby increasing the entropy over final states.”

• cites Kakade and Dayan, 2002Not checked yet

  Not checked in the free run.

With imprecise (flat) prior beliefs that all final states are equally valuable, the agent will try to keep its options open and maximize the entropy over those states, exploring according to the maximum entropy principle.

“Conversely, with imprecise (flat) prior beliefs that all final states are equally valuable, the agent will try to keep its options open and maximize the entropy over those states: in other words, it will explore according to the maximum entropy principle (Jaynes, 1957).”

• cites Jaynes, 1957Not checked yet

  Not checked in the free run.

These prior beliefs can be expressed in terms of relative entropy that defines the value of policies.

“These prior beliefs can be expressed in terms of relative entropy that defines the value of policies.”

Concrete notions like exploration, exploitation and expected utility emerge as straightforward consequences of this formulation.

“These arguments may seem a bit abstract but, happily, concrete notions like exploration, exploitation and expected utility emerge as straightforward consequences:”

The relative contribution of entropy and expected utility depends upon the precision of prior beliefs about the final state or, equivalently, the relative utility of different states.

“The relative contribution of entropy and expected utility depends upon the precision of prior beliefs about the final state or, equivalently, the relative utility of different states.”

Utility-based free energy treatments of bounded rationality consider optimal agents to minimize the KL divergence between controlled and desired outcomes.

“These schemes consider optimal agents to minimize the KL divergence between controlled and desired outcomes.”

If prior beliefs about the final state are very precise (informative), they will dominate and the agent will (believe it will) maximize expected utility.

“If these beliefs are very precise (informative), they will dominate and the agent will (believe it will) maximize expected utility.”

Basing beliefs about future choices on relative entropy is formally related to optimization schemes based on KL control, particularly risk sensitive control.

“Basing beliefs about future choices on relative entropy is formally related to optimization schemes based on KL control; particularly risk sensitive control; e.g., (van den Broek et al., 2010).”

• cites van den Broek et al., 2010Not checked yet

  Not checked in the free run.

Restricting to policies in which a choice is made once is particularly appropriate for irreversible Markovian processes that have absorbing states.

“There is nothing lawful about this restriction; however, it is particularly appropriate for irreversible Markovian processes that have absorbing states (that render further action pointless).”

Precision plays a fundamental role in moderating an optimism bias when forming beliefs about hidden states of the world.

“In fact, precision plays a fundamental role in moderating an optimism bias when forming beliefs about hidden states of the world (Sharot et al., 2012).”

• cites Sharot et al., 2012Not checked yet

  Not checked in the free run.

This is exactly the softmax choice rule that predominates in QRE theory and other normative models.

“Crucially, this is exactly the softmax choice rule that predominates in QRE theory and other normative models (Haile et al., 2008).”

• cites Haile et al., 2008Not checked yet

  Not checked in the free run.

The rate parameter is no longer a free parameter but a sufficient statistic of the unknown precision of beliefs about policies.

“Crucially, it is no longer a free parameter but a sufficient statistic of the unknown precision of beliefs about policies.”

The formalism of variational Bayes allows one to specify constraints on the form of the approximate posterior that makes prior assumptions or beliefs about allowable choices explicit.

“The main point here is that the formalism of variational Bayes allows one to specify constraints on the form of the approximate posterior that makes prior assumptions or beliefs about allowable choices explicit.”

The factorization with respect to precision is an approximation, because the true posterior will show (mild) conditional dependencies between precision and hidden states.

“Conversely, the factorization with respect to precision is an approximation, because the true posterior will show (mild) conditional dependencies between precision and hidden states.”

Utilitarian formulations overlook the symmetry between the expected value over states-that provides the value of a choice, and the expected value over choices-that provides the value of a state.

“However, utilitarian formulations overlook the symmetry between the expected value over states—that provides the value of a choice, and the expected value over choices—that provides the value of a state.”

Variational Bayes optimizes the sufficient statistics μ with a series of variational updates.

“Variational Bayes optimizes the sufficient statistics μ ϵ ℝ+ with a series of variational updates.”

The problem of a vast number of control sequences can be finessed by only considering allowable or a priori plausible policies.

“However, there are potentially a vast number of control sequences or policies that could require an enormous number of sufficient statistics. This problem can be finessed by only considering allowable or a priori plausible policies.”

The marginal over control states has not been factorized because the final outcome depends, in general, on the particular history of choices.

“The marginal over control states has not been factorized because the final outcome depends, in general, on the particular history of choices.”

The variational scheme has a computational form that resembles many aspects of neuronal processing in the brain.

“The variational scheme above has a computational form that resembles many aspects of neuronal processing in the brain.”

Precision moderates this bias, much like dopamine.

“We can now see why precision moderates this bias, much like dopamine (Sharot et al., 2012).”

• cites Sharot et al., 2012Not checked yet

  Not checked in the free run.

The particular mean field assumptions used are not approximations, because the true generative process is Markovian.

“These particular mean field assumptions are not approximations, because the true generative process is Markovian.”

• AssumptionThe authors state this as an assumption: The true generative process is Markovian.

The rate parameter of the approximate marginal over precision corresponds to temperature in classic formulations.

“This rate parameter corresponds to temperature in classic formulations.”

Variational Bayes provides a generic and relatively simple scheme for approximate Bayesian inference that finesses the combinatoric and analytic intractability of exact inference.

“Variational Bayes now provides a generic and relatively simple scheme for approximate Bayesian inference that finesses the combinatoric and analytic intractability of exact inference (Beal, 2003; Fox and Roberts, 2012).”

• cites Beal, 2003Not checked yet

  Not checked in the free run.

• cites Fox and Roberts, 2012Not checked yet

  Not checked in the free run.

This allows one to factorize the (approximate) posterior distribution into marginal distributions, which greatly reduces the size of the state space that has to be represented.

“This allows one to factorize the (approximate) posterior distribution into marginal distributions, which greatly reduces the size of the state space that has to be represented.”

Explorative behavior is not just a random version of exploitative behavior but can itself be very precise, with a clearly defined objective (to maximize the entropy of final outcomes).

“In other words, explorative behavior is not just a random version of exploitative behavior but can itself be very precise, with a clearly defined objective (to maximize the entropy of final outcomes).”

In this setting, notions like value, expected utility and exploration bonus are consequences of the underlying imperative to minimize (relative) entropy, entailed by the priors of an agent's generative model.

“In this setting, notions like value, expected utility and exploration bonus are consequences of the underlying imperative to minimize (relative) entropy, entailed by the priors of an agent's generative model.”

This confers a temporal contiguity on inference, where empirical prior beliefs about the current state conspire with sensory evidence to produce posterior beliefs.

“This confers a temporal contiguity on inference, where empirical prior beliefs about the current state conspire with sensory evidence to produce posterior beliefs.”

The expected value over choices and states π⌢T · Q · s⌢t specifies the optimal precision or inverse temperature, which is overlooked in classic treatments.

“Furthermore, the expected value over choices and states π⌢T · Q · s⌢t specifies the optimal precision or inverse temperature, which is overlooked in classic treatments.”

The efficiency of variational Bayes rests on replacing posterior dependencies among hidden variables with dependencies among the sufficient statistics of marginal probabilities over subsets.

“The efficiency of variational Bayes rests on replacing posterior dependencies among hidden variables with dependencies among the sufficient statistics of marginal probabilities over subsets.”

The mean field assumption, enforced by the combinatorics and intractability of exact Bayesian inference, implies a segregation of inference into separable cognitive processes and their neuronal substrates (functional segregation).

“The mean field assumption, enforced by the combinatorics and intractability of exact Bayesian inference, implies a segregation of inference into separable cognitive processes and their neuronal substrates (functional segregation).”

• AssumptionThe authors state this as an assumption: The approximate posterior factorizes over past hidden states, future control states, and precision, and factorizes successive states over time.

The particular mean field assumption used here implies distinct perceptual, choice and evaluation processes that can be associated with distributed cortical and subcortical systems in the brain.

“The particular mean field assumption used here implies distinct perceptual, choice and evaluation processes that can be associated with distributed cortical and subcortical systems in the brain.”

• AssumptionThe authors state this as an assumption: The approximate posterior factorizes over past hidden states, future control states, and precision, and factorizes successive states over time.

This reflects the fact that the final outcome is assured with a high degree of confidence, once the high offer has been made and subsequently accepted.

“This reflects the fact that the final outcome is assured with a high degree of confidence, once the high offer has been made and subsequently accepted.”

If there is a clear and precise path from the current state to a goal, then precision will be high.

“Conversely, if there is a clear and precise path from the current state to a goal, then precision will be high.”

The agent waits patiently because it thinks it is more likely to accept an offer in the future than to accept the current offer.

“This means that it waits, patiently, because it thinks it is more likely to accept an offer in the future than to accept the current offer.”

Precision reports the attainability of goals in terms of value.

“This means that precision reports the attainability of goals in terms of value.”

If precision was zero, there would be a failure of action selection in the sense that the value of all choices would be the same.

“However, there would be a failure of action selection in the sense that the value of all choices would be the same.”

Circular causality is one of the most important features of the variational scheme and means that one can consider not just the computational role of precision but also how it is controlled by the representations (posterior expectations) it optimizes.

“This circular causality is one of the most important features of the variational scheme and means that one can consider not just the computational role of precision but also how it is controlled by the representations (posterior expectations) it optimizes.”

Convolving the simulated dopamine responses with an exponential decay function would reproduce the Bayes optimal updates.

“In other words, convolving the simulated dopamine responses with an exponential decay function would reproduce the Bayes optimal updates.”

This is the second key role of precision; namely, to modulate the selection of competing representations of future action.

“This is the second key role of precision; namely, to modulate the selection of competing representations of future action: cf. (Cisek, 2007; Frank et al., 2007; Jocham et al., 2012).”

• cites Cisek, 2007Not checked yet

  Not checked in the free run.

• cites Frank et al., 2007Not checked yet

  Not checked in the free run.

• cites Jocham et al., 2012Not checked yet

  Not checked in the free run.

The agent accepts because this is the most probable choice-in the face of a high offer-under its prior beliefs that it is most likely to have accepted the higher offer at the end of the game.

“It accepts because this is most probable choice—in the face of a high offer—under its prior beliefs that it is most likely to have accepted the higher offer at the end of the game.”

Precision has to be optimized.

“One of the key insights, afforded by the variational scheme, is that precision has to be optimized.”

Neurobiologically, the softmax policy updates would correspond to a winner-take-all or biased competition among competing choices or policies, where competition is modulated by precision.

“Neurobiologically, the softmax policy updates would correspond to a winner-take-all or biased competition among competing choices or policies, where competition is modulated by precision.”

If precision was zero, then perception would be unbiased and represent a veridical representation of worldly states.

“If precision was zero, then perception would be unbiased and represent a veridical representation of worldly states.”

Simply optimizing a temporal discounting parameter in light of observed behavior cannot disambiguate among the prior beliefs an agent may entertain.

“Although hyperbolic (or exponential) discounting may be sufficient for descriptive purposes, simply optimizing a temporal discounting parameter (Daw and Touretzky, 2002), in light of observed behavior, cannot disambiguate among the prior beliefs an agent may entertain.”

• cites Daw and Touretzky, 2002Not checked yet

  Not checked in the free run.

As the utility of a particular final state increases, precision increases more slowly-because the implicit distribution over final states is less likely to be realized.

“Generally, as the utility of a particular final state increases, precision increases more slowly—because the implicit distribution over final states is less likely to be realized.”

The expected utility of the outcome decreases the further it recedes into the future.

“In other words, the expected utility of the outcome decreases, the further it recedes into the future.”

The agent believes it is more likely to secure an offer later in the game because it now knows the offer has not been withdrawn.

“This suggests that the agent believes it is more likely to secure an offer later in the game, because it now knows the offer has not been withdrawn; in other words, the possibility of an early withdrawal cannot be discounted at the beginning of the game.”

The delay from including entropy in value serves to ensure a greater latitude of options.

“Note that including the entropy in value slightly delays the time to choice—to ensure a greater latitude of options.”

Hyperbolic discounting can be derived under a simple generative model of losing a reward, given exponential priors on the hazard rate.

“Having said this (economic) hyperbolic discounting can be derived under a simple generative model of losing a reward, given exponential priors on the hazard rate (Sozou, 1998).”

• cites Sozou, 1998Not checked yet

  Not checked in the free run.

The possibility of an early withdrawal cannot be discounted at the beginning of the game.

“in other words, the possibility of an early withdrawal cannot be discounted at the beginning of the game.”

The discounting of expected utility depends upon inference about the current state, future choices and precision-all of which change with time in an interdependent fashion.

“This is because the discounting of expected utility depends upon inference about the current state, future choices and precision—all of which change with time in an interdependent fashion.”

There is value in keeping one's options open.

“Intuitively, there is value in keeping ones options open: cf. a novelty bonus (Krebs et al., 2009).”

• cites Krebs et al., 2009Not checked yet

  Not checked in the free run.

This illustrates the interdependency of expectations about precision and hidden states of the world-which change after the offer has been withdrawn.

“This illustrates the interdependency of expectations about precision and hidden states of the world—which change after the offer has been withdrawn.”

The correspondence between precision and dopaminergic neurotransmission is supported by exactly the same conclusions reached when examining predictive coding schemes and hierarchical inference using volatility models.

“For people familiar with previous discussions of dopamine in the context of active inference, the correspondence between precision and dopaminergic neurotransmission will come as no surprise—exactly the same conclusions have been reached when examining predictive coding schemes (Friston et al., 2012b) and hierarchical inference using volatility models (Mathys et al., 2011).”

• cites Friston et al., 2012bNot checked yet

  Not checked in the free run.

• cites Mathys et al., 2011Not checked yet

  Not checked in the free run.

There is a convex relationship between marginal utility and utility.

“This leads to a convex relationship between marginal utility and utility: cf. the law of diminishing marginal utility (Kauder, 1953).”

• cites Kauder, 1953Not checked yet

  Not checked in the free run.

This is because all allowable policies entail a choice but with no preference for when that choice is made.

“This is because all allowable policies entail a choice but with no preference for when that choice is made.”

Accepting an offer precludes other outcomes and therefore reduces the entropy of the distribution over final states.

“This reflects the fact that accepting an offer precludes other outcomes and therefore reduces the entropy of the distribution over final states.”

The emergent role of dopamine is to report the precision or salience of perceptual cues that portend a predictable sequence of sensorimotor events.

““In brief, the emergent role of dopamine is to report the precision or salience of perceptual cues that portend a predictable sequence of sensorimotor events.”

• cites Friston et al., 2012bNot checked yet

  Not checked in the free run.

Expected marginal utility directly informs choices (assuming the entropy term in Equation (4) is small enough to be ignored).

“Assuming the entropy term in Equation (4) small enough to be ignored, it is easy to see that expected marginal utility directly informs choices: (10)”

• AssumptionThe authors state this as an assumption: The entropy term in Equation (4) is small enough to be ignored.

Temporal discounting in this example can be explained by prior beliefs that the offer will be withdrawn before the final outcome is reached.

“This is the essence of temporal discounting, which—in this example—can be explained simply by prior beliefs that the offer will be withdrawn before the final outcome is reached.”

• AssumptionRests on an assumption: The offer will be withdrawn before the final outcome is reached (prior belief).

The value of the current state depends upon how easy it is to access the final state.

“This means that the value of the current state depends upon how easy it is to access the final state.”

The ensuing choice depends upon precision.

“Furthermore, the ensuing choice depends upon precision, suggesting that the effect of value on choice can be expressed in terms of an effective utility γ · c(sT) that we will call marginal utility (for consistency with economic theory).”

The marginal utility depends on the confidence that a goal can be reached.

“Intuitively, the marginal utility depends on the confidence that a goal can be reached.”

The asymmetry in marginal utility results from the nature of utility and its dependency upon precision.

“In the current setup, this asymmetry results from the nature of utility and its dependency upon precision.”

Many classic phenomena in utilitarian and economic theory resurface as natural consequences of Bayes optimal (active) inference under a relatively simple generative model.

“In summary, many classic phenomena in utilitarian and economic theory resurface here as natural consequences of Bayes optimal (active) inference under a relatively simple generative model.”

Beliefs about precision depend upon expected value under both states and choices.

“Beliefs about precision depend upon expected value under both states and choices.”

Optimal behavior can be cast as a pure inference problem, in which valuable outcomes are defined in terms of prior beliefs about future states.

“Optimal behavior can be cast as a pure inference problem, in which valuable outcomes are defined in terms of prior beliefs about future states.”

Precision has to be optimized to balance prior beliefs about choices and sensory evidence for hidden states.

“Precision has to be optimized to balance prior beliefs about choices and sensory evidence for hidden states.”

Normative accounts do not distinguish between action and beliefs about action (control).

“One reason is that normative accounts do not distinguish between action and beliefs about action (control).”

These beliefs entail both content (expectations) and uncertainty (precision).

“These beliefs entail both content (expectations) and uncertainty (precision).”

The notion that dopamine encodes precision is now receiving support from several lines of evidence.

“The notion that dopamine encodes precision is now receiving support from several lines of evidence; from purely theoretical treatments of hierarchical Bayesian inference (Mathys et al., 2011), from theoretical neurobiology (Frank et al., 2007; Fletcher and Frith, 2009; Friston et al., 2012b; Pellicano and Burr, 2012) and from empirical studies (Fiorillo et al., 2008; Coull et al., 2011; Galea et a”

• cites Mathys et al., 2011Not checked yet

  Not checked in the free run.

• cites Frank et al., 2007Not checked yet

  Not checked in the free run.

• cites Fiorillo et al., 2008Not checked yet

  Not checked in the free run.

Variational free energy provides a bound on Bayesian model evidence (marginal likelihood) that is optimized by bounded rational behavior.

“Variational free energy provides a bound on Bayesian model evidence (marginal likelihood) that is optimized by bounded rational behavior.”

This mandates beliefs about action (control) that are distinct from action per se-beliefs that entail a precision.

“This mandates beliefs about action (control) that are distinct from action per se—beliefs that entail a precision.”

These beliefs can be cast in terms of minimizing the relative entropy or divergence between prior goals-over final states-and conditional distributions, given the current state of the world and future choices.

“These beliefs can be cast in terms of minimizing the relative entropy or divergence between prior goals—over final states—and conditional distributions, given the current state of the world and future choices.”

This requires (approximate Bayesian) inference on both hidden states of the world and (future) control states.

“This requires (approximate Bayesian) inference on both hidden states of the world and (future) control states.”

There is one comforting point of convergence between variational Bayes and more neurobiologically plausible predictive coding schemes: the solution for both is exactly the same.

“However, there is one comforting point of convergence between variational Bayes and more neurobiologically plausible predictive coding schemes (Bastos et al., 2012): this is the fact that the solution for both is exactly the same.”

• cites Bastos et al., 2012Not checked yet

  Not checked in the free run.