The why, what, where, when and how of goal-directed choice: neuronal and computational principles

“CSL is consistent with ‘predictive brain’ frameworks and ‘free energy minimization’ principles [29,30].”

“This idea has its roots in Gestalt psychology [20] and is a standard approach towards robot navigation [21] but generalizes towards optimal path planning in mobile robots [22] and social interaction in humanoid robots [23].”

“The AL allows the agent to overcome the predefined behavioural repertoire of the RL and to engage an a priori unpredictable world, solving the notorious symbol grounding that led to the demise of classical AI [3,28].”

“The first view holds that the hippocampus codes an agent’s position in space, as inspired by the Cognitive Map theory of Tolman [63] as well as the body of place-cell research initiated by O’Keefe & Dostrovsky [64].”

“The second notion accounts for hippocampal function as a ‘recorder’ of experience—an organ for the formation of episodic memories that would be transferred, with the passage of time, to neocortical sites where also generalization (semanticization) of memory may take place [65].”

“Building on previous models that framed the basal ganglia as an actor–critic architecture for (model-free) reinforcement learning [120,121], we recently argued that especially the ‘Critic’ (goal-predictive) function of the striatum is well supported by the data, whereas its implementation of the ‘Actor’ component is m…”

“These sensorimotor loops are organized in fundamental and opposing behaviour systems that support basic needs such as fight, flight [15], seek and play [16].”

“In addition to storing information, the hippocampus is able to retrieve previously stored information and to self-generate internal sequences of cell activity that are subsequently used to map novel environments or situations [82–84].”

“Neuroscience and neuroeconomics have often benefited from a close linkage to computational methods but these often abstract from many details of situated action (e.g. model-based reinforcement learning theories [134,135] and several contributions to this special issue).”

“With the cognitive revolution, these notions were again incorporated in the terminology of mind and brain [1] but the mechanistic functioning and neuronal underpinnings of goal-directed behaviour remained largely unknown.”

“associations (and ensuing habits) can be acquired via the dorsal striatal system [80,81].”

“As much as there is a ‘hard problem’ in the explanation of consciousness [4,5], we can argue that a similar explanatory gap exists in understanding goal-oriented behaviour, it also critically depends on inferences by the observer on the beliefs an organism entertains with respect to its goals and tasks.”

“Even structures such as posterior parietal cortex and primary sensory cortex (A1, V1) are deeply affected by associative stimulus–reward and action–reward learning [56–62].”

“Importantly, orbito-frontal and medial prefrontal-anterior cingulate neurons are sensitive to the motivational value of cues [100–104] and actions associated with goal pursuit [105,106].”

“Moreover, the representation of the task seems to follow a multiplexing of input streams combining sensory, location and action information at both the input and memory stages of hippocampal processing wood [66,71–73].”

“B 369 : 20130483 associations (and ensuing habits) can be acquired via the dorsal striatal system [80,81].”

“Furthermore, if the value/utility of a goal changes, for instance because of a change in motivational state, the organism should refrain from persistently conducting a fixed stimulus–response sequence or habit, but must adjust its response according to the change in value [7].”

“Lesion studies have confirmed a causal involvement of prefrontal structures in representing goals and task rules, implying orbitofrontal cortex in reversal learning and medial prefrontal cortex in both extra-dimensional shifting [107,108] and learning action–outcome relationships [7,109].”

“…by the RL, following the paradigm of classical conditioning where initially neutral or conditioned stimuli (CS) obtain the ability to trigger actions, or conditioned reponses (CR), by virtue of their contingent presentation with intrinsically motivational stimuli or unconditioned stimuli (US) [12].”

“Lesion studies have confirmed a causal involvement of prefrontal structures in representing goals and task rules, implying orbitofrontal cortex in reversal learning and medial prefrontal cortex in both extra-dimensional shifting [107,108] and learning action–outcome relationships [7,109].”

“…such as comparison—at least in the domain of reward and appetitive learning—is the reward-prediction-error mechanism that may be implemented by mesodiencephalic dopaminergic neurons projecting to striatum and prefrontal cortex [125], while the habenula has been implied in aversive learning [126].”

“Importantly, orbitofrontal and medial prefrontal-anterior cingulate neurons are sensitive to the motivational value of cues [100–104] and actions associated with goal pursuit [105,106].”

“Thus, during ongoing goal-directed behaviour, as well as ‘off-line’, the hippocampus has multiple modes to (re-)generate and recall information from memory, which can be flexibly used to guide decision-making and/or support consolidation [90,91].”

“…inputs), dorsomedial sector (mainly anterior cingulate and prelimbic inputs), a ventrolateral sector (the‘core’ ofnucleusaccumbens; mainlyamygdaloid, prelimbic and dorsal hippocampal input) and a ventromedial sector (‘shell’; mainly prelimbic, infralimbic and ventral hippocampal input [115]).”

“The dorsomedial striatum, for instance, has been implied in action-outcome learning [122], whereas the ventral striatal core functions in cue-outcome learning, in conjunction with its strong amygda-loid input [9,123].”

“Neurophysiological studies in rodents and primates indicate that prefrontal neurons can encode task rules that need to be followed to obtain a goal [94], individual actions or chunks of actions leading up to a goal [95–97], and goals and goal sites themselves [98,99].”

“Even structures such as posterior parietal cortex and primary sensory cortex (A1, V1) are deeply affected by associative stimulus–reward and action–reward learning [56–62].”

“In addition to storing information, the hippocampus is able to retrieve previously stored information and to self-generate internal sequences of cell activity that are subsequently used to map novel environments or situations [82–84].”

“…such as comparison—at least in the domain of reward and appetitive learning—is the reward-prediction-error mechanism that may be implemented by mesodiencephalic dopaminergic neurons projecting to striatum and prefrontal cortex [125], while the habenula has been implied in aversive learning [126].”

“…retrieval is thought to take place as ‘replay’ mainly occurring during hippocampal local-field potential (LFP) events called sharp-wave ripples [85–88], but also during bouts of LFP theta-band activity during running and other ongoing behaviour, resulting in ‘forward sweeps’ of hippocampal place…”

“neutral environmental cues such as odours, and cues that predict reward, as well as the time spent in a situation conducive to goal pursuit [66–70].”

“For goal-directed behaviour, a first point to note is that drives and their associated behavioural expression, such as aggression, thirst, hunger, sex, sleep and safety, are determined in basic form at the level of the hypothalamus together with lower-order structures [50], analogous to the functioning of the RL in DA…”

“Self-initiated retrieval is thought to take place as ‘replay’ mainly occurring during hippocampal local-field potential (LFP) events called sharp-wave ripples [85–88], but also during bouts of LFP theta-band activity during running and other ongoing behaviour, resulting in ‘forward sweeps’ of hippocampal place represe…”

“neutral environmental cues such as odours, and cues that predict reward, as well as the time spent in a situation conducive to goal pursuit [66–70].”

“Even structures such as posterior parietal cortex and primary sensory cortex (A1, V1) are deeply affected by associative stimulus–reward and action–reward learning [56–62].”

“…and events relevant for pursuing a goal in space and time (What, When and Where), the prefrontal cortex appears better equipped to represent a task space, i.e. the set of rules, constraints, goals and goal-predictive values of cues and actions available as options to pursue goals (How) (cf. [110]).”

“How hippocampal events are precisely linked to prefrontal processing is less clear yet, but the medial prefrontal cortex does exhibit replay, and its activity coheres with hippocampal theta activity [132,133].”

“neutral environmental cues such as odours, and cues that predict reward, as well as the time spent in a situation conducive to goal pursuit [66–70].”

“Furthermore, the motor cortices are meant to include premotor, supplementary motor and frontal oculomotor areas (based on [40–42]).”

“Importantly, orbito-frontal and medial prefrontal-anterior cingulate neurons are sensitive to the motivational value of cues [100–104] and actions associated with goal pursuit [105,106].”

“For instance, the preoptic area contains thermosensitive neurons that, at the same time, are involved in the generation and dissipation of body heat [43].”

“[45–49]).”

“[45–49]).”

“The dorsomedial striatum, for instance, has been implied in action-outcome learning [122], whereas the ventral striatal core functions in cue-outcome learning, in conjunction with its strong amygdaloid input [9,123].”

“However, striatal principal cells are connected via GABAergic recurrent collaterals, providing a potential mechanism for competitive selection [117,118].”

“This study also illustrates the relationship between taxon versus route navigation strategies [39] in a robotic system.”

“Even structures such as posterior parietal cortex and primary sensory cortex (A1, V1) are deeply affected by associative stimulus–reward and action–reward learning [56–62].”

“CSL is consistent with ‘predictive brain’ frameworks and ‘free energy minimization’ principles [29,30].”

“Error signals incorporating model-based, highdimensional information (‘I got a banana instead of the apple I expected, but they are worth the same’) may involve cortical systems such as the anterior cingulate and orbitofrontal cortex, but also striatal regions [9,127,128] (see [31] for a more detailed computational an…”

“Neurophysiological studies in rodents and primates indicate that prefrontal neurons can encode task rules that need to be followed to obtain a goal [94], individual actions or chunks of actions leading up to a goal [95–97], and goals and goal sites themselves [98,99].”

“Furthermore, the motor cortices are meant to include premotor, supplementary motor and frontal oculomotor areas (based on [40–42]).”

“[18,51]).”

“With the cognitive revolution, these notions were again incorporated in the terminology of mind and brain [1] but the mechanistic functioning and neuronal underpinnings of goal-directed behaviour remained largely unknown.”

“model-based reinforcement learning theories [134,135] and several contributions to this special issue).”

“non-neuronal feedback resulting from both the high spatio-temporal correlation in sensory streams combined with experience-dependent biased sampling of the sensory space [13,24].”

“Thus, during ongoing goal-directed behaviour, as well as ‘off-line’, the hippocampus has multiple modes to (re-)generate and recall information from memory, which can be flexibly used to guide decision-making and/or support consolidation [90,91].”

“…neurophysiological and neurorobotic approach holds the promise to simultaneously tackle the problem of goal-directed choice at multiple levels [140–142]: (i) the func-tional-behavioural level, e.g. how goals are selected and realized in the real world based on their expected value and the cost of…”

“In this integration, we take as our specific question whether goals should be seen as single scalar functions against which behaviour is optimized or whether they are multi-dimensional processes that emerge from the interaction across a number of perceptual, affective, cognitive and motor systems (see also [6]).”

“[45–49]).”

“Third, the strict distinction between retrospective model-free and prospective model-based decision-making is misleading in the sense that in both cases, the state space on which decision-making operates is defined based on models albeit of varying spatio-temporal complexity [31].”

“Even structures such as posterior parietal cortex and primary sensory cortex (A1, V1) are deeply affected by associative stimulus–reward and action–reward learning [56–62].”

“Even structures such as posterior parietal cortex and primary sensory cortex (A1, V1) are deeply affected by associative stimulus–reward and action–reward learning [56–62].”

“The update on the second notion of the recording of experience is that the hippocampus has turned out not only to record (i.e. encode and store) spatial experiences, but also—and more generally—chains of associated events and sequences of motor actions [77–79].”

“Importantly, orbito-frontal and medial prefrontal-anterior cingulate neurons are sensitive to the motivational value of cues [100–104] and actions associated with goal pursuit [105,106].”

“However, through the state space learning systems of the AL and the task learning systems of the CL, these goals become incrementally more abstracted from their homeostatic origins (see also [137,138]).”

“The reactive layer (RL) of DAC supports the basic functionality of the SL and generates control signals that drive and modulate the engagement of higher control layers and their epistemic functions [14] (figure 1, lower right panel).”

“encode and store) spatial experiences, but also—and more generally—chains of associated events and sequences of motor actions [77–79].”

“During hippocampal forward sweeps, ventral striatal neurons code covert reward expectations, while orbitofrontal neurons code expected outcome at alternative sites that had not been selected for visiting [129,130].”

“However, through the state space learning systems of the AL and the task learning systems of the CL, these goals become incrementally more abstracted from their homeostatic origins (see also [137,138]).”

“…during a countermanding task has shown that the inter trial variability of the neuronal response to the movement cue is directly proportional to the errors the animal has committed (i.e. task memory) and fully predicts performance, i.e. error rate and reaction time rather than firing rate [114].”

“[45–49]).”

“Neurophysiological studies in rodents and primates indicate that prefrontal neurons can encode task rules that need to be followed to obtain a goal [94], individual actions or chunks of actions leading up to a goal [95–97], and goals and goal sites themselves [98,99].”

“However, the inputs to the hippocampus from the grid cells of the medial entorhinal cortex can be interpreted as representing heading direction and thus action, a source of information that in turn strongly dominates memory dynamics [72].”

“…context, the amygdaloid complex should be included as part of a larger network for affectively driven goal-directed behaviour not only mediating value based Pavlovian response behaviours, but should also be goal-oriented instrumental behaviours which can be invigorated by Pavlovian cues [111,112].”

“These hippocampal replay events are likely to impact on the ventral striatum, where replay events have been shown to occur in succession to place-cell replay in the hippocampus [85,87].”

“Neurophysiological studies in rodents and primates indicate that prefrontal neurons can encode task rules that need to be followed to obtain a goal [94], individual actions or chunks of actions leading up to a goal [95–97], and goals and goal sites themselves [98,99].”

“Neurophysiological studies in rodents and primates indicate that prefrontal neurons can encode task rules that need to be followed to obtain a goal [94], individual actions or chunks of actions leading up to a goal [95–97], and goals and goal sites themselves [98,99].”

“In addition, these functions address not only homeostatic regulation, but also allostasis, referring especially to responses to challenges that require system-wide, dynamic adaptation and predictive regulation in anticipation of upcoming homeostatic disturbances [17,18].”

“Thus, especially the orbitofrontal cortex may not only provide model-based information on cue/object value before choices are made [102,103,131], but also engage in post-decisional evaluation and ‘looking back’ on previous decisions.”

“This idea has its roots in Gestalt psychology [20] and is a standard approach towards robot navigation [21] but generalizes towards optimal path planning in mobile robots [22] and social interaction in humanoid robots [23].”

“The CL of DAC models how state-action sequences that lead to reinforcement can be stored and increasingly support more complex goal-directed strategies [25,27].”

“The CL of DAC models how state-action sequences that lead to reinforcement can be stored and increasingly support more complex goal-directed strategies [25,27].”

“This transition towards allocentric goal-oriented navigation leads to robust robot multi-modal (proximity sensors, chemosensing and vision) foraging including recovery from kidnapping and noise [35] and optimal maze navigation in the presence of distracters [36].”

“It is still largely unknown how such a drive for novel information or exploration (or reduction of uncertainty [52–55]) would be realized by the brain, although some higher-level control systems have been implicated (e.g. prefrontal cortex and hippocampus; see below).”

“Some of these architectures also include a (simplified) internal physiology that grounds higher-level goals (Verschure et al., 2014).”

“Calling on early cybernetic models [16–18], the view that the brain uses control hierarchies has inspired many recent proposals [19–25]; for example, hierarchical temporal structures [26], hierarchical reinforcement learning [9], hierarchical mixture of experts [23], distributed adaptive control [8,27] and hierarchica…”