The emulation theory of representation: Motor control, imagery, and perception

“Indeed, most of the brain is known to be involved to some extent in aspects of movement such as sensorimotor coordination, movement planning, and execution, which entail offline emulation of the body and the environment, 50 required for efficient solving of various kinematic and dynamic tasks.”

“Furthermore, it has been proposed that motor ‘emulations’ (e.g., motor plans) provide feedback to the ‘controller’ (e.g., cognitive system) that is less delayed than the feedback from overt actions (Grush, 2004).”

“In fact, not only is the inner loop active while the agent takes action, but it can also be decoupled from the original sensorimotor loop and run off-line, in simulation of potential action(s) to support motor planning, preparation, and imagery (Grush, 2004; Jeannerod, 2001, 2006; Pezzulo and Castelfranchi, 2007, 2009…”

“In the former category, Grush [38] has described a framework based on forward modelling and Kalman-filter signal processing in which neural circuits act as models of body–environment interactions.”

“Consequently, the current MST framework would need to be extended to account for this possibility, perhaps borrowing from emulation theory which promotes the idea that both motor and sensory systems are simulated (Grush, 2004).”

“The idea that the elements of thought are not words or symbols, but visual and motor images is at the core of recent theories of cognition (e.g., Barsalou, 1999b; Glenberg, 1997; Grush, 2004; Zwaan, 1999), that are commonly referred to by the label grounded cognition.”

“MI thus involves motor simulation, i.e., predictive models derived from the efferent copy of the motor command (Grush 2004; Wolpert and Flanagan 2001; Demougeot and Papaxanthis 2011; Wolpert and Ghahramani 2000).”

“We observed that elicited inner speaking caused a decrease in activation of Heschl’s gyrus, in accord with prior elicited-inner-speech research and with some understandings of external speech, which hold that when we talk, there is a reduction in sensitivity to our own speech sounds [36].”

“Conceptual structures of a cognitive agent can also include 'nested' simulations (Grush, 2004) to conceive phenomena which progressively achieve higher levels of abstraction and detachment.”

“It has been proposed that MI is a simulation based on own motor representations that are also used while executing an action (Jeannerod 2001; Grush 2004).”

“14 See also Grush (2004).”

“(for an overview and further references, see Kosslyn 1994; Kind 2006; Grush 2004; Currie and Ravenscroft 2002:71ff).”

“Thus, as established for the domain of action performance, imagery, and observation, motor networks may be considered providing forward models that enable attention to change beyond the action domain [Grush, 2004; Wolpert and Flanagan, 2001].”

“Motor imagery is a cognitive ability commonly defined as ‘menal simulation’ or ‘mental rehearsal’ of movements without actual ovements (Decety, 1996; Grush, 2004; Jeannerod, 1994).”

“When considering these findings along with Grush's ( 2004 ) emulation theory of representation, it is conceivable that improving inhibitory abilities could account for the developmental trend we observed in compatibility effects.”

“…proximal and distal sensory consequences of an action, and which can serve in comparisons with either actual sensory feedback or the desired state of the agent (for predictive in-the-head models of cognition in general also see Wolpert et al. 2003; Knill and Pouget 2004; Grush 2004; or Seth 2014).”

“…that objects can be recognized in terms of the expected sensory consequences of possible actions produced by forward models; for example, a sponge can be understood in terms of the (anticipated) softness when squeezing it or imagining squeezing it (Grush, 2004; Pezzulo, 2008, 2011; Roy, 2005).”

“Functional and effective connectivity patterns alter rapidly as we perform our cognitive tasks, with changes occurring within hundreds of milliseconds.”

“…move away from seeing mental representations as indicators or detectors, and towards construing them in terms of internal models (Bartels 2006; Grush 2004; Gładziejewski 2016a, b; Isaac 2012; O’Brien 2014; O’Brien and Opie 2004; Ramsey 2007; Rescorla 2009; Ryder 2004; Shagrir 2012; Shea 2014;…”

“Model 2), it can exploit offline computation of 935 the likely consequences of different policies under various hypothetical conditions 936 (Gärdenfors, 1995; Grush, 2004; Pezzulo, 2008).”

“In a way, the whole brain can function as a simulation machine, with different modality areas being recruited depending on the goals in a particular task (Grush, 2004).”

“…engage partly overlapping and partly distinct cortical networks (Herholz et al. 2012) and have been described as parallel processes, such that imagery represents the " offline " use of the top-down model of expected outcomes from perception and action (Grush 2004; Rauschecker and Scott 2009).”

“…theories claim that structures represent 3 Thisway of putting thingsmay go to the heart of some of themore radical formulations of the implications of PEM-style accounts for the nature of the mind-world relation—for example, claims that perception is “controlled hallucination” (Grush 2004).”

“…for predictive coding (a general proposal about neural functioning; see Clark, 2013), is consistent with the proposal that the brain uses bidirectional feedback, formed through associative processes, to self-generate predictions (i.e. simulations) in a probabilistic fashion (see also Grush, 2004).”

“Frontiers in Human Neuroscience www.frontiersin.org July 2013 | Volume 7 | Article 415 | 1 HUMAN NEUROSCIENCE The idea developed by Grush (2004) is that in addition to simply engaging with the body and environment, the brain constructs neural circuits that act as models of the body and environment.”

“It has been stated that such a (covert) motor simulation may facilitate the prediction of others’ actions (e.g., Wolpert and Flanagan 2001; Wolpert et al. 2003; Grush 2004; Wilson 2001, 2005; Wilson and Knoblich 2005; Blakemore and Frith 2005; Prinz 2006) and is therefore of obvious adaptive value.”

“The idea of coupling between action execution and anticipation of its effects is also central to recent theories of internal modeling (Grush, 2004; Kawato, 1999; Ito, 1993; Miall and Wolpert, 1996; Oztop et al., 2006; Wolpert and Ghahramani, 2004), which we adopt here in continuity with the ideas…”

“Applying the action simulation account to the present study, suggests that motor programs involved in controlling right arm movements were activated during occlusion and produced predictions of the sensory consequences of motor actions (Grush, 2004; Wilson and Knoblich, 2005).”

“Specifically, growing interdisciplinary work has now begun to relate different theoretical approaches and empirical findings to explain also highercognitive skills like mind reading in social contexts (Gallese & Goldman, 1998) or mental imagery (Grush, 2004).”

“This also explains the disappearance of foresight and agency in the FM (Dietrich, 2015), as evidenced by the quote of Philip Guston, since these experiences also depend on prediction processes (Frith, 1992; Grush, 2004).”

“At a processing level, simulation (emulation) theories (Grush, 2004; Jeannerod, 2001; Wilson & Knoblich, 2005) posit the existence of internal models that act as predictors for action outcomes.”

“Similarly, as Grush would be the first to admit, his 'emulation theory' did not spring fully formed from the head of Zeus, but rather offers a synthesis of existing work on the neuroscience of motor control; 'emulator' is effectively Grush's term for a kind of theoretical posit—a forward model—that is ubiquitous in ma…”

“…internalize certain action programs and apply them to mentally simulated objects and situations instead (Frick & M€ohring, 2016; Frick, M€ohring, & Newcombe, 2014; Funk, Brugger, & Wilkening, 2005; Grush, 2004; Ito, 2008; Jansen & Heil, 2010; Jeannerod, 2001; Lehmann, Quaiser-Pohl, & Jansen, 2014).”

“One of the key issues in cognition is the importance of internal simulation in accelerating the scaffolding of this early developmentally-acquired sensorimotor knowledge to provide a means to predict future events, to reconstruct or explain observed events (constructing a causal chain leading to that event), or to ima…”

“the kinematics of the body ( Grush, 2004 ) so as to match the incoming proprioceptive and other”

“can have a dual role in facilitating action understanding, which is essential to plan complementary actions, and at the same time afford prediction of the timing of actions, the trajectory of movements, and in general better perceptual processing (Grush 2004; Wilson and Knoblich 2005), all of which are useful for real…”

“This ‘emulator’ (Grush, 2004) is controlled by feedback from a Kalman filter, weighing predictions against analysis of the speech input”

“According to Grush (2004)’s emulation theory, representation is the ability to emulate internally part of external reality by means of internal models such as Kalman filters, which can also ‘nest’ and realize abstraction.”

“…domain of representational contents is represented itself by a temporal succession of events in the domain of the representational vehicles—in our case neural processes (for simulation or emulation accounts of mental representation see Grush 2004; Werning 2012; Mroczko-Wąsowicz and Werning 2012).”

“An emulation represents an activity by reenacting it in a circumscribed and modified way – for example, as an internal process that models, but does not loop through, peripheral sensory and motor systems (Grush, 2004).”

“By generating an efferent copy of the motor commands, the motor system can simulate the motor act and therefore, predict and anticipate the sensory consequences of self-generated movements through an emulation process (Grush, 2004).”

“This theory is somewhat similar to the theory of kinesthetic imagery proposed by Grush (2004), and it finds parallels in the theory of Levelt (1983) and has been supported by brain-imaging studies (Kauramäki et al., 2010; Numminen & Curio, 1999; Tian & Poeppel, 2010) and behavioral research (Scott,…”

“This is consistent with the embodied cognition view on mental representation (e.g., Barsalou, 1999, 2008a; Grush, 2004), which proposes that concepts are grounded in sensory-motor processing.”

“DOI: https://doi.org/10.7554/eLife.53588 10 of 19 Grush, 2004; Hobson and Friston, 2012; Moulton and Kosslyn, 2009).”

“It has been suggested that a particularly efficient way of achieving action prediction may be based on exploiting predictive functions of the observer’s own motor system in a simulation mode [Blakemore and Decety, 2001; Grush, 2004; Wilson and Knoblich, 2005].”

“Although enriched by a variety of tools, such as the use of off-line emulators and filtering techniques (Grush, 2004), negative feedback remains at the center of the modern field of control theory.”

“4 Grush (2004) explicitly argues for such an account of neural representation, applying it to”

“In the last decades similar views were thoroughly developed under the terms “simulation” (Hesslow, 2002, 2012) and “emulation” (Grusz, 2004).”

“[72]) which consider mental imagery to be strongly coupled to actions.”

“They monitor the system as a whole, gathering feedback on its dispositions to behaviour, whether by observation or by using a predictive or simulative device (Grush 2004).”

“The simulation theory of cognition [1] says that humans learn forward and inverse models of the plants they are going to control [22, 23] (starting with, but not limited to, their own body), so that sensorial consequences of actionsmay be predicted, and actions that achieve desired perceptual goals may be produced [1,…”

“…Vinson, Lewis, & Garrett, 2004)—that is, grounded in sensory–motor processing—as has been advocated by the embodied-cognition view on mental representation (see, e.g., Barsalou, 1999, 2008; Grush, 2004) and confirmed by neurophysiological studies (Martin & Chao, 2001; Vigliocco et al., 2006).”

“This results in attenuated somatosensory erception, even when motor plans are generated but not exeuted (Voss, Ingram, Wolpert, & Haggard, 2008) or merely imagined Cheron & Borenstein, 1992; Grush, 2004).”

“This interpretation refers to the well-known control theory framework [24,25].”

“According to motor control theory, these steps imply the generation of predictive internal forward and inverse models that link motor intentions, motor commands and sensory effects (Wolpert et al. 1995; Wolpert and Miall 1996; Kawato 1999; Grush 2004; Shadmehr and Krakauer 2008 ).”

“An emerging organizing theme in neuroscience is that the brain has evolved, fundamentally, to make predictions (e.g., Bar, 2007, 2009; Friston & Stephan, 2007; Grush, 2004; Llinas, 2001; Pezzulo, Butz, Castelfranchi, & Falcone, 2008).”

“Grush (2004) extended the formal work of Wolpert and Kawato by proposing an articulated, organism/environment forward model.”

“This can be regarded a non-symbolic, emulative way of representation (Grush, 2004; Mroczko-Wąsowicz & Werning, 2012; Werning, 2012).”

“Our simple model agrees with the core functions for self-monitoring and speech control in other computational models (e.g., Hickok, 2012; Houde & Nagarajan, 2011; Guenther, Ghosh, & Tourville, 2006; Grush, 2004).”

“This type of “dynamic predictive coding” model (Clark, 2013) has been fruitfully applied to perception-action systems (Grush, 2004; Friston, 2005) as well as interoceptive systems (Paulus and Stein, 2010; Seth et al.”

“Being dynamical systems with representational powers, they may also serve as forward models and generate predictions (Grush, 2004; Bartels, 2006).”

“, 2006) or other theories that emphasize predictions, simulations and emulations (e.g., Bar, 2007; Grush, 2004; Hesslow, 2002; Jeannerod, 2006) than the traditional information-processing accounts that describe cognition in terms of a transformation of stimuli into patterns of action through a sequence of stages.”

“…process in question which permits predicting and adjusting future cognitive performance on a given task (Proust 2007, 2008, 2009a; Peacocke 2007, 2008, 2009), as happens in the case of bodily actions where the subject runs an offline motor program to predict her future performance (Grush 2004).”

“This enables an agent to detach itself from the environment and perform non-reflexive behavior such as planning future actions through mental simulations of the environment (Grush 2004).”

“This could be interestingly related to the proposals that treat the cerebellum as an “emulator”—a device that mimics and predicts the outcomes of motor actions (Grush, 2004).”

“…trajectory of one system unfolds in lockstep with that of the other system, because the dynamics of the two systems are sufficiently close to one another; for example, formal neurons can be wired up into a network whose dynamics would emulate (Grush, 2004) that of the falling rock mentioned above.”

“This process provides stability to the motor system by predicting and anticipating the sensory consequences of self-generated movements (Grush, 2004; Pelgrims, Andres, & Olivier, 2005; Wolpert, 1997).”