Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 42 of the 52 sources its claims cite were checked for support.

How multisensory neurons solve causal inference.

Open access
Rideaux, Reuben, Storrs, Katherine R., Maiello, Guido, Welchman, Andrew E. · 2021 · Proceedings of the National Academy of Sciences

Abstract

Sitting in a static railway carriage can produce illusory self-motion if the train on an adjoining track moves off. While our visual system registers motion, vestibular signals indicate that we are stationary. The brain is faced with a difficult challenge: is there a single cause of sensations (I am moving) or two causes (I am static, another train is moving)? If a single cause, integrating signals produces a more precise estimate of self-motion, but if not, one cue should be ignored. In many cases, this process of causal inference works without error, but how does the brain achieve it? Electrophysiological recordings show that the macaque medial superior temporal area contains many neurons that encode combinations of vestibular and visual motion cues. Some respond best to vestibular and visual motion in the same direction ("congruent" neurons), while others prefer opposing directions ("opposite" neurons). Congruent neurons could underlie cue integration, but the function of opposite neurons remains a puzzle. Here, we seek to explain this computational arrangement by training a neural network model to solve causal inference for motion estimation. Like biological systems, the model develops congruent and opposite units and recapitulates known behavioral and neurophysiological observations. We show that all units (both congruent and opposite) contribute to motion estimation. Importantly, however, it is the balance between their activity that distinguishes whether visual and vestibular cues should be integrated or separated. This explains the computational purpose of puzzling neural representations and shows how a relatively simple feedforward network can solve causal inference.

Overview

CoChat Summary

TL;DR

A neural network trained to combine or separate visual and vestibular signals for self- and scene-motion estimation recapitulates key multisensory neural and behavioral signatures, including congruent/opposite units and reliability-based cue weighting. Its causal inference is solved by the balance of activity between congruent and opposite MSTd-like units, with both contributing to motion estimates.

Key findings

  • MultiNet achieved high correspondence between actual and estimated self- and scene-motion velocities (all r > 0.95) and causal-inference accuracy of ∼85% across movement directions (chance = 50%).
  • The network developed congruent and opposite MSTd-like units; the distribution of preferred visual–vestibular direction differences was bimodal with peaks at 0° and 180°, similar to macaque MSTd, and congruent but not opposite unit thresholds were near optimal.
  • It reproduced reliability-based cue weighting and causal-inference-related judgment biases: self-motion estimates were biased toward the more reliable cue, and on one-event trials were more biased by visual input, with the opposite pattern for scene-motion on two-event trials.
  • Congruent units primarily promoted combine decisions and self-motion estimates, while opposite units primarily promoted separate decisions and scene-motion estimates; both contributed to both estimates, with cue-sensitivity asymmetries correlating with congruent-scission weights (r = 0.54, CI95% = [0.48, 0.60]) and opposite-fusion weights (r = 0.39, CI95% = [0.34, 0.45]).

Methods

  • MultiNet had visual and vestibular streams: visual input passed through a convolutional V1 layer and fully connected MT layer; vestibular input passed through a fully connected PIVC layer; both converged on a fully connected MSTd layer, decoded by eight fusion/scission regression units and four binary causal-inference units for vx, vy, vz, and vr.
  • Training used 64,000 randomly generated visual–vestibular input combinations over 50 epochs, with a 75/25 train/test split, minibatch gradient descent (batch size 32, learning rate 1.0 × 10−4); visual input was natural image sequences (64 × 64 × 6 pixels, ±4 pixels/frame) and vestibular input was four noisy Gaussian distributions.
  • Analyses compared network outputs to extracted/replotted macaque and human data, used bootstrapping (50,000 iterations) for 95% CIs, Pearson correlations, ROC/sigmoid threshold estimation, and artificial lesion decoding.

Limitations

  • Stated: biological and artificial systems differ greatly; the model covers only a small portion of one sensory hierarchy, is optimized for a single task, lacks meaningful energy constraints, and uses artificial neurons that are radically simpler than biological neurons.
  • Stated: vestibular signals were abstracted and defined by velocity even though biological vestibular systems are sensitive to acceleration; visual input comprised whole-field self- and scene-motion contributions, not spatially localized scene motion; generality to other sensory pairings remains untested.
  • Evident: the architecture is exclusively feedforward, so it cannot directly test feedback accounts of causal inference; causal-inference outputs were interpreted as binary above/below 0.5, and comparisons to biology rely on previously published data rather than new biological experiments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2106235118

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. Solving causal inference with a multisensory neural network. (A) Two cases (Top and Bottom) illustrate how causal inference must be solved to estimate movement through the environment. In the first case (Top), you are sitting on a train and receive visual and vestibular signals that you are moving forward. These unisensory signals can be represented as probability distributions of velocity. The signals should be combined to produce a more accurate representation of your movement (self-motion, green distribution), it is assumed that the environment is not moving (scene motion, magenta distribution), and you infer the signals were caused by the same event (i.e., your train is moving forward). Note, semitransparent visual and vestibular signals are shown in the “multisensory representation” plot for reference. In the second case (Bottom), you receive a visual signal that you are moving forward and a vestibular signal that you are stationary. The signals should be kept separate, self- and scene motion are estimated from vestibular and the difference between vestibular and visual signals, respectively, and you infer that they were caused by different events, (i.e., the other train is moving). (B) The network comprised two input streams: visual (blue) and vestibular (orange). The visual stream consisted of an input layer, a convolutional (V1) layer and a fully connected (MT) layer. The vestibular stream consisted of an input layer and a fully connected (PIVC) layer. These streams converged on a common fully connected (MSTd) layer. The activity of the MSTd layer was decoded by four “fusion” regression units to produce estimates of (integrated) self-motion, four “scission” regression units to produce estimates of scene motion, and four binary “causal inference” units to decide whether the motion was produced by the same, or different, events, for each (vx, vy, vz, and vr) axis of velocity. The numbers associated with each layer indicate their dimensions (e.g., there are four vestibular inputs that are each represented by 32 units).
Fig. 1. Solving causal inference with a multisensory neural network. (A) Two cases (Top and Bottom) illustrate how causal inference must be solved to estimate movement through the environment. In the first case (Top), you are sitting on a train and receive visual and vestibular signals that you are moving forward. These unisensory signals can be represented as probability distributions of velocity. The signals should be combined to produce a more accurate representation of your movement (self-motion, green distribution), it is assumed that the environment is not moving (scene motion, magenta distribution), and you infer the signals were caused by the same event (i.e., your train is moving forward). Note, semitransparent visual and vestibular signals are shown in the “multisensory representation” plot for reference. In the second case (Bottom), you receive a visual signal that you are moving forward and a vestibular signal that you are stationary. The signals should be kept separate, self- and scene motion are estimated from vestibular and the difference between vestibular and visual signals, respectively, and you infer that they were caused by different events, (i.e., the other train is moving). (B) The network comprised two input streams: visual (blue) and vestibular (orange). The visual stream consisted of an input layer, a convolutional (V1) layer and a fully connected (MT) layer. The vestibular stream consisted of an input layer and a fully connected (PIVC) layer. These streams converged on a common fully connected (MSTd) layer. The activity of the MSTd layer was decoded by four “fusion” regression units to produce estimates of (integrated) self-motion, four “scission” regression units to produce estimates of scene motion, and four binary “causal inference” units to decide whether the motion was produced by the same, or different, events, for each (vx, vy, vz, and vr) axis of velocity. The numbers associated with each layer indicate their dimensions (e.g., there are four vestibular inputs that are each represented by 32 units).CC BY · PubMed Central
Fig. 2. Reliability-based cue weighting and relationship between causal inference and motion judgements. (A) Macaques were presented with vestibular (moving platform) and visual cues (dot motion) signaling forward movement at a range of offsets around straight forward. A small cue conflict was introduced by displacing cues (±4°) in opposite directions from the test offset. The macaques’ task was to indicate whether the direction of movement was to the left or right of straight ahead. (B, Top) Single-cue performance on the task using just vestibular (brown dots) cues and visual cues with high (dark blue squares) or low (light blue triangles) reliability. (C, Top) Performance on the task using combined cues with no conflict (black), −4° (green), or 4° (orange) conflict when the reliability of the visual cue is high (C, Top Left) or low (C, Top Right). (B and C, Bottom) Same as B and C, Top but from self-motion estimates decoded from MultiNet; note, for the network we used a cue conflict of ±9°. Data in B and C, Top extracted and replotted from ref. 9; the illustration shown in A is also adapted from (9). (D) Data from ref. 18 showing human self-motion judgement biases of dot motion stimuli consistent with forward motion containing a localized region in which dots move to the right. Average biases are shown for trials on which participants judged there was just global motion (one event) or an additional local motion cue (two events). Error bars indicate SEM. (E) Same as D but for MultiNet self-motion judgements of visual and vestibular cues on the decision boundary of causal inference. Error bars indicate bootstrapped 95% CIs. (F) Distribution of self- (Left) and scene-motion (Right) MultiNet estimates for trials on which the network judged there to be either one or two events. Solid lines indicate rolling averages across five histogram bins, and arrows indicate the velocities of vestibular and visual inputs. (G) The average vestibular input across trials that the network judged there was either one (green) or two (magenta) events. Shaded regions indicate 95% CIs. (H) Illustration of possible information flow resulting in biased judgements. The decision of causal inference, based on sensory input, may influence judgements of self-/scene motion (yellow arrows). Alternatively, both causal inference and self-/scene-motion judgements may be decoded from sensory input (cyan arrows). The latter is how MultiNet operates (i.e., self-/scene motion and causal inference are decoded from MSTd activity). To implement the former process, the self-/scene-motion units in MultiNet could either be placed after the causal inference units or be influenced by them via lateral/feedback connections.
Fig. 2. Reliability-based cue weighting and relationship between causal inference and motion judgements. (A) Macaques were presented with vestibular (moving platform) and visual cues (dot motion) signaling forward movement at a range of offsets around straight forward. A small cue conflict was introduced by displacing cues (±4°) in opposite directions from the test offset. The macaques’ task was to indicate whether the direction of movement was to the left or right of straight ahead. (B, Top) Single-cue performance on the task using just vestibular (brown dots) cues and visual cues with high (dark blue squares) or low (light blue triangles) reliability. (C, Top) Performance on the task using combined cues with no conflict (black), −4° (green), or 4° (orange) conflict when the reliability of the visual cue is high (C, Top Left) or low (C, Top Right). (B and C, Bottom) Same as B and C, Top but from self-motion estimates decoded from MultiNet; note, for the network we used a cue conflict of ±9°. Data in B and C, Top extracted and replotted from ref. 9; the illustration shown in A is also adapted from (9). (D) Data from ref. 18 showing human self-motion judgement biases of dot motion stimuli consistent with forward motion containing a localized region in which dots move to the right. Average biases are shown for trials on which participants judged there was just global motion (one event) or an additional local motion cue (two events). Error bars indicate SEM. (E) Same as D but for MultiNet self-motion judgements of visual and vestibular cues on the decision boundary of causal inference. Error bars indicate bootstrapped 95% CIs. (F) Distribution of self- (Left) and scene-motion (Right) MultiNet estimates for trials on which the network judged there to be either one or two events. Solid lines indicate rolling averages across five histogram bins, and arrows indicate the velocities of vestibular and visual inputs. (G) The average vestibular input across trials that the network judged there was either one (green) or two (magenta) events. Shaded regions indicate 95% CIs. (H) Illustration of possible information flow resulting in biased judgements. The decision of causal inference, based on sensory input, may influence judgements of self-/scene motion (yellow arrows). Alternatively, both causal inference and self-/scene-motion judgements may be decoded from sensory input (cyan arrows). The latter is how MultiNet operates (i.e., self-/scene motion and causal inference are decoded from MSTd activity). To implement the former process, the self-/scene-motion units in MultiNet could either be placed after the causal inference units or be influenced by them via lateral/feedback connections.CC BY · PubMed Central
Fig. 3. Multisensory tuning properties of MultiNet MSTd units. (A and B, Top) The responses of two macaque MSTd neurons to visual (blue) and vestibular (orange) cues as a function of heading direction. The tuning of one neuron is the same for both cues (A, congruent neuron), while the tuning of the other neuron is different (B, opposite neuron). (C, Top) Same as A, Top but for a narrower range of heading directions around straight forward, including the response to both cues at the same time (green) and the response predicted by a linear summation model (gray). (D, Top) The ratio between the neuronal threshold of macaque MSTd neurons for congruent visual and vestibular cues to that predicted by optimal cue integration as a function of the congruency between their tuning for these cues in isolation. Filled symbols denote neurons for which the congruency index significantly differs from zero. Triangle and circles denote data from different monkeys. The bottom of A through D shows the same analysis as the top but for MultiNet MSTd units. (E) Average neuronal thresholds for (left) congruent and (right) incongruent neurons. Error bars indicate SEM. (F) Same as E but for MultiNet MSTd units. (G, Left) The distribution of macaque MSTd neurons as a function of the difference between preferred visual and vestibular heading direction. (G, Right) Same as G, Left but for MultiNet MSTd units. Data shown in A through D, Top and E were extracted from ref. 8, and data shown in G, Left were extracted from 6. Error bars in E and F indicate SEM between neurons and bootstrapped 95% CIs, respectively.
Fig. 3. Multisensory tuning properties of MultiNet MSTd units. (A and B, Top) The responses of two macaque MSTd neurons to visual (blue) and vestibular (orange) cues as a function of heading direction. The tuning of one neuron is the same for both cues (A, congruent neuron), while the tuning of the other neuron is different (B, opposite neuron). (C, Top) Same as A, Top but for a narrower range of heading directions around straight forward, including the response to both cues at the same time (green) and the response predicted by a linear summation model (gray). (D, Top) The ratio between the neuronal threshold of macaque MSTd neurons for congruent visual and vestibular cues to that predicted by optimal cue integration as a function of the congruency between their tuning for these cues in isolation. Filled symbols denote neurons for which the congruency index significantly differs from zero. Triangle and circles denote data from different monkeys. The bottom of A through D shows the same analysis as the top but for MultiNet MSTd units. (E) Average neuronal thresholds for (left) congruent and (right) incongruent neurons. Error bars indicate SEM. (F) Same as E but for MultiNet MSTd units. (G, Left) The distribution of macaque MSTd neurons as a function of the difference between preferred visual and vestibular heading direction. (G, Right) Same as G, Left but for MultiNet MSTd units. Data shown in A through D, Top and E were extracted from ref. 8, and data shown in G, Left were extracted from 6. Error bars in E and F indicate SEM between neurons and bootstrapped 95% CIs, respectively.CC BY · PubMed Central
Fig. 4. Solving causal inference with an artificial network. (A) The difference between the average response of congruent and opposite MultiNet MSTd units to different combinations of visual and vestibular signals. Positive (red) values indicate congruent units are more active, and negative (blue) indicate opposite units are more active. Note that activity is weaker at the center (around 0° direction; straight ahead motion), because the congruent/opposite classification method is not designed to categorize units tuned to near-zero directions. (B, Left) The average weights between congruent and opposite MSTd units and the causal inference decision units. (B, Right) The same as B, Left but for absolute weights between MSTd units and fusion/scission regression units. (C) The weights between (opposite/congruent) MSTd units and (fusion/scission) regression units as a function of their preferred visual and vestibular velocity. The average weights, binned as a function of the velocity to which MSTd units were maximally responsive, are shown above and to the right of scatter plots. Error bars in B and C indicate bootstrapped 95% CIs. (D, Top) Self- and (Bottom) scene-velocity estimated by the network in response to a range of visual–vestibular inputs decoded from the activity of all the MSTd units, the congruent units, or the opposite units. The final column shows the difference in (normalized) velocity estimates from congruent and opposite units (opposite minus congruent). All visual velocities are in pixels/frame and vestibular velocities are equivalently mapped to visual velocity; weights are in arbitrary units (a.u.).
Fig. 4. Solving causal inference with an artificial network. (A) The difference between the average response of congruent and opposite MultiNet MSTd units to different combinations of visual and vestibular signals. Positive (red) values indicate congruent units are more active, and negative (blue) indicate opposite units are more active. Note that activity is weaker at the center (around 0° direction; straight ahead motion), because the congruent/opposite classification method is not designed to categorize units tuned to near-zero directions. (B, Left) The average weights between congruent and opposite MSTd units and the causal inference decision units. (B, Right) The same as B, Left but for absolute weights between MSTd units and fusion/scission regression units. (C) The weights between (opposite/congruent) MSTd units and (fusion/scission) regression units as a function of their preferred visual and vestibular velocity. The average weights, binned as a function of the velocity to which MSTd units were maximally responsive, are shown above and to the right of scatter plots. Error bars in B and C indicate bootstrapped 95% CIs. (D, Top) Self- and (Bottom) scene-velocity estimated by the network in response to a range of visual–vestibular inputs decoded from the activity of all the MSTd units, the congruent units, or the opposite units. The final column shows the difference in (normalized) velocity estimates from congruent and opposite units (opposite minus congruent). All visual velocities are in pixels/frame and vestibular velocities are equivalently mapped to visual velocity; weights are in arbitrary units (a.u.).CC BY · PubMed Central

Verification

Claims the paper makes

17 Valid · 3 Not valid · 4 Not checked · 63 Own results and interpretations

56 more claims · 6 with their cited sources checked

Create a free account to see all 87 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 12
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 48 references

Cited by

See all 74

Metadata

Authors
Rideaux, R., Storrs, K. R., Maiello, G., & Welchman, A. E.
Affiliations
  • The University of Queensland (AU)
  • Justus-Liebig-Universität Gießen (DE)
  • University of Cambridge (GB)
Corresponding author
Rideaux, Reuben
Venue
Proceedings of the National Academy of Sciences
Published
2021-08-04
Volume
118
Issue
32
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W3192206917
Cited by
65
References
48
Citations by year
2026: 9 · 2025: 18 · 2024: 16 · 2023: 11 · 2022: 8 · 2021: 3
FWCI
9.17
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
  • ePrints Soton (University of Southampton) (acceptedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • Apollo (cc-by)
Funders
  • Australian Research Council (DE210100790)
  • Leverhulme Trust (ECF-2017-573)
  • Isaac Newton Trust (17.08(o))
  • Deutsche Forschungsgemeinschaft (SFB-TRR-135)
  • Alexander von Humboldt-Stiftung (#)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Life Sciences › Neuroscience › Neurology › Vestibular and auditory disorders
Keywords
causal inference, multisensory integration, self-motion perception, illusory self-motion, medial superior temporal area, neural network model, feedforward neural networks, cue integration, macaque
MeSH terms
Animals, Cues, Macaca mulatta, Motion Perception, Photic Stimulation, Sensory Receptor Cells, Temporal Lobe, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: How multisensory neurons solve causal inference. https://app.cochat.ai/papers/10.1073/pnas.2106235118
The paper
Rideaux, R., Storrs, K. R., Maiello, G., & Welchman, A. E. (2021). How multisensory neurons solve causal inference. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2106235118

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.