CoChat

Computing Arm Movements with a Monkey Brainet

Open access

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.

Abstract

Traditionally, brain-machine interfaces (BMIs) extract motor commands from a single brain to control the movements of artificial devices. Here, we introduce a Brainet that utilizes very-large-scale brain activity (VLSBA) from two (B2) or three (B3) nonhuman primates to engage in a common motor behaviour. A B2 generated 2D movements of an avatar arm where each monkey contributed equally to X and Y coordinates; or one monkey fully controlled the X-coordinate and the other controlled the Y-coordinate. A B3 produced arm movements in 3D space, while each monkey generated movements in 2D subspaces (X-Y, Y-Z, or X-Z). With long-term training we observed increased coordination of behavior, increased correlations in neuronal activity between different brains and modifications to neuronal representation of the motor plan. Overall, performance of the Brainet improved owing to collective monkey behaviour. These results suggest that primate brains can be integrated into a Brainet, which self-adapts to achieve a common motor goal.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/srep10767

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Experimental setups for B2 and B3 experiments. (A) Monkeys were seated in separate rooms, each facing a computer monitor showing the virtual avatar arm (inset in C) from a 1st person perspective. (B) Shows the shared control task, (X,Y) position of the virtual arm was decoded during centre-out movements from the two monkeys’ brains with each given 50% control of the arm. Electrode array location shown on brains. (C) Shows the partitioned control task. X position of the arm was decoded from one monkey and Y position from the other during centre-out movements toward targets. (D) Shows the 3-monkey task. Each monkey observed and had 50% control over 2 of the 3 dimensions (X, Y, or Z). Together, the three monkeys must accurately perform a 3-D centre-out movement to achieve reward. Drawings by Miguel A.L. Nicolelis.
Figure 1 Experimental setups for B2 and B3 experiments. (A) Monkeys were seated in separate rooms, each facing a computer monitor showing the virtual avatar arm (inset in C) from a 1st person perspective. (B) Shows the shared control task, (X,Y) position of the virtual arm was decoded during centre-out movements from the two monkeys’ brains with each given 50% control of the arm. Electrode array location shown on brains. (C) Shows the partitioned control task. X position of the arm was decoded from one monkey and Y position from the other during centre-out movements toward targets. (D) Shows the 3-monkey task. Each monkey observed and had 50% control over 2 of the 3 dimensions (X, Y, or Z). Together, the three monkeys must accurately perform a 3-D centre-out movement to achieve reward. Drawings by Miguel A.L. Nicolelis.CC BY · PubMed Central
Figure 2 B2 shared control task. (A,B) Boxplots comparing target acquisition time (left panel), trial duration (centre panel), inter-reward interval (right panel) in hand control (A) and in brain control (B). (C,D) Changes in trial duration with conjoint training across weeks. Mean ± SEM trial duration for each of the four weeks of hand control experiments (C) and three weeks of brain control (D) for dyad M&O. Dashed line shows linear regression fit. Trial duration reduced significantly with improvement in coordination between monkeys. (E,F) Trial-averaged movement profiles from an early session (dashed) and a late session (solid) for monkey M (red) and O (blue) aligned to the time of target onset (time = 0 ms.). Plots are shown up to 1.5 s for hand control (with mean target acquisition time of 0.8 s) and 3 s for brain control (with mean target acquisition time of 2.2 s). The target was located 9 cm from the centre (y-axis). As the dyad trained together over a span of 7 weeks the reaction time lag, the time of movement onset of monkey O relative to monkey M, decreased during both hand control (E) and brain control (F) (G,H) Neural activity PETH from an early session and a late session for hand control (G) and brain control (H) Each row is a single neuron, colour denotes normalized firing amplitude. Monkey M and O neurons marked by black (M) and grey (O) vertical bands. Notice that neuronal modulations are more intense and synchronized across the monkey pair in the later session as compared to the early session. (I) Changes in reaction time lag between monkey M and O across experiments. Lag was derived from peak of cross-correlation of two monkeys’ behavioural traces (E,F) Trends fit with linear regression. As the dyad trained together the reaction time lag reduced to 10 ± 27 ms. (J) Lag in neural activity between two monkeys over the same experiments as in (E) computed again by finding the peak of cross correlogram on each session. As the dyad trained together the reaction time lag reduced to 4.2 ± 5 ms.
Figure 2 B2 shared control task. (A,B) Boxplots comparing target acquisition time (left panel), trial duration (centre panel), inter-reward interval (right panel) in hand control (A) and in brain control (B). (C,D) Changes in trial duration with conjoint training across weeks. Mean ± SEM trial duration for each of the four weeks of hand control experiments (C) and three weeks of brain control (D) for dyad M&O. Dashed line shows linear regression fit. Trial duration reduced significantly with improvement in coordination between monkeys. (E,F) Trial-averaged movement profiles from an early session (dashed) and a late session (solid) for monkey M (red) and O (blue) aligned to the time of target onset (time = 0 ms.). Plots are shown up to 1.5 s for hand control (with mean target acquisition time of 0.8 s) and 3 s for brain control (with mean target acquisition time of 2.2 s). The target was located 9 cm from the centre (y-axis). As the dyad trained together over a span of 7 weeks the reaction time lag, the time of movement onset of monkey O relative to monkey M, decreased during both hand control (E) and brain control (F) (G,H) Neural activity PETH from an early session and a late session for hand control (G) and brain control (H) Each row is a single neuron, colour denotes normalized firing amplitude. Monkey M and O neurons marked by black (M) and grey (O) vertical bands. Notice that neuronal modulations are more intense and synchronized across the monkey pair in the later session as compared to the early session. (I) Changes in reaction time lag between monkey M and O across experiments. Lag was derived from peak of cross-correlation of two monkeys’ behavioural traces (E,F) Trends fit with linear regression. As the dyad trained together the reaction time lag reduced to 10 ± 27 ms. (J) Lag in neural activity between two monkeys over the same experiments as in (E) computed again by finding the peak of cross correlogram on each session. As the dyad trained together the reaction time lag reduced to 4.2 ± 5 ms.CC BY · PubMed Central
Figure 3 (A-B) Cross correlogram of Monkey M vs. Monkey O neural activity on correct trials (upper left panel) and error trials (lower left panel) for both a representative hand control (A) and brain control session (B) Mean cross correlogram for the correct and error trial group shown in panel on right of each panel. The correlation is higher in correct trials indicative of increased synchronous neuronal modulations between monkeys. (C,D) k-NN prediction of trial outcome (reward or error) using the mean neuronal cross correlogram on a single trial (i.e. left panels in A-B) between the two monkeys either prior to target appearance (“Pre-target”) or after target appearance (“Post-target”). Chance level prediction (95% confidence interval) shown in yellow band. * denotes P < 0.01, unpaired t-test. Neuronal synchrony between monkeys before and after target onset was predictive of trial outcome. (E,F) Extra correlation analysis. Velocity profile in a trial for dyad M&O were cross-correlated. The average cross correlation (trial specific correlation, red trace) was estimated for all trials: hand control (E) and brain control (F) Extra correlation was the excess correlation (in the red trace) that cannot be accounted for by the distribution of across-trial correlation (enclosed by the grey dashed lines). The vertical line shows the time lag at peak correlation. (G-H) Partitioned control task: (G) Panels show the average trajectory to the 4 target locations, denoted by colour, from the first (left column) and the last (right column) session. Trajectories derived from the monkeys’ controlled axes predictions (controlled) are shown in the upper panels and non-controlled axes predictions (non-controlled) are shown in the lower panels. Non-controlled traces were initially more similar to the controlled traces (left panels) but over time became shrunk and convoluted (right panels). (H) The fraction of rewarded trials was computed based on trials when the avatar arm is moving according to the predictions of the controlled axis (blue) or the non-controlled axes (red). Mean fraction correct trials shown for each of the three weeks of experiments. Shown separately is fraction correct amongst all trials (upper panel) and only among trials where a reward was achieved (lower panel). The percentage of trials in which the complementary trace reached the target (red bars) decreased significantly over training.
Figure 3 (A-B) Cross correlogram of Monkey M vs. Monkey O neural activity on correct trials (upper left panel) and error trials (lower left panel) for both a representative hand control (A) and brain control session (B) Mean cross correlogram for the correct and error trial group shown in panel on right of each panel. The correlation is higher in correct trials indicative of increased synchronous neuronal modulations between monkeys. (C,D) k-NN prediction of trial outcome (reward or error) using the mean neuronal cross correlogram on a single trial (i.e. left panels in A-B) between the two monkeys either prior to target appearance (“Pre-target”) or after target appearance (“Post-target”). Chance level prediction (95% confidence interval) shown in yellow band. * denotes P < 0.01, unpaired t-test. Neuronal synchrony between monkeys before and after target onset was predictive of trial outcome. (E,F) Extra correlation analysis. Velocity profile in a trial for dyad M&O were cross-correlated. The average cross correlation (trial specific correlation, red trace) was estimated for all trials: hand control (E) and brain control (F) Extra correlation was the excess correlation (in the red trace) that cannot be accounted for by the distribution of across-trial correlation (enclosed by the grey dashed lines). The vertical line shows the time lag at peak correlation. (G-H) Partitioned control task: (G) Panels show the average trajectory to the 4 target locations, denoted by colour, from the first (left column) and the last (right column) session. Trajectories derived from the monkeys’ controlled axes predictions (controlled) are shown in the upper panels and non-controlled axes predictions (non-controlled) are shown in the lower panels. Non-controlled traces were initially more similar to the controlled traces (left panels) but over time became shrunk and convoluted (right panels). (H) The fraction of rewarded trials was computed based on trials when the avatar arm is moving according to the predictions of the controlled axis (blue) or the non-controlled axes (red). Mean fraction correct trials shown for each of the three weeks of experiments. Shown separately is fraction correct amongst all trials (upper panel) and only among trials where a reward was achieved (lower panel). The percentage of trials in which the complementary trace reached the target (red bars) decreased significantly over training.CC BY · PubMed Central
Figure 4 Triad (B3) control of 3D movements. (A) Boxplots comparing target acquisition time (left panel), trial duration (centre panel), inter-reward interval (right panel). (B) Reduction in trial duration (left panel) and concurrent increase in the reward rate (right panel) with conjoint training across weeks. (C) Normalized contribution of each of the three monkeys across a representative subset of 30 trials. The relative contribution of each monkey varied from trial to trial. (D) Fraction of trials that were correctly performed by a dyad (black) as a triad (green), or incorrectly (purple) shifted across the 11 triad experiments. The fraction of total trials with a rewarded outcome in which all three monkeys contributed (green), or those in which two monkeys contributed (black) increased significantly within each week and across sessions whereas the fraction of erroneous/unattempted trials reduced significantly. (E,F) Decoded trajectories and neural data from the triad experiment. (E) Mean X,Y,Z traces produced by individual monkeys shown separately by colour among trials where all monkeys contributed (left column) or when only monkeys M and C contributed (right column). Mean X,Y,Z (the value used to move avatar) shown in black. Distance to target in each axis is 5 screen-cm. When one monkey (monkey K) opted out, the working dyad generated higher-amplitude trajectories (Right column, X axis and Z axis) as opposed to when all the members contributed (left column). (F) PETHs aligned on target onset for same trial subsets as in (E) Rows represent individual neurons and colour indicates normalized firing rate (z-score). Neurons from different monkeys marked by colour along right edge (same colours as in (E)). Increased effort by the working dyad also resulted in stronger cortical modulations between the members (right panel) as compared to when all the members contributed (left panel).
Figure 4 Triad (B3) control of 3D movements. (A) Boxplots comparing target acquisition time (left panel), trial duration (centre panel), inter-reward interval (right panel). (B) Reduction in trial duration (left panel) and concurrent increase in the reward rate (right panel) with conjoint training across weeks. (C) Normalized contribution of each of the three monkeys across a representative subset of 30 trials. The relative contribution of each monkey varied from trial to trial. (D) Fraction of trials that were correctly performed by a dyad (black) as a triad (green), or incorrectly (purple) shifted across the 11 triad experiments. The fraction of total trials with a rewarded outcome in which all three monkeys contributed (green), or those in which two monkeys contributed (black) increased significantly within each week and across sessions whereas the fraction of erroneous/unattempted trials reduced significantly. (E,F) Decoded trajectories and neural data from the triad experiment. (E) Mean X,Y,Z traces produced by individual monkeys shown separately by colour among trials where all monkeys contributed (left column) or when only monkeys M and C contributed (right column). Mean X,Y,Z (the value used to move avatar) shown in black. Distance to target in each axis is 5 screen-cm. When one monkey (monkey K) opted out, the working dyad generated higher-amplitude trajectories (Right column, X axis and Z axis) as opposed to when all the members contributed (left column). (F) PETHs aligned on target onset for same trial subsets as in (E) Rows represent individual neurons and colour indicates normalized firing rate (z-score). Neurons from different monkeys marked by colour along right edge (same colours as in (E)). Increased effort by the working dyad also resulted in stronger cortical modulations between the members (right panel) as compared to when all the members contributed (left panel).CC BY · PubMed Central
Figure 5 Neuronal representations during B2 and B3 shared control experiments. Neuronal modulations from monkey M (orange) and C (green) during a 10 second window of passive observation. (A) and brain control (B) Centre and peripheral target onset times are denoted by grey vertical lines. Cortical ensembles in each monkey exhibited clear task-related activity during both passive observation and brain control modes. (C) Passively observed trajectory (black) compared with predicted trajectory using only Monkey M (green), only Monkey C (red), or both (blue) neuronal ensembles. Grey vertical lines from (A) again denote relevant task events. Accuracy of arm movement decoding improved when VLSBA was recorded and combined from multiple brains. (D) Decoded X and Y trajectories from monkey M (black, solid) and monkey C (red), as well as the average of the two (dotted) during 8 second window of brain control experiment. (E,F) Neuron dropping curves (NDC) showing effect of ensemble size of each of the two monkeys on prediction accuracy during B2 passive observation. The number of neurons used from Mk1 marked by x-axis. The percent of Mk2 population used for predictions denoted by colour (see Legend). Accuracy of predictions measured as correlation coefficient r. The decoding accuracy benefited from mixing the contributions from different brains as well as the overall neuronal mass. (G-I) Same NDC analysis as (E,F) except for prediction of X, Y, and Z position during B3 passive observation. Prediction of X (G), Y (H), and Z (I) shown separately. Again (as seen in E-F), the decoding accuracy benefited from mixing the contributions from different brains as well as the overall neuronal mass.
Figure 5 Neuronal representations during B2 and B3 shared control experiments. Neuronal modulations from monkey M (orange) and C (green) during a 10 second window of passive observation. (A) and brain control (B) Centre and peripheral target onset times are denoted by grey vertical lines. Cortical ensembles in each monkey exhibited clear task-related activity during both passive observation and brain control modes. (C) Passively observed trajectory (black) compared with predicted trajectory using only Monkey M (green), only Monkey C (red), or both (blue) neuronal ensembles. Grey vertical lines from (A) again denote relevant task events. Accuracy of arm movement decoding improved when VLSBA was recorded and combined from multiple brains. (D) Decoded X and Y trajectories from monkey M (black, solid) and monkey C (red), as well as the average of the two (dotted) during 8 second window of brain control experiment. (E,F) Neuron dropping curves (NDC) showing effect of ensemble size of each of the two monkeys on prediction accuracy during B2 passive observation. The number of neurons used from Mk1 marked by x-axis. The percent of Mk2 population used for predictions denoted by colour (see Legend). Accuracy of predictions measured as correlation coefficient r. The decoding accuracy benefited from mixing the contributions from different brains as well as the overall neuronal mass. (G-I) Same NDC analysis as (E,F) except for prediction of X, Y, and Z position during B3 passive observation. Prediction of X (G), Y (H), and Z (I) shown separately. Again (as seen in E-F), the decoding accuracy benefited from mixing the contributions from different brains as well as the overall neuronal mass.CC BY · PubMed Central
Figure 6 Decoding accuracy in individual monkeys. A UKF decoder trained post hoc predicted monkey’s own movements in every session. (A-C) Upper panels show the performance of the decoder for the x-component of the trajectories and lower panels for y-component of the trajectories. Each data point represents the average performance for the week. Correlation coefficient r was utilized to measure the accuracy of predictions. The performance of the decoder was compared across weeks to monitor the changes in the contribution of the neuronal population during the course of training in the shared control task (A,B): manual control in (A) and passive observation epoch preceding brain control in (B) and the partitioned control task (C). (D) shows the decoder performance for the x-, y- and z- components of the trajectories (top, middle and bottom panels, respectively) in the triad task. Overall, the performance of the decoder remained stable and no consistent trends were observed.
Figure 6 Decoding accuracy in individual monkeys. A UKF decoder trained post hoc predicted monkey’s own movements in every session. (A-C) Upper panels show the performance of the decoder for the x-component of the trajectories and lower panels for y-component of the trajectories. Each data point represents the average performance for the week. Correlation coefficient r was utilized to measure the accuracy of predictions. The performance of the decoder was compared across weeks to monitor the changes in the contribution of the neuronal population during the course of training in the shared control task (A,B): manual control in (A) and passive observation epoch preceding brain control in (B) and the partitioned control task (C). (D) shows the decoder performance for the x-, y- and z- components of the trajectories (top, middle and bottom panels, respectively) in the triad task. Overall, the performance of the decoder remained stable and no consistent trends were observed.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 46 references

Research works printed but not linked: 1

  1. Gibson, J. J. The theory of affordances in Perceiving, Acting and Knowing (eds. Shaw, R. & Bransford, J. ) 67–82 (Wiley, 1977).

Cited by

Metadata

Authors
Ramakrishnan, A., Ifft, P. J., Pais-Vieira, M., Byun, Y. W., Zhuang, K. Z., Lebedev, M. A., & Nicolelis, M. A. L.
Affiliations
Duke University (US)
Corresponding author
Nicolelis, Miguel A.L.
Venue
Scientific Reports
Published
2015-07-09
Volume
5
Issue
1
Pages
10767-10767
ISSN
2045-2322 (ISSN-L)
OpenAlex
W998936125
Cited by
54
References
46
Citations by year
2026: 2 · 2025: 2 · 2024: 5 · 2023: 7 · 2022: 5 · 2021: 7 · 2020: 4 · 2019: 4 · 2018: 5 · 2017: 6 · 2016: 6 · 2015: 1
FWCI
3.57
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Fundação Bial
  • National Institutes of Health
  • National Institute of Mental Health (DP1MH099903)
  • National Center for Chronic Disease Prevention and Health Promotion (DP1MH099903)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neuroscience and Neural Engineering
Keywords
brain-computer interface, non-human primates, arm movement control
MeSH terms
Animals, Haplorhini, Arm, Behavior, Animal, Brain, Electrodes, Implanted, Movement, Neurons, Brain-Computer Interfaces

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.