Blending of brain-machine interface and vision-guided autonomous robotics improves neuroprosthetic arm performance during grasping
Open accessAbstract
BACKGROUND: Recent studies have shown that brain-machine interfaces (BMIs) offer great potential for restoring upper limb function. However, grasping objects is a complicated task and the signals extracted from the brain may not always be capable of driving these movements reliably. Vision-guided robotic assistance is one possible way to improve BMI performance. We describe a method of shared control where the user controls a prosthetic arm using a BMI and receives assistance with positioning the hand when it approaches an object. METHODS: Two human subjects with tetraplegia used a robotic arm to complete object transport tasks with and without shared control. The shared control system was designed to provide a balance between BMI-derived intention and computer assistance. An autonomous robotic grasping system identified and tracked objects and defined stable grasp positions for these objects. The system identified when the user intended to interact with an object based on the BMI-controlled movements of the robotic arm. Using shared control, BMI controlled movements and autonomous grasping commands were blended to ensure secure grasps. RESULTS: Both subjects were more successful on object transfer tasks when using shared control compared to BMI control alone. Movements made using shared control were more accurate, more efficient, and less difficult. One participant attempted a task with multiple objects and successfully lifted one of two closely spaced objects in 92 % of trials, demonstrating the potential for users to accurately execute their intention while using shared control. CONCLUSIONS: Integration of BMI control with vision-guided robotic assistance led to improved performance on object transfer tasks. Providing assistance while maintaining generalizability will make BMI systems more attractive to potential users. TRIAL REGISTRATION: NCT01364480 and NCT01894802 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ICMS can deliver localized tactile percepts with natural features and can be used in human-brain-computer interface applications for better control using area S1.
“ICMS can also be employed to deliver localized tactile percepts with natural features as well as in human-brain-computer interface applications for better control using area S1 (Collinger et al., 2013; Wodlinger et al., 2015; Downey et al., 2016).”
From Selectivity and Longevity of Peripheral-Nerve and Machine Interfaces: A Review. (2017) · cites this paper as Downey et al., 2016
Methods > Subjects (page unavailable) reports S1 recording arrays in a human BMI participant, and the Background reports improved robotic-arm performance with shared control. The paper does not report ICMS, localized or natural-feeling tactile percepts, or improved control through S1 stimulation; its Background instead says the arm-control BMIs discussed lacked somatosensory feedback. (LLM, conf=0.86)
Experimental — accuracy not yet measuredShared control movements using a brain-machine interface and vision guidance were more accurate, efficient, and less complicated than transfer tasks using BMI alone.
“Experiments proved that shared control movements were more accurate, efficient, and less complicated than transfer tasks using BMI alone.”
From Advanced Applications of Industrial Robotics: New Trends and Possibilities (2021) · cites this paper as [66]
Stimulation of the peripheral nerve or a somatosensory area of the brain (S1) with trains of electrical pulses evokes percepts that transmit information from the grasping object.
“The necessary methodology entails stimulation of the peripheral nerve or a somatosensory area of the brain (S1) with trains of electrical pulses to evoke percepts that transmit information from the grasping object (Downey et al., 2016).”
From Selectivity and Longevity of Peripheral-Nerve and Machine Interfaces: A Review. (2017) · cites this paper as Downey et al., 2016
The Background (page unavailable) says arm-control BMIs do not provide somatosensory feedback, and Methods > Neural recording describes recording neural signals rather than delivering stimulation. Searches for electrical pulses, stimulation, and peripheral-nerve feedback found no passage in our copy addressing whether such stimulation evokes object-related percepts. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reach and grasp by people with tetraplegia using a neurally controlled robotic arm
Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., Haddadin, S., Liu, J., Cash, S. S., van der Smagt, P., et al. (2012) · Nature · doi:10.1038/nature11076 · Cited by 2,719 · see citing works
Coding and use of tactile signals from the fingertips in object manipulation tasks
Johansson, R. S., & Flanagan, J. R. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2621 · Cited by 2,063 · see citing works
High-performance neuroprosthetic control by an individual with tetraplegia
Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., McMorland, A. J., Velliste, M., Boninger, M. L., & Schwartz, A. B. (2013) · The Lancet · doi:10.1016/s0140-6736(12)61816-9 · Cited by 1,848 · see citing works
Self-citationCortical control of a prosthetic arm for self-feeding
Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008) · Nature · doi:10.1038/nature06996 · Cited by 1,591 · see citing works
Self-citationLyle, R. C. (1981) · International Journal of Rehabilitation Research · doi:10.1097/00004356-198112000-00001 · Cited by 1,335 · see citing works
Robotics: Science and Systems (RSS)
Trinkle, J., & Matsuoka, Y. (2010) · AI Magazine · doi:10.1609/aimag.v31i1.2237 · Cited by 1,082 · see citing works
Wodlinger, B., Downey, J. E., Tyler-Kabara, E. C., Schwartz, A. B., Boninger, M. L., & Collinger, J. L. (2015) · Journal of Neural Engineering · doi:10.1088/1741-2560/12/1/016011 · Cited by 536 · see citing works
Self-citationControl of a humanoid robot by a noninvasive brain–computer interface in humans
Bell, C. J., Shenoy, P., Chalodhorn, R., & Rao, R. P. N. (2008) · Journal of Neural Engineering · doi:10.1088/1741-2560/5/2/012 · Cited by 387 · see citing works
Restoring sensorimotor function through intracortical interfaces: progress and looming challenges
Bensmaia, S. J., & Miller, L. E. (2014) · Nature Reviews Neuroscience · doi:10.1038/nrn3724 · Cited by 368 · see citing works
Robotics: Science and Systems V
(2009) · doi:10.15607/rss.2009.v · Cited by 368 · see citing works
Chestek, C. A., Gilja, V., Nuyujukian, P., Foster, J. D., Fan, J. M., Kaufman, M. T., Churchland, M. M., Rivera-Alvidrez, Z., Cunningham, J. P., Ryu, S. I., et al. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/4/045005 · Cited by 350 · see citing works
Perge, J. A., Homer, M. L., Malik, W. Q., Cash, S., Eskandar, E., Friehs, G., Donoghue, J. P., & Hochberg, L. R. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/3/036004 · Cited by 243 · see citing works
Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury
Collinger, J. L., Boninger, M. L., Bruns, T. M., Curley, K. C., Wang, W., & Weber, D. J. (2013) · The Journal of Rehabilitation Research and Development · doi:10.1682/jrrd.2011.11.0213 · Cited by 241 · see citing works
Self-citationMcMullen, D. P., Hotson, G., Katyal, K. D., Wester, B. A., Fifer, M. S., McGee, T. G., Harris, A., Johannes, M. S., Vogelstein, R. J., Ravitz, A., et al. (2014) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2013.2294685 · Cited by 189 · see citing works
Kim, D., Hazlett-Knudsen, R., Culver-Godfrey, H., Rucks, G., Cunningham, T., Portee, D., Bricout, J. C., Wang, Z., & Behal, A. (2011) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2011.2159589 · Cited by 164 · see citing works
An integrated system for autonomous robotics manipulation
Bagnell, J. A., Cavalcanti, F., Cui, L., et al. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385888 · Cited by 124 · see citing works
Self-citationAutonomy infused teleoperation with application to brain computer interface controlled manipulation
Muelling, K., Venkatraman, A., Valois, J., Downey, J. E., Weiss, J. M., Javdani, S., Hebert, M., Schwartz, A. B., Collinger, J. L., & Bagnell, J. A. (2017) · Autonomous Robots · doi:10.1007/s10514-017-9622-4 · Cited by 122 · see citing works
Self-citationNonlinear Body Pose Estimation from Depth Images
Grest, D., Woetzel, J., & Koch, R. (2005) · Lecture notes in computer science · doi:10.1007/11550518_36 · Cited by 120 · see citing works
Huggins, J. E., Moinuddin, A. A., Chiodo, A. E., & Wren, P. A. (2015) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2014.05.028 · Cited by 97 · see citing works
Shape-based object recognition in videos using 3D synthetic object models
Toshev, A., Makadia, A., & Daniilidis, K. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206803 · Cited by 60 · see citing works
Vogel, J., Haddadin, S., Jarosiewicz, B., Simeral, J. D., Bacher, D., Hochberg, L. R., Donoghue, J. P., & van der Smagt, P. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914561535 · Cited by 59 · see citing works
A collaborative BCI approach to autonomous control of a prosthetic limb system
Katyal, K. D., Johannes, M. S., Kellis, S., Aflalo, T., Klaes, C., McGee, T. G., Para, M. P., Shi, Y., Lee, B., Pejsa, K., et al. (2014) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2014.6974124 · Cited by 48 · see citing works
Autonomy Infused Teleoperation with Application to BCI Manipulation
Mülling, K., Venkatraman, A., Valois, J., Downey, J. E., Weiss, J. M., Javdani, S., Hebert, M., Schwartz, A. B., Collinger, J. L., & Bagnell, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.039 · Cited by 47 · see citing works
Self-citationDevelopments in brain–machine interfaces from the perspective of robotics
Kim, H. K., Park, S., & Srinivasan, M. A. (2009) · Human Movement Science · doi:10.1016/j.humov.2008.12.001 · Cited by 19 · see citing works
Autonomy Infused Teleoperation with Application to BCI ManipulationPreprint
Muelling, K., Venkatraman, A., Valois, J., Downey, J. E., Weiss, J. M., Javdani, S., Hebert, M., Schwartz, A. B., Collinger, J. L., & Bagnell, J. A. (2015) · arXiv (Cornell University) · doi:10.1184/r1/6552149 · Cited by 19 · see citing works
Self-citationMotor performance benefits of matched limb imitation in prosthesis users
Cusack, W. F., Patterson, R., Thach, S., et al. (2014) · Experimental Brain Research · doi:10.1007/s00221-014-3904-2 · Cited by 17 · see citing works
Neuroprosthetic control and tetraplegia – Authors'reply
Boninger, M. L., Mitchell, G. W., Tyler‐Kabara, E. C., Collinger, J. L., & Schwartz, A. B. (2013) · The Lancet · doi:10.1016/s0140-6736(13)61154-x · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Advanced Applications of Industrial Robotics: New Trends and Possibilities
Dzedzickis, A., Subačiūtė-Žemaitienė, J., Šutinys, E., Samukaitė-Bubnienė, U., & Bučinskas, V. (2021) · Applied Sciences · doi:10.3390/app12010135 · Cited by 228 · see citing works
Intracortical recording stability in human brain–computer interface users
Downey, J. E., Schwed, N., Chase, S. M., Schwartz, A. B., & Collinger, J. L. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aab7a0 · Cited by 149 · see citing works
Self-citationPlug-and-play control of a brain–computer interface through neural map stabilization
Silversmith, D. B., Abiri, R., Hardy, N. F., Natraj, N., Tu-Chan, A., Chang, E. F., & Ganguly, K. (2021) · Nature Biotechnology · doi:10.1038/s41587-020-0662-5 · Cited by 126 · see citing works
Selectivity and Longevity of Peripheral-Nerve and Machine Interfaces: A Review.
Ghafoor, U., Kim, S., & Hong, K.-S. (2017) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2017.00059 · Cited by 88 · see citing works
“The necessary methodology entails stimulation of the peripheral nerve or a somatosensory area of the brain (S1) with trains of electrical pulses to evoke percepts that transmit information from the grasping object (Downey et al., 2016).”
Motor cortical activity changes during neuroprosthetic-controlled object interaction
Downey, J. E., Brane, L., Gaunt, R. A., Tyler-Kabara, E. C., Boninger, M. L., & Collinger, J. L. (2017) · Scientific Reports · doi:10.1038/s41598-017-17222-3 · Cited by 63 · see citing works
Self-citationBullard, A. J., Hutchison, B. C., Lee, J., Chestek, C. A., & Patil, P. G. (2020) · Neuromodulation: Technology at the Neural Interface · doi:10.1111/ner.13069 · Cited by 65 · see citing works
- Authors
- Downey, J. E., Weiss, J. M., Muelling, K., Venkatraman, A., Valois, J.-S., Hebert, M., Bagnell, J. A., Schwartz, A. B., & Collinger, J. L.
- Affiliations
- University of Pittsburgh (US)
- Carnegie Mellon University (US)
- Corresponding author
- Collinger, Jennifer L.
- Venue
- Journal of NeuroEngineering and Rehabilitation
- Published
- 2016-03-18
- Volume
- 13
- Issue
- 1
- Pages
- 28-28
- ISSN
- 1743-0003 (ISSN-L)
- PMC
- PMC4797113
- OpenAlex
- W2300987908
- Cited by
- 111
- References
- 28
- Citations by year
- 2026: 6 · 2025: 12 · 2024: 11 · 2023: 12 · 2022: 3 · 2021: 16 · 2020: 12 · 2019: 15 · 2018: 12 · 2017: 11 · 2016: 1
- FWCI
- 8.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Journal of NeuroEngineering and Rehabilitation (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- D-Scholarship@Pitt (University of Pittsburgh) (acceptedVersion)
- KiltHub Repository (submittedVersion, other-oa)
- Figshare (submittedVersion, cc-by)
- KiltHub Repository (cc-by)
- Funders
- Defense Advanced Research Projects Agency (ID0EY1AG644)
- National Science Foundation (ID0EG3AG645)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
- Health Sciences › Medicine › Rehabilitation › Stroke Rehabilitation and Recovery
- Keywords
- brain-computer interface, shared control, grasping, tetraplegia, object tracking, assistive robotics
- MeSH terms
- Neurological Rehabilitation, Adult, Brain, Female, Hand, Humans, Male, Middle Aged, Movement, Quadriplegia, Robotics, Hand Strength, Upper Extremity, Brain-Computer Interfaces
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
