Multimodal BCIs: Target Detection, Multidimensional Control, and Awareness Evaluation in Patients With Disorder of Consciousness
The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Abstract
Despite rapid advances in the study of brain–computer interfaces (BCIs) in recent decades, two fundamental challenges, namely, improvement of target detection performance and multidimensional control, continue to be major barriers for further development and applications. In this paper, we review the recent progress in multimodal BCIs (also called hybrid BCIs), which may provide potential solutions for addressing these challenges. In particular, improved target detection can be achieved by developing multimodal BCIs that utilize multiple brain patterns, multimodal signals, or multisensory stimuli. Furthermore, multidimensional object control can be accomplished by generating multiple control signals from different brain patterns or signal modalities. Here, we highlight several representative multimodal BCI systems by analyzing their paradigm designs, detection/control methods, and experimental results. To demonstrate their practicality, we report several initial clinical applications of these multimodal BCI systems, including awareness evaluation/detection in patients with disorder of consciousness (DOC). As an evolving research area, the study of multimodal BCIs is increasingly requiring more synergetic efforts from multiple disciplines for the exploration of the underlying brain mechanisms, the design of new effective paradigms and means of neurofeedback, and the expansion of the clinical applications of these systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It should be noted that SVM classifier was widely used in the EEG-based brain computer interface in practice [26–29].”
From Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition (2017) · cites this paper as [27]
Citations
Cites
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927 · cited 1×
“A brain–computer interface (BCI) system directly translates the signals produced by brain activities into control signals and uses these signals to control external devices without the participation of peripheral nerves and muscles [1].”
BackgroundEvent-related EEG/MEG synchronization and desynchronization: basic principles
Pfurtscheller, G., & Lopes da Silva, F. H. (1999) · Clinical Neurophysiology · doi:10.1016/s1388-2457(99)00141-8 · Cited by 7,328 · cited 1×
“The brain patterns that are utilized for EEG-based BCIs mainly include the P300 potential [5]; steady-state evoked potentials (SSEP), particularly steady-state visual evoked potentials (SSVEPs) [6]; and event-related desynchronization/synchronization (ERD/ERS) produced by motor imagery (MI) [7].”
BackgroundFarwell, L. A., & Donchin, E. (1988) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(88)90149-6 · Cited by 3,228 · cited 1×
“The brain patterns that are utilized for EEG-based BCIs mainly include the P300 potential [5]; steady-state evoked potentials (SSEP), particularly steady-state visual evoked potentials (SSVEPs) [6]; and event-related desynchronization/synchronization (ERD/ERS) produced by motor imagery (MI) [7].”
BackgroundFeigenson, L., Dehaene, S., & Spelke, E. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.05.002 · Cited by 2,466
The minimally conscious state: Definition and diagnostic criteria
Giacino, J. T., Ashwal, S., Childs, N., Cranford, R., Jennett, B., Katz, D. I., Kelly, J. P., Rosenberg, J. H., Whyte, J., Zafonte, R. D., et al. (2002) · Neurology · doi:10.1212/wnl.58.3.349 · Cited by 2,422
A spelling device for the paralysed
Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., Perelmouter, J., Taub, E., & Flor, H. (1999) · Nature · doi:10.1038/18581 · Cited by 1,577 · cited 1×
“For instance, BCIs can offer augmented or repaired sensory-motor functions and may appeal to individuals with severe motor disabilities [2].”
BackgroundControl of a two-dimensional movement signal by a noninvasive brain-computer interface in humans
Wolpaw, J. R., & McFarland, D. J. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0403504101 · Cited by 1,512 · cited 3×
“Similar to the system presented in [18], the target in this system is also confined to one of eight candidates.”
BackgroundMethodResultWillful Modulation of Brain Activity in Disorders of Consciousness
Monti, M. M., Vanhaudenhuyse, A., Coleman, M. R., Boly, M., Pickard, J. D., Tshibanda, L., Owen, A. M., & Laureys, S. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa0905370 · Cited by 1,461
Schnakers, C., Vanhaudenhuyse, A., Giacino, J., Ventura, M., Boly, M., Majerus, S., Moonen, G., & Laureys, S. (2009) · BMC Neurology · doi:10.1186/1471-2377-9-35 · Cited by 1,348 · cited 1×
“For instance, the misdiagnosis rates for VS/UWS and MCS patients are rather high, ranging from 37% to 43% [64].”
BackgroundPERSISTENT VEGETATIVE STATE AFTER BRAIN DAMAGE
Jennett, B., & Plum, F. (1972) · The Lancet · doi:10.1016/s0140-6736(72)90242-5 · Cited by 1,227 · cited 1×
“be awake but show no awareness of themselves or their environment [62].”
BackgroundDisorders of consciousness after acquired brain injury: the state of the science
Giacino, J. T., Fins, J. J., Laureys, S., & Schiff, N. D. (2014) · Nature Reviews Neurology · doi:10.1038/nrneurol.2013.279 · Cited by 883 · cited 1×
“Patients with DOC include patients in a coma, patients in a vegetative state (VS) or with unresponsive wakefulness syndrome (UWS), patients in a minimally conscious state (MCS), and patients who have emerged from an MCS (EMCS) [60], [61].”
BackgroundImpaired recognition and experience of disgust following brain injury
Calder, A. J., Keane, J., Manes, F., Antoun, N., & Young, A. W. (2000) · Nature Neuroscience · doi:10.1038/80586 · Cited by 850 · cited 1×
“However, with a few exceptions, most BCI systems were developed for research and were tested mostly with healthy subjects; thus, they may not perform well on patients for the following reasons: a) in general, there are large differences in brain signals between healthy subjects and patients with severe brain injuries…”
BackgroundToward Brain-Computer Interfacing
(2007) · The MIT Press eBooks · doi:10.7551/mitpress/7493.001.0001 · Cited by 785
How many people are able to control a P300-based brain-computer interface (BCI)?
Guger, C., Daban, S., Sellers, E., Holzner, C., Krausz, G., Carabalona, R., Gramatica, F., & Edlinger, G. (2009) · Neuroscience Letters · doi:10.1016/j.neulet.2009.06.045 · Cited by 652 · cited 1×
“A nonnegligible proportion of healthy users cannot effectively control simple BCIs based on MI, P300, or SSVEP brain pattern [16], [17], [53].”
BackgroundPfurtscheller, G. (2010) · Frontiers in Neuroscience · doi:10.3389/fnpro.2010.00003 · Cited by 625 · cited 7×
“likely differ considerably among users [17].”
BackgroundInfluentialEnhanced performance by a hybrid NIRS–EEG brain computer interface
Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K.-R., & Blankertz, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.07.084 · Cited by 599
Iturrate, I., Antelis, J. M., Kubler, A., & Minguez, J. (2009) · IEEE Transactions on Robotics · doi:10.1109/tro.2009.2020347 · Cited by 512 · cited 1×
“In one study [44], shared control of a wheelchair was achieved in a series of stages.”
BackgroundElectroencephalographic (EEG) control of three-dimensional movement
McFarland, D. J., Sarnacki, W. A., & Wolpaw, J. R. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/3/036007 · Cited by 454 · cited 1×
“presented a BCI system for continuous 3-D cursor movement control, which was implemented through the modulation of various EEG rhythms [19].”
MethodKübler, A., & Birbaumer, N. (2008) · Clinical Neurophysiology · doi:10.1016/j.clinph.2008.06.019 · Cited by 441 · cited 10×
“The average accuracies across all three runs were 72%, 69.3%, and 75.3% for patients VS1, MCS1, and LIS1, respectively.”
BackgroundMethodInfluentialBrain-Controlled Wheelchairs: A Robotic Architecture
Carlson, T., & del R. Millan, J. (2013) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2012.2229936 · Cited by 432 · cited 2×
“For instance, an intelligent wheelchair may incorporate the advantages of both a BCI system and an autonomous navigation system [37], [43]–[46].”
BackgroundA Hybrid BCI System Combining P300 and SSVEP and Its Application to Wheelchair Control
Li, Y., Pan, J., Wang, F., & Yu, Z. (2013) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2013.2270283 · Cited by 421 · cited 10×
“In this system, the ‘‘go’’ and ‘‘stop’’ commands were always sent under static and in-motion conditions, respectively, and the direction control was achieved based on left- and right-hand MI [21].”
BackgroundMethodInfluentialSelf-citationTalsma, D., & Woldorff, M. G. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929054475172 · Cited by 419 · cited 5×
“Studies have indicated that multimodal integration associated with multimodal stimuli has the potential to enhance brain patterns [33], which may be beneficial for BCI performance.”
BackgroundMethodInfluentialA Brain Controlled Wheelchair to Navigate in Familiar Environments
Rebsamen, B., Guan, C., Zhang, H., Wang, C., Teo, C., Ang, M. H., & Burdet, E. (2010) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2010.2049862 · Cited by 405 · cited 2×
“One feasible solution to this problem is to integrate a BCI with an automated navigation system to implement shared control [43]–[45], in which the subject may simply focus on his or her final target and the execution of any actions necessary to achieve that target is performed by the automated navigation system (see…”
BackgroundKircher, T. T. J., Senior, C., Phillips, M. L., Rabe-Hesketh, S., Benson, P. J., Bullmore, E. T., Brammer, M., Simmons, A., Bartels, M., & David, A. S. (2001) · Cognition · doi:10.1016/s0010-0277(00)00104-9 · Cited by 341 · cited 2×
“This result may have occurred because emotional value or familiarity increases the likelihood that one’s attention will be attracted to a photograph of one’s own face [66].”
BackgroundHuman Cerebral Activation during Steady-State Visual-Evoked Responses
Pastor, M. A., Artieda, J., Arbizu, J., Valencia, M., & Masdeu, J. C. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-37-11621.2003 · Cited by 324 · cited 1×
“Notably, it has been demonstrated that both P300 and SSVEP responses involve sequential activations of cortical networks and reflect higher order cognitive abilities [71].”
BackgroundJackson, P. L., Lafleur, M. F., Malouin, F., Richards, C. L., & Doyon, J. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00369-0 · Cited by 315 · cited 1×
“Mental practice of movement sequences has been shown to activate similar brain areas as physical practice and also leads to improved performance when learning a sequence of foot movements [79].”
BackgroundFlashing characters with famous faces improves ERP-based brain–computer interface performance
Kaufmann, T., Schulz, S. M., Grünzinger, C., & Kübler, A. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/5/056016 · Cited by 300 · cited 1×
“Furthermore, several BCI studies [67], [68] have demonstrated that multiple ERP components, including the N170, the vertex-positive potential (VPP), and the P300, can be elicited using human faces as stimuli, and these components are useful for improving the target detection performance of BCIs.”
BackgroundAng, K. K., Guan, C., Chua, K. S. G., Ang, B. T., Kuah, C. W. K., Wang, C., Phua, K. S., Chin, Z. Y., & Zhang, H. (2011) · Clinical EEG and Neuroscience · doi:10.1177/155005941104200411 · Cited by 299 · cited 1×
“MIT-Manus planar shoulder and elbow robotic arm [77].”
An EEG-Based BCI System for 2-D Cursor Control by Combining Mu/Beta Rhythm and P300 Potential
Li, Y., Long, J., Yu, T., Yu, Z., Wang, C., Zhang, H., & Guan, C. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2055564 · Cited by 298 · cited 6×
“In fact, in addition to the P300, other ERPs such as the N200 and N400 may also be evoked by the oddball paradigm, and these potentials are also useful for BCI classification, as shown in several studies [8], [28]”
BackgroundMethodInfluentialSelf-citationLong, J., Li, Y., Wang, H., Yu, T., Pan, J., & Li, F. (2012) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2012.2197221 · Cited by 293 · cited 2×
“Furthermore, this multimodal BCI was extended to several applications, including a web browser [55], a mail client [56], a Windows-based explorer [57], and a wheelchair [21], [58].”
MethodSelf-citationToward a hybrid brain–computer interface based on imagined movement and visual attention
Allison, B. Z., Brunner, C., Kaiser, V, Müller-Putz, G. R., Neuper, C., & Pfurtscheller, G. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/2/026007 · Cited by 291 · cited 3×
“Multimodal BCIs combining MI and SSVEPs have also been reported [25], [26] and used in various applications, such as for controlling a cursor [26] or an orthosis [27].”
MethodSteady-state somatosensory evoked potentials: suitable brain signals for brain-computer interfaces?
Muller-Putz, G. R., Scherer, R., Neuper, C., & Pfurtscheller, G. (2006) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2005.863842 · Cited by 288 · cited 1×
“The brain patterns that are utilized for EEG-based BCIs mainly include the P300 potential [5]; steady-state evoked potentials (SSEP), particularly steady-state visual evoked potentials (SSVEPs) [6]; and event-related desynchronization/synchronization (ERD/ERS) produced by motor imagery (MI) [7].”
BackgroundPfurtscheller, G., Solis-Escalante, T., Ortner, R., Linortner, P., & Muller-Putz, G. R. (2010) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2010.2040837 · Cited by 276 · cited 12×
“Multimodal BCIs combining MI and SSVEPs have also been reported [25], [26] and used in various applications, such as for controlling a cursor [26] or an orthosis [27].”
MethodInfluentialDoud, A. J., Lucas, J. P., Pisansky, M. T., & He, B. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026322 · Cited by 244 · cited 2×
“developed several BCI systems to control a virtual helicopter in 3-D space with commands generated through MI of left hand, right hand, tongue, foot and both hands, and the resting state [20].”
BackgroundZhang, R., Li, Y., Yan, Y., Zhang, H., Wu, S., Yu, T., & Gu, Z. (2016) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2015.2439298 · Cited by 239 · cited 1×
“For instance, an intelligent wheelchair may incorporate the advantages of both a BCI system and an autonomous navigation system [37], [43]–[46].”
BackgroundSelf-citationA novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm
Yin, E., Zhou, Z., Jiang, J., Chen, F., Liu, Y., & Hu, D. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/2/026012 · Cited by 225 · cited 4×
“Several groups have proposed multimodal BCIs combining the P300 and SSVEPs to improve the performance of an on/off brain switch [21] or a BCI speller [22]–[24].”
BackgroundMethodInfluentialEnhancement of Left-Right Sensorimotor EEG Differences During Feedback-Regulated Motor Imagery
Neuper, C., Schlögl, A., & Pfurtscheller, G. (1999) · Journal of Clinical Neurophysiology · doi:10.1097/00004691-199907000-00010 · Cited by 225
Ma, J., Zhang, Y., Cichocki, A., & Matsuno, F. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2369483 · Cited by 209 · cited 1×
“modal BCIs combining eye movements and EEG have also been reported and used for robot control [38], [39].”
MethodAn Asynchronous P300 BCI With SSVEP-Based Control State Detection
Panicker, R. C., Puthusserypady, S., & Sun, Y. (2011) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2011.2116018 · Cited by 203
Lulé, D., Noirhomme, Q., Kleih, S. C., Chatelle, C., Halder, S., Demertzi, A., Bruno, M.-A., Gosseries, O., Vanhaudenhuyse, A., Schnakers, C., et al. (2013) · Clinical Neurophysiology · doi:10.1016/j.clinph.2012.04.030 · Cited by 189 · cited 3×
“tested a four-choice auditory oddball BCI in a study involving 16 healthy subjects and 18 patients with DOC [13 MCS, three VS, and two locked-in syndrome (LIS)], and 13 healthy subjects, one MCS patient, and one LIS patient were shown to be able to communicate using the BCI [14].”
BackgroundMethodRohm, M., Schneiders, M., Müller, C., Kreilinger, A., Kaiser, V., Müller-Putz, G. R., & Rupp, R. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2013.07.004 · Cited by 187 · cited 2×
“thesis control, such as that proposed in [59], which will be reviewed in Section V.”
BackgroundNeural mechanisms of brain–computer interface control
Halder, S., Agorastos, D., Veit, R., Hammer, E. M., Lee, S., Varkuti, B., Bogdan, M., Rosenstiel, W., Birbaumer, N., & Kübler, A. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.01.021 · Cited by 166 · cited 1×
“MI is generally assumed to activate the same representations as the execution of the corresponding motor tasks [78].”
BackgroundECoG factors underlying multimodal control of a brain-computer interface
Wilson, J. A., Felton, E. A., Garell, P. C., Schalk, G., & Williams, J. C. (2006) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2006.875570 · Cited by 163 · cited 1×
“electrodes are implanted in the cerebral cortex (microelectrode arrays or neurotropic electrodes) [3] or on the cortical surface [electrocorticography (ECoG)] [4].”
BackgroundP300 and probability: comparison of oddball and single-stimulus paradigms
Polich, J., & Margala, C. (1997) · International Journal of Psychophysiology · doi:10.1016/s0167-8760(96)00742-8 · Cited by 162 · cited 1×
“Specifically, the oddball effect is generated by randomly setting the time interval between two adjacent stimuli, as has been verified in previous studies [52].”
ResultA new hybrid BCI paradigm based on P300 and SSVEP
Wang, M., Daly, I. P., Allison, B. Z., Jin, J., Zhang, Y., Chen, L., & Wang, X. (2014) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2014.06.003 · Cited by 160
Muller-Putz, G., Leeb, R., Tangermann, M., Hohne, J., Kubler, A., Cincotti, F., Mattia, D., Rupp, R., Muller, K.-R., & Millan, J. D. R. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2411333 · Cited by 158 · cited 2×
“For instance, an intelligent wheelchair may incorporate the advantages of both a BCI system and an autonomous navigation system [37], [43]–[46].”
BackgroundA novel BCI based on ERP components sensitive to configural processing of human faces
Zhang, Y., Zhao, Q., Jing, J., Wang, X., & Cichocki, A. (2012) · Journal of Neural Engineering · doi:10.1088/1741-2560/9/2/026018 · Cited by 156 · cited 1×
“Furthermore, several BCI studies [67], [68] have demonstrated that multiple ERP components, including the N170, the vertex-positive potential (VPP), and the P300, can be elicited using human faces as stimuli, and these components are useful for improving the target detection performance of BCIs.”
BackgroundA hybrid ERD/SSVEP BCI for continuous simultaneous two dimensional cursor control
Allison, B. Z., Brunner, C., Altstätter, C., Wagner, I. C., Grissmann, S., & Neuper, C. (2012) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2012.06.022 · Cited by 155
Hybrid fNIRS-EEG based classification of auditory and visual perception processes
Putze, F., Heßlinger, S., Tse, C., Huang, Y., Herff, C., Guan, C., & Schultz, T. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00373 · Cited by 154 · cited 1×
“The use of a combination of fMRI and EEG in BCIs may allow the spatial and temporal information of brain activity to be fully exploited [35], [40], which would enable the study of the basic psychophysiological mechanisms of BCIs, such as the correlations between local BOLD-responses and changes in the slow cortical po…”
BackgroundSpatial-Temporal Discriminant Analysis for ERP-Based Brain-Computer Interface
Zhang, Y., Zhou, G., Zhao, Q., Jin, J., Wang, X., & Cichocki, A. S. (2013) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2013.2243471 · Cited by 150 · cited 1×
“Several modern signal processing techniques have also been developed for BCIs, such as multiway learning and tensor factorization [10], [11].”
MethodTools for brain-computer interaction: a general concept for a hybrid BCI
Müller-Putz, G. R. (2011) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2011.00030 · Cited by 142 · cited 1×
“BCI-Based Shared Control Systems In multimodal BCIs of this type, BCIs are combined with other intelligent systems to achieve shared control and to make the BCI systems more reliable, flexible, user friendly, and efficient by allowing the subjects to focus their attention on the ultimate target while ignoring lowlevel…”
MethodChoi, B., & Jo, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0074583 · Cited by 141 · cited 2×
“Two or more of EEG, MEG, fMRI, fNIRS, EMG, and electrooculography (EOG) signals can be combined to serve as inputs to a multimodal BCI system [17], [34]–[39].”
BackgroundMethodA hybrid BCI speller paradigm combining P300 potential and the SSVEP blocking feature
Xu, M., Qi, H., Wan, B., Yin, T., Liu, Z., & Ming, D. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/2/026001 · Cited by 133 · cited 1×
“[23] M.”
Target Selection With Hybrid Feature for BCI-Based 2-D Cursor Control
Long, J., Li, Y., Yu, T., & Gu, Z. (2012) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2011.2167718 · Cited by 128 · cited 5×
“each trial, we extract the features of the P300 and MI signals separately and then concatenate them to construct a hybrid feature vector, which is fed into the SVM for classification [54].”
MethodResultInfluentialSelf-citationMalouin, F., Richards, C. L., & Durand, A. (2010) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2010.03.007 · Cited by 126 · cited 1×
“In addition, there is evidence to suggest that mental rehearsal may have rehabilitative effects in cases of stroke and spinal cord injury [80].”
BackgroundEnhanced Motor Imagery Training Using a Hybrid BCI With Feedback
Yu, T., Xiao, J., Wang, F., Zhang, R., Gu, Z., Cichocki, A., & Li, Y. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2402283 · Cited by 124 · cited 2×
“In [29], a multimodal BCI combining MI and SSVEPs was proposed to provide positive feedback and facilitate MI training.”
MethodSelf-citationA Hybrid Brain–Computer Interface Based on the Fusion of P300 and SSVEP Scores
Yin, E., Zeyl, T., Saab, R., Chau, T., Hu, D., & Zhou, Z. (2015) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2015.2403270 · Cited by 120 · cited 1×
“In a subsequent study [49], a maximum-probability estimation (MPE) fusion approach was proposed to combine P300 and SSVEPs at the decision level; in this approach, the scores obtained in P300 and SSVEP detections were converted into probabilities and the button with the maximum probability was selected as the target.”
MethodJin, J., Allison, B. Z., Zhang, Y., Wang, X., & Cichocki, A. S. (2014) · International Journal of Neural Systems · doi:10.1142/s0129065714500270 · Cited by 115 · cited 1×
“Furthermore, it has been shown that different familiar face stimuli enhance the N200 and N400 components and thus decrease actual error outputs [9].”
BackgroundCruse, D., Chennu, S., Fernández-Espejo, D., Payne, W. L., Young, G. B., & Owen, A. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049933 · Cited by 114 · cited 1×
“, [74]) studies have also shown that some patients who meet the behavioral criteria for VS may have residual cognitive functions and even conscious awareness.”
BackgroundPan, J., Xie, Q., He, Y., Wang, F., Di, H., Laureys, S., Yu, R., & Li, Y. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/5/056007 · Cited by 110 · cited 7×
“right sides of the GUI, respectively [15].”
BackgroundMethodInfluentialSelf-citationLehembre, R., Gosseries, O., Lugo, Z., Jedidi, Z., Chatelle, C., Sadzot, B., Laureys, S., & Noirhomme, Q. (2012) · PubMed · doi:10.4449/aib.v150i2.1374 · Cited by 103 · cited 1×
“However, with a few exceptions, most BCI systems were developed for research and were tested mostly with healthy subjects; thus, they may not perform well on patients for the following reasons: a) in general, there are large differences in brain signals between healthy subjects and patients with severe brain injuries…”
BackgroundBelitski, A., Farquhar, J., & Desain, P. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/2/025022 · Cited by 96 · cited 2×
“The data analysis results that were presented for the offline audiovisual P300-based speller proposed in [30] showed that the strength of the P300 response was higher in the audiovisual condition than in the visual-only or auditory-only condition.”
ResultEEG and MEG brain-computer interface for tetraplegic patients
Kauhanen, L., Nykopp, T., Lehtonen, J., Jylanki, P., Heikkonen, J., Rantanen, P., Alaranta, H., & Sams, M. (2006) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2006.875546 · Cited by 92 · cited 3×
“Two or more of EEG, MEG, fMRI, fNIRS, EMG, and electrooculography (EOG) signals can be combined to serve as inputs to a multimodal BCI system [17], [34]–[39].”
BackgroundMethodAn optimized ERP brain–computer interface based on facial expression changes
Jin, J., Daly, I. P., Zhang, Y., Wang, X., & Cichocki, A. S. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/036004 · Cited by 91 · cited 2×
“proposed a method of feedback based on changes in facial expression and thus improved the detection performance of an event-related potential (ERP)-based BCI [8].”
MethodPan, J., Li, Y., Zhang, R., Gu, Z., & Li, F. (2013) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2013.2253801 · Cited by 90 · cited 1×
“We designed this multimodal BCI primarily for a brain switch application, in which one group of buttons (top buttons) is designated as the target for ‘‘on/off’’ commands and the other buttons are designated as pseudokeys that do not activate any commands (as explained in detail in [48]).”
MethodSelf-citationZhao, Q., Zhou, G., Adalı, T., Zhang, L., & Cichocki, A. S. (2013) · IEEE Signal Processing Magazine · doi:10.1109/msp.2013.2255334 · Cited by 84 · cited 1×
“Several modern signal processing techniques have also been developed for BCIs, such as multiway learning and tensor factorization [10], [11].”
MethodChronic Consciousness Disorders
Bernat, J. L. (2009) · Annual Review of Medicine · doi:10.1146/annurev.med.60.060107.091250 · Cited by 82 · cited 6×
“Eleven patients (six VS, three MCS, and two EMCS, diagnosed using the JFK CRS-R; five males; native Chinese; mean age AE SD, 38.6 AE 12 years) participated in runs 1 and 2, and only those patients who achieved the accuracies significantly higher than the chance level (50”
BackgroundInfluentialConverting thoughts into action
Scott, S. H. (2006) · Nature · doi:10.1038/442141a · Cited by 74 · cited 2×
“However, for this class of BCIs, there are prominent risks related to surgery as well as concerns regarding potential infection and the long-term viability of chronic invasive neural probes [3].”
BackgroundSurfing the internet with a BCI mouse
Yu, T., Li, Y., Long, J., & Gu, Z. (2012) · Journal of Neural Engineering · doi:10.1088/1741-2560/9/3/036012 · Cited by 74 · cited 4×
“Specifically, in [55], we proposed a BCI web browser that allows the user to surf the Internet without any limb movements.”
BackgroundMethodInfluentialSelf-citationAn EEG-Driven Brain-Computer Interface Combined With Functional Magnetic Resonance Imaging (fMRI)
Hinterberger, T., Weiskopf, N., Veit, R., Wilhelm, B., Betta, E., & Birbaumer, N. (2004) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2004.827069 · Cited by 71 · cited 1×
“The use of a combination of fMRI and EEG in BCIs may allow the spatial and temporal information of brain activity to be fully exploited [35], [40], which would enable the study of the basic psychophysiological mechanisms of BCIs, such as the correlations between local BOLD-responses and changes in the slow cortical po…”
BackgroundTactile and bone-conduction auditory brain computer interface for vision and hearing impaired users
Rutkowski, T. M., & Mori, H. (2014) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2014.04.010 · Cited by 67
A four-choice hybrid P300/SSVEP BCI for improved accuracy
Allison, B. Z., Jin, J., Zhang, Y., & Wang, X. (2014) · Brain-Computer Interfaces · doi:10.1080/2326263x.2013.869003 · Cited by 66 · cited 10×
“it to be more annoying, difficult, or fatiguing to use than the P300- or SSVEP-based BCI [50].”
MethodInfluentialA Novel Audiovisual Brain-Computer Interface and Its Application in Awareness Detection
Wang, F., He, Y., Pan, J., Xie, Q., Yu, R., Zhang, R., & Li, Y. (2015) · Scientific Reports · doi:10.1038/srep09962 · Cited by 54 · cited 6×
“Here, we will present an audiovisual BCI and investigate the effect of audiovisual stimuli on the BCI performance [51].”
BackgroundMethodInfluentialSelf-citationDoes bimodal stimulus presentation increase ERP components usable in BCIs?
Thurlings, M. E., Brouwer, A.-M., van Erp, J. B. F., Blankertz, B., & Werkhoven, P. J. (2012) · Journal of Neural Engineering · doi:10.1088/1741-2560/9/4/045005 · Cited by 51 · cited 1×
“investigated the effects of bimodal visual-tactile stimulus presentation on ERP components and reported the enhancement of early components such as N1, which may improve BCI performance [31].”
BackgroundA brain computer interface-based explorer
Bai, L., Yu, T., & Li, Y. (2014) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2014.06.015 · Cited by 36 · cited 2×
“The principal methods of implementation for the BCI mail client and BCI file explorer, which are presented in [56] and [57], are similar to those of the BCI web browser.”
MethodSelf-citationGerschlager, W., Bloem, B. R., Alesch, F. X. E. O., Lang, W., Deecke, L., & Cunnington, R. (2001) · Journal of Neurology · doi:10.1007/s004150170202 · Cited by 30 · cited 1×
“For instance, the latent period of the P300 was generally longer for the patients than for the healthy controls, a finding that has also been reported in other studies [69], [70].”
ResultA Hybrid Brain-Computer Interface-Based Mail Client
Yu, T., Li, Y., Long, J., & Li, F. (2013) · Computational and Mathematical Methods in Medicine · doi:10.1155/2013/750934 · Cited by 26 · cited 3×
“proposed an alternative BCI-mouse-based mail client that can implement electronic mail communication [56].”
BackgroundMethodSelf-citationBCI and FES Training of a Spinal Cord Injured End-User to Control a Neuroprosthesis
Kreilinger, A., Kaiser, V., Rohm, M., Rupp, R., & Muller-Putz, G. R. (2013) · Biomedizinische Technik/Biomedical Engineering · doi:10.1515/bmt-2013-4443 · Cited by 25 · cited 1×
“For instance, EEG and EMG can be combined in multimodal BCIs for the multidimensional control of a neuroprosthesis [37], [59], [75] or for the rehabilitation of severely motor-restricted patients, such as those with stroke or spinal cord injuries [76].”
BackgroundLonger Auditory and Visual P300 Latencies in Patients with Narcolepsy
Sangal, R. B., Sangal, J. M., & Belisle, C. (1999) · Clinical Electroencephalography · doi:10.1177/155005949903000111 · Cited by 22 · cited 1×
“For instance, the latent period of the P300 was generally longer for the patients than for the healthy controls, a finding that has also been reported in other studies [69], [70].”
ResultHybrid-P300 BCI: Usability Testing by Severely Motor-restricted End-Users
Reichert, J., Leotta, F., Aricò, P., Febo Cincotti, A., Donatella Mattia, A., Kübler, A., & Holz, E. M. (2013) · Cited by 9
Hybrid Brain-Computer Interaction for Functional Motor Recovery after Stroke
Mattia, D., Pichiorri, F., Aricò, P., Aloise, F., & Cincotti, F. (2013) · Biosystems & biorobotics · doi:10.1007/978-3-642-34546-3_213 · Cited by 3 · cited 2×
“Furthermore, neurofeedback-based training can play an important role in rehabilitation for many patients, such as those with stroke [76].”
Background
Cited by
A Benchmark Dataset for SSVEP-Based Brain–Computer Interfaces
Wang, Y., Chen, X., Gao, X., & Gao, S. (2017) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2016.2627556 · Cited by 501
“, P300) have become more and more popular in recent studies [10]–[13].”
BackgroundEEG Based Emotion Recognition by Combining Functional Connectivity Network and Local Activations
Li, P., Liu, H., Si, Y., Li, C., Li, F., Zhu, X., Huang, X., Zeng, Y., Yao, D., Zhang, Y., et al. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2019.2897651 · Cited by 396
“tion methods may lead to some differences in optimal feature subsets because of their different design purposes [34], [38], [39], [46], [64], which may finally account for the different”
MethodMulti-kernel extreme learning machine for EEG classification in brain-computer interfaces
Zhang, Y., Wang, Y., Zhou, G., Jin, J., Wang, B., Wang, X., & Cichocki, A. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.12.015 · Cited by 255
Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., & Pan, J. (2020) · Brain Sciences · doi:10.3390/brainsci10100687 · Cited by 151
“[9] proposed that in addition to combining different input signals, emotion recognition should include a variety of heterogeneous sensory stimuli (such as audio-visual stimulation) to induce emotions.”
BackgroundSelf-citationLi, F., Wang, J., Liao, Y., Yi, C., Jiang, Y., Si, Y., Peng, W., Yao, D., Zhang, Y., Dong, W., et al. (2019) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2019.2900725 · Cited by 124
Metadata
- Authors
- Li, Y., Pan, J., Long, J., Yu, T., Wang, F., Yu, Z., & Wu, W.
- Affiliations
- South China Normal University (CN)
- Venue
- Proceedings of the IEEE
- Published
- 2016-02
- Volume
- 104
- Issue
- 2
- Pages
- 332-352
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W2247338144
- Cited by
- 117
- References
- 81
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 8 · 2023: 8 · 2022: 16 · 2021: 11 · 2020: 10 · 2019: 17 · 2018: 14 · 2017: 14 · 2016: 6
- FWCI
- 6.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the IEEE (publishedVersion)
- Funders
- National High-Tech R&D Program of China (2012AA011601)
- National Key Basic Research Program of China (2015CB351703)
- National Natural Science Foundation of China (91120305, 91420302, 61175114, 61503143)
- Guangdong Natural Science Foundation (2014A030312005, 2014A030310244)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Social Sciences › Arts and Humanities › Philosophy › Free Will and Agency
- Keywords
- hybrid BCI, target detection, multidimensional control, disorder of consciousness, multisensory stimuli, neurofeedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.