Multimodal BCIs: Target Detection, Multidimensional Control, and Awareness Evaluation in Patients With Disorder of Consciousness
The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Abstract
Despite rapid advances in the study of brain–computer interfaces (BCIs) in recent decades, two fundamental challenges, namely, improvement of target detection performance and multidimensional control, continue to be major barriers for further development and applications. In this paper, we review the recent progress in multimodal BCIs (also called hybrid BCIs), which may provide potential solutions for addressing these challenges. In particular, improved target detection can be achieved by developing multimodal BCIs that utilize multiple brain patterns, multimodal signals, or multisensory stimuli. Furthermore, multidimensional object control can be accomplished by generating multiple control signals from different brain patterns or signal modalities. Here, we highlight several representative multimodal BCI systems by analyzing their paradigm designs, detection/control methods, and experimental results. To demonstrate their practicality, we report several initial clinical applications of these multimodal BCI systems, including awareness evaluation/detection in patients with disorder of consciousness (DOC). As an evolving research area, the study of multimodal BCIs is increasingly requiring more synergetic efforts from multiple disciplines for the exploration of the underlying brain mechanisms, the design of new effective paradigms and means of neurofeedback, and the expansion of the clinical applications of these systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It should be noted that SVM classifier was widely used in the EEG-based brain computer interface in practice [26–29].”
From Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition (2017) · cites this paper as [27]
Citations
Cites
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927 · cited 1×
“A brain–computer interface (BCI) system directly translates the signals produced by brain activities into control signals and uses these signals to control external devices without the participation of peripheral nerves and muscles [1].”
BackgroundEvent-related EEG/MEG synchronization and desynchronization: basic principles
Pfurtscheller, G., & Lopes da Silva, F. H. (1999) · Clinical Neurophysiology · doi:10.1016/s1388-2457(99)00141-8 · Cited by 7,328 · cited 1×
“The brain patterns that are utilized for EEG-based BCIs mainly include the P300 potential [5]; steady-state evoked potentials (SSEP), particularly steady-state visual evoked potentials (SSVEPs) [6]; and event-related desynchronization/synchronization (ERD/ERS) produced by motor imagery (MI) [7].”
BackgroundFarwell, L. A., & Donchin, E. (1988) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(88)90149-6 · Cited by 3,228 · cited 1×
“The brain patterns that are utilized for EEG-based BCIs mainly include the P300 potential [5]; steady-state evoked potentials (SSEP), particularly steady-state visual evoked potentials (SSVEPs) [6]; and event-related desynchronization/synchronization (ERD/ERS) produced by motor imagery (MI) [7].”
BackgroundFeigenson, L., Dehaene, S., & Spelke, E. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.05.002 · Cited by 2,466
The minimally conscious state: Definition and diagnostic criteria
Giacino, J. T., Ashwal, S., Childs, N., Cranford, R., Jennett, B., Katz, D. I., Kelly, J. P., Rosenberg, J. H., Whyte, J., Zafonte, R. D., et al. (2002) · Neurology · doi:10.1212/wnl.58.3.349 · Cited by 2,422
A spelling device for the paralysed
Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., Perelmouter, J., Taub, E., & Flor, H. (1999) · Nature · doi:10.1038/18581 · Cited by 1,577 · cited 1×
“For instance, BCIs can offer augmented or repaired sensory-motor functions and may appeal to individuals with severe motor disabilities [2].”
BackgroundControl of a two-dimensional movement signal by a noninvasive brain-computer interface in humans
Wolpaw, J. R., & McFarland, D. J. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0403504101 · Cited by 1,512 · cited 3×
“Similar to the system presented in [18], the target in this system is also confined to one of eight candidates.”
BackgroundMethodResultWillful Modulation of Brain Activity in Disorders of Consciousness
Monti, M. M., Vanhaudenhuyse, A., Coleman, M. R., Boly, M., Pickard, J. D., Tshibanda, L., Owen, A. M., & Laureys, S. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa0905370 · Cited by 1,461
Schnakers, C., Vanhaudenhuyse, A., Giacino, J., Ventura, M., Boly, M., Majerus, S., Moonen, G., & Laureys, S. (2009) · BMC Neurology · doi:10.1186/1471-2377-9-35 · Cited by 1,348 · cited 1×
“For instance, the misdiagnosis rates for VS/UWS and MCS patients are rather high, ranging from 37% to 43% [64].”
BackgroundPERSISTENT VEGETATIVE STATE AFTER BRAIN DAMAGE
Jennett, B., & Plum, F. (1972) · The Lancet · doi:10.1016/s0140-6736(72)90242-5 · Cited by 1,227 · cited 1×
“be awake but show no awareness of themselves or their environment [62].”
Background
Cited by
Items 101–118 of 118
Zhang, Y., Chen, B., & Zhang, X. (2022) · Security and Communication Networks · doi:10.1155/2022/3143528 · Cited by 2
RetractedFilter bank temporally delayed CCA for uncalibrated SSVEP-BCI
Yin, X., Yang, C., Hui, D., Liang, J., & Lin, M. (2024) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-024-03193-x · Cited by 3
“There are also studies to explore its hybrid [10].”
Ma, S., Xiao, T., Shen, Y., Zhan, Y., Lei, B., Ng, M. K.-P., & Wang, S. (2026) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2026.3677888 · Cited by 1
“…combined with representation-level fusion and develop more interpretable fusion rules to improve system performance and decision reliability [81], [96]. where p m , t ∈ R K is the posterior probability vector of modality m , L m , t is the instantaneous cross-entropy (or MSE), and η > 0 is…”
An End-to-End Hemisphere Discrepancy Network for Subject-Independent Motor Imagery Classification
Nie, L., Cai, H. J., Wu, Y., & Zhang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92238-2_27 · Cited by 1
Brain Computer Interfaces and Emotional Involvement: Theory, Research, and Applications
Lucchiari, C. (2022) · MDPI eBooks · doi:10.3390/books978-3-0365-5377-1 · Cited by 1
Novel BCI paradigm for ALS patients based on EEG and Pupillary Accommodative Response
D'Adamo, D., Robert, E., Cristina Gena, A., & Roatta, S. (2023) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3605390.3605415 · Cited by 1
Asynchronous Controlled Single-Channel EOG-Based Puzzle Solver Robot
Panigrahi, P. K., & Bisoy, S. K. (2025) · IEEE Transactions on Games · doi:10.1109/tg.2025.3534425 · Cited by 1
“…BCI systems are widely used in many applications, such as games [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], spellers [2], [21], [22], [23], assistive robots [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], augmented reality (AR) [26], and virtual reality (VR)…”
QELDBA: Query-Efficient and Low Distortion Black-Box Attack for Brainprint Recognition
Qian, J., Yi, H., Liu, H., Jin, X., & Kong, W. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3563446 · Cited by 2
Brain Computer Integration Controlled Unmanned Vehicle for Target Reconnaissance
Duan, X., Xie, S., Meng, Y., Zhou, L., & Xu, Z. (2019) · IEEE International Conference on Unmanned Systems (ICUS) · doi:10.1109/icus48101.2019.8995912 · Cited by 1
MODIFICATIONS OF UNSUPERVISED NEURAL NETWORKS FOR SINGLE TRIAL P300 DETECTION
Vařeka, L., & Mautner, P. (2018) · Neural Network World · doi:10.14311/nnw.2018.28.001 · Cited by 2
“Alternatively, in [22], multimodal BCIs were proposed that significantly outperform classic P300 BCIs.”
BackgroundContinuous use of ERP-based BCIs with different visual angles in ALS patientsPreprint
Jin, J., Allison, B. Z., Zhang, Y., Chen, Y., Zhou, S., Dong, Y., Wang, X., & Chchocki, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09089 · Cited by 1
“In last two decades, many methods were explored to improve the P300-based BCI in terms of classification accuracy [18], information transfer rate [19], convenience, and comfort [20].”
MethodExploring the Formalization of Brain Cognition in the Age of Artificial Intelligence
Libo, Y. (2020) · Smart innovation, systems and technologies · doi:10.1007/978-981-15-3867-4_35
2016-2018 Index|Proceedings of the IEEE Vol. 104-106
(2018) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2884160 · Cited by 1
Track and Field Image Target Detection Based on Feature Learning
Li, W. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-19-3632-6_31
Neurotechnology for Disorders of Consciousness
Chaudhary, U. (2025) · Springer eBooks · doi:10.1007/978-3-031-78991-5_8
Galiotta, V. (2025) · IRIS Research product catalog (Sapienza University of Rome)
Subdomain Adaption Method for Cross-Domain Emotion Recognition and Clinical Consciousness Detection
Pan, J., He, Z., Zhu, J., Liang, R., Chen, Z., Xie, Q., Li, J., & Li, Y. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3650676
“The EEG-based consciousness detection method provides a noninvasive and objective means to evaluate the residual cognitive function of patients with DOC [28], [29], [30], [31].”
Self-citationOpportunities for Technology to Facilitate Treatment for People with Disorders of Consciousness
Kauffman, T., Moore, R., & Wiese, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791533
Metadata
- Authors
- Li, Y., Pan, J., Long, J., Yu, T., Wang, F., Yu, Z., & Wu, W.
- Affiliations
- South China Normal University (CN)
- Venue
- Proceedings of the IEEE
- Published
- 2016-02
- Volume
- 104
- Issue
- 2
- Pages
- 332-352
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W2247338144
- Cited by
- 117
- References
- 81
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 8 · 2023: 8 · 2022: 16 · 2021: 11 · 2020: 10 · 2019: 17 · 2018: 14 · 2017: 14 · 2016: 6
- FWCI
- 6.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the IEEE (publishedVersion)
- Funders
- National High-Tech R&D Program of China (2012AA011601)
- National Key Basic Research Program of China (2015CB351703)
- National Natural Science Foundation of China (91120305, 91420302, 61175114, 61503143)
- Guangdong Natural Science Foundation (2014A030312005, 2014A030310244)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Social Sciences › Arts and Humanities › Philosophy › Free Will and Agency
- Keywords
- hybrid BCI, target detection, multidimensional control, disorder of consciousness, multisensory stimuli, neurofeedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.