Benchmarking Brain-Computer Interfaces Outside the Laboratory: The Cybathlon 2016
Open accessAbstract
This paper presents a new approach to benchmarking brain-computer interfaces (BCIs) outside the lab. A computer game was created that mimics a real-world application of assistive BCIs, with the main outcome metric being the time needed to complete the game. This approach was used at the Cybathlon 2016, a competition for people with disabilities who use assistive technology to achieve tasks. The paper summarizes the technical challenges of BCIs, describes the design of the benchmarking game, then describes the rules for acceptable hardware, software and inclusion of human pilots in the BCI competition at the Cybathlon. The 11 participating teams, their approaches, and their results at the Cybathlon are presented. Though the benchmarking procedure has some limitations (for instance, we were unable to identify any factors that clearly contribute to BCI performance), it can be successfully used to analyze BCI performance in realistic, less structured conditions. In the future, the parameters of the benchmarking game could be modified to better mimic different applications (e.g., the need to use some commands more frequently than others). Furthermore, the Cybathlon has the potential to showcase such devices to the general public.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A recent study analyzing the Cybathlon BCI race states that it was not possible to identify any factor (hardware, signal processing, machine learning, and pilot's conditions) explaining the performance.
“In fact, a recent study analyzing the results of the Cybathlon BCI race states that it was not possible to identify any factor (hardware, signal processing, machine learning, and pilot’s conditions) explaining the performance [8].”
From The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users (2018) · cites this paper as [8]
The impact of training with the BCI application might be related to a need for getting accustomed to the actual application demands but probably also to increased user motivation provided by the gaming application and the goal of participating in an international competition.
“This might be related to a need for getting accustomed to the actual application demands [47] but probably also to increased user motivation [68] provided both by the gaming application and the goal of participating in an international competition [8].”
From The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users (2018) · cites this paper as [8]
Differences in machine-learning methods of all participating teams were too subtle to explain the competition outcomes according to the organizers.
“Of note, the differences in machine-learning methods of all participating teams were too subtle to explain the competition outcomes according to the organizers [8].”
From The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users (2018) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Optimal spatial filtering of single trial EEG during imagined hand movement
Ramoser, H., Muller-Gerking, J., & Pfurtscheller, G. (2000) · IEEE Transactions on Rehabilitation Engineering · doi:10.1109/86.895946 · Cited by 2,457 · see citing works
Brain Computer Interfaces, a Review
Nicolas-Alonso, L. F., & Gomez-Gil, J. (2012) · Sensors · doi:10.3390/s120201211 · Cited by 2,143 · see citing works
Optimizing Spatial filters for Robust EEG Single-Trial Analysis
Blankertz, B., Tomioka, R., Lemm, S., Kawanabe, M., & Muller, K.-R. (2008) · IEEE Signal Processing Magazine · doi:10.1109/msp.2008.4408441 · Cited by 2,080 · cited 1× · see citing works
“Thus, EEG signals are almost always filtered to remove noise and artifacts prior to further analysis (Ramoser et al., 2000; Blankertz et al., 2008).”
BackgroundPlasticity during stroke recovery: from synapse to behaviour
Murphy, T. H., & Corbett, D. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2735 · Cited by 1,938 · see citing works
Delorme, A., Sejnowski, T., & Makeig, S. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.11.004 · Cited by 1,759 · see citing works
Review of the BCI Competition IV
Tangermann, M., Müller, K.-R., Aertsen, A., Birbaumer, N., Braun, C., Brunner, C., Leeb, R., Mehring, C., Miller, K. J., Müller-Putz, G. R., et al. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00055 · Cited by 1,288 · see citing works
Dry-Contact and Noncontact Biopotential Electrodes: Methodological Review
Chi, Y. M., Jung, T., & Cauwenberghs, G. (2010) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2010.2084078 · Cited by 1,069 · see citing works
Bashashati, A., Fatourechi, M., Ward, R. K., & Birch, G. E. (2007) · Journal of Neural Engineering · doi:10.1088/1741-2560/4/2/r03 · Cited by 907 · see citing works
EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks.
Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., Olmstead, R. E., Tremoulet, P. D., & Craven, P. L. (2007) · PubMed · Cited by 765 · see citing works
Brain-computer interfaces based on the steady-state visual-evoked response
Middendorf, M. S., McMillan, G. R., Calhoun, G. L., & Jones, K. S. (2000) · IEEE Transactions on Rehabilitation Engineering · doi:10.1109/86.847819 · Cited by 685 · see citing works
Blankertz, B., Muller, K.-R., Curio, G., Vaughan, T. M., Schalk, G., Wolpaw, J. R., Schlogl, A., Neuper, C., Pfurtscheller, G., Hinterberger, T., et al. (2004) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2004.826692 · Cited by 670 · see citing works
A BCI-based environmental controller for the motion-disabled
Gao, X., Xu, D., Cheng, M. J., & Gao, S. (2003) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2003.814449 · Cited by 637 · see citing works
EMG and EOG artifacts in brain computer interface systems: A survey
Fatourechi, M., Bashashati, A., Ward, R. K., & Birch, G. E. (2006) · Clinical Neurophysiology · doi:10.1016/j.clinph.2006.10.019 · Cited by 634 · see citing works
Enhanced performance by a hybrid NIRS–EEG brain computer interface
Fazli, S., Mehnert, J., Steinbrink, J. M., Curio, G., Villringer, A., Müller, K., & Blankertz, B. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.07.084 · Cited by 607 · see citing works
Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K., & Birbaumer, N. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.11.005 · Cited by 590 · see citing works
LaFleur, K., Cassady, K., Doud, A., Shades, K., Rogin, E., & He, B. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/4/046003 · Cited by 583 · see citing works
Brain-Controlled Wheelchairs: A Robotic Architecture
Carlson, T., & Millán, J. D. R. (2013) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2012.2229936 · Cited by 431 · see citing works
P300 brain computer interface: current challenges and emerging trends
Fazel-Rezai, R., Allison, B. Z., Guger, C., Sellers, E. W., Kleih, S. C., & Kübler, A. (2012) · Frontiers in Neuroengineering · doi:10.3389/fneng.2012.00014 · Cited by 362 · see citing works
Information transfer rate in a five-classes brain-computer interface
Obermaier, B., Neuper, C., Guger, C., & Pfurtscheller, G. (2001) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/7333.948456 · Cited by 338 · see citing works
Errare machinale est: the use of error-related potentials in brain-machine interfaces
Chavarriaga, R., Sobolewski, A., & Millán, J. D. R. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00208 · Cited by 315 · see citing works
Lotte, F., Larrue, F., & Mühl, C. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00568 · Cited by 307 · see citing works
Graimann, B., Allison, B. Z., & Pfurtscheller, G. (2010) · The frontiers collection · doi:10.1007/978-3-642-02091-9 · Cited by 304 · see citing works
A brain-controlled switch for asynchronous control applications
Mason, S. G., & Birch, G. E. (2000) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.871402 · Cited by 293 · see citing works
An SSVEP BCI to Control a Hand Orthosis for Persons With Tetraplegia
Ortner, R., Allison, B. Z., Korisek, G., Gaggl, H., & Pfurtscheller, G. (2010) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2010.2076364 · Cited by 281 · see citing works
Pfurtscheller, G., Solis‐Escalante, T., Ortner, R., Linortner, P., & Muller-Putz, G. R. (2010) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2010.2040837 · Cited by 276 · see citing works
Psychological predictors of SMR-BCI performance
Hammer, E. M., Halder, S., Blankertz, B., Sannelli, C., Dickhaus, T., Kleih, S., Müller, K.-R., & Kübler, A. (2012) · Biological Psychology · doi:10.1016/j.biopsycho.2011.09.006 · Cited by 267 · see citing works
A survey of sensor fusion methods in wearable robotics
Novak, D., & Riener, R. (2015) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2014.08.012 · Cited by 252 · see citing works
Self-citationLi, Y., Guan, C., Li, H., & Chin, Z. Y. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.01.030 · Cited by 228 · see citing works
Is Accurate Mapping of EMG Signals on Kinematics Needed for Precise Online Myoelectric Control?
Jiang, N., Vujaklija, I., Rehbaum, H., Graimann, B., & Farina, D. (2013) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2013.2287383 · Cited by 218 · see citing works
Comparison of Dry and Gel Based Electrodes for P300 Brain–Computer Interfaces
Guger, C., Krausz, G., Allison, B. Z., & Edlinger, G. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00060 · Cited by 190 · see citing works
Brain-computer interface controlled robotic gait orthosis
Do, A. H., Wang, P. T., King, C., Chun, S., & Nenadić, Z. (2013) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-10-111 · Cited by 190 · see citing works
Towards Independence: A BCI Telepresence Robot for People With Severe Motor Disabilities
Leeb, R., Tonin, L., Rohm, M., Desideri, L., Carlson, T., & Millan, J. D. R. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2419736 · Cited by 187 · see citing works
Rohm, M., Schneiders, M., Müller, C., Kreilinger, A., Kaiser, V., Muller-Putz, G. R., & Rupp, R. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2013.07.004 · Cited by 185 · see citing works
Cunningham, J. P., Nuyujukian, P., Gilja, V., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00503.2010 · Cited by 176 · see citing works
Onose, G., Grozea, C., Anghelescu, A., Daia, C. O., Sinescu, C. J., Ciurea, A. V., Spircu, T., Mirea, A., Andone, I., Spînu, A., et al. (2012) · Spinal Cord · doi:10.1038/sc.2012.14 · Cited by 169 · see citing works
Muller-Putz, G. R., Leeb, R., Tangermann, M., Höhne, J., Kübler, A., Febo Cincotti, A., Donatella Mattia, A., Rupp, R., Müller, K., & Millán, J. D. R. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2411333 · Cited by 158 · see citing works
Kokotilo, K. J., Eng, J. J., & Curt, A. E. P. (2009) · Journal of Neurotrauma · doi:10.1089/neu.2008.0688 · Cited by 157 · see citing works
On-Line Detection of Drowsiness Using Brain and Visual Information
Picot, A., Charbonnier, S., & Caplier, A. (2011) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2011.2164242 · Cited by 156 · see citing works
Aricò, P., Borghini, G., Di Flumeri, G., Colosimo, A., Pozzi, S., & Babiloni, F. (2016) · Progress in Brain Research · doi:10.1016/bs.pbr.2016.04.021 · Cited by 145 · see citing works
Advances in user-training for mental-imagery-based BCI control
Jeunet, C., N’Kaoua, B., & Lotte, F. (2016) · Progress in brain research · doi:10.1016/bs.pbr.2016.04.002 · Cited by 145 · see citing works
P300 speller BCI with a mobile EEG system: comparison to a traditional amplifier
Vos, M. D., Kroesen, M., Emkes, R., et al. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/036008 · Cited by 141 · see citing works
Noninvasive brain-computer interface enables communication after brainstem stroke
Sellers, E. W., Ryan, D. B., & Hauser, C. K. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3007801 · Cited by 140 · see citing works
Sajda, P., Gerson, A., Muller, K.-R., Blankertz, B., & Parra, L. (2003) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2003.814453 · Cited by 139 · see citing works
The Cybathlon promotes the development of assistive technology for people with physical disabilities
Riener, R. (2016) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-016-0157-2 · Cited by 128 · see citing works
Self-citationCurrent trends in hardware and software for brain–computer interfaces (BCIs)
Brunner, P., Bianchi, L., Guger, C., Febo Cincotti, A., & Schalk, G. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/2/025001 · Cited by 126 · see citing works
The effect of distinct mental strategies on classification performance for brain–computer interfaces
Friedrich, E. V. C., Scherer, R., & Neuper, C. (2012) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2012.01.014 · Cited by 125 · see citing works
EEG-Based Asynchronous BCI Controls Functional Electrical Stimulation in a Tetraplegic Patient
Pfurtscheller, G., Muller-Putz, G. R., Pfurtscheller, J., & Rupp, R. (2005) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/asp.2005.3152 · Cited by 122 · see citing works
Neurofeedback Training for BCI Control
Neuper, C., & Pfurtscheller, G. (2009) · The Frontiers Collection · doi:10.1007/978-3-642-02091-9_4 · Cited by 102 · see citing works
Detection of motor execution using a hybrid fNIRS-biosignal BCI: a feasibility study
Zimmermann, R., Marchal–Crespo, L., Edelmann, J., Lambercy, O., Fluet, M.-C., Riener, R., Wolf, M., & Gassert, R. (2013) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-10-4 · Cited by 93 · see citing works
Self-citationPan, J., Li, Y., Zhang, R., et al. (2013) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2013.2253801 · Cited by 89 · see citing works
Brain–computer interface using water-based electrodes
Volosyak, I., Valbuena, D., Malechka, T., Peuscher, J., & Gräser, A. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/6/066007 · Cited by 75 · see citing works
Muller-Putz, G. R., Daly, I. P., & Kaiser, V. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/035011 · Cited by 70 · see citing works
A BMI-based occupational therapy assist suit: asynchronous control by SSVEP
Sakurada, T., Kawase, T., Takano, K., Komatsu, T., & Kansaku, K. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00172 · Cited by 64 · see citing works
Williams, J. J., Rouse, A. G., Thongpang, S., Williams, J. C., & Moran, D. W. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/4/046001 · Cited by 45 · see citing works
How Much Control Is Enough? Influence of Unreliable Input on User Experience
van de Laar, B., Bos, D. P.-O., Reuderink, B., Poel, M., & Nijholt, A. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2282279 · Cited by 39 · see citing works
Guest Editorial: Brain/neuronal - Computer game interfaces and interaction
Coyle, D., Príncipe, J. C., Lotte, F., et al. (2013) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2013.2264736 · Cited by 38 · see citing works
Flight simulation using a Brain-Computer Interface: A pilot, pilot study
Kryger, M., Wester, B. A., Pohlmeyer, E. A., Rich, M., John, B., Beaty, J. D., McLoughlin, M. P., Boninger, M. L., & Tyler‐Kabara, E. C. (2016) · Experimental Neurology · doi:10.1016/j.expneurol.2016.05.013 · Cited by 35 · see citing works
Proceedings of the 5th International Brain-Computer Interface Conference 2011
Muller-Putz, G. R. (2020) · TUGraz OPEN Library (Graz University of Technology) · doi:10.3217/978-3-85125-140-1 · Cited by 33 · see citing works
A public data hub for benchmarking common brain–computer interface algorithms
Zander, T. O., Ihme, K. A., Gärtner, M., & Rötting, M. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/2/025021 · Cited by 13 · see citing works
Perdikis, S., Tonin, L., & del R. Millan, J. (2017) · IEEE Spectrum · doi:10.1109/mspec.2017.8012239 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users
Perdikis, S., Tonin, L., Saeedi, S., Schneider, C., & Millán, J. D. R. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003787 · Cited by 166 · see citing works
- Authors
- Novak, D., Sigrist, R., Gerig, N. J., Wyss, D., Bauer, R., Götz, U., & Riener, R.
- Affiliations
- University of Wyoming (US)
- ETH Zurich (CH)
- Zurich University of the Arts (CH)
- Corresponding author
- Novak, Domen
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2018-01-11
- Volume
- 11
- Pages
- 756-756
- ISSN
- 1662-453X (ISSN-L)
- PMC
- PMC5768650
- OpenAlex
- W2782903092
- Cited by
- 41
- References
- 60
- Citations by year
- 2024: 2 · 2023: 6 · 2022: 12 · 2021: 9 · 2020: 4 · 2019: 5 · 2018: 3
- FWCI
- 3.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Frontiers in Neuroscience (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
- ETH Zürich Research Collection (cc-by)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Eidgenössische Technische Hochschule Zürich
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Engineering › Biomedical Engineering › Muscle activation and electromyography studies
- Keywords
- brain-computer interface, benchmarking, assistive technology, people with disabilities, real-world performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
