CoChat

Study of Multimodal Interfaces and the Improvements on Teleoperation

Open access

The DOI matches its Crossref record; 68 of the 72 references it cites were found in a registry or an index; 7 of the 56 sources its claims cite were checked for support.

Abstract

Research in multimodal interfaces aims to provide immersive solutions and to increase overall human performance. A promising direction is to combine auditory, visual and haptic interaction between the user and the simulated environment. However, no extensive comparison exists to show how combining audiovisuohaptic interfaces would affect human perception and by extent reflected on task performance. Our paper explores this idea and presents a thorough, full-factorial comparison of how all combinations of audio, visual and haptic interfaces affect performance during manipulation. We evaluated how each combination affects the performance in a study (N=25 ) consisting of manipulation tasks with various difficulties. The overall performance was assessed using both subjective measures, by assessing cognitive workload and system usability, and objective measurements, by incorporating time and spatial accuracy-based metrics. The results showed that regardless of task complexity, the combination of stereoscopic-vision with the virtual reality headset increased performance across all measurements by 40%, compared to monocular-vision from a generic display monitor. Besides, using haptic feedback improved outcomes by 10% and auditory feedback accounted for approximately 5% improvement.

Overview

CoChat Summary

## TL;DR

This paper reports a full-factorial study of 8 multimodal teleoperation interface conditions across 25 participants performing pick-and-place manipulation tasks of varying difficulty. The strongest performance gains came from stereoscopic VRHMD vision, with additional benefits from haptic feedback and only limited effects from auditory feedback.

## Key findings

  • Across subjective and objective measures, stereoscopic VRHMD vision outperformed the monocular monitor; the paper reports an overall performance increase of around 40% with VRHMD compared to the monitor.
  • Adding somatosensory (haptic) feedback improved outcomes by a further 10% overall, but this benefit was significant mainly when paired with stereoscopic vision.
  • Auditory feedback had no significant effect on most measures; the paper reports only an approximately 5% improvement in spatial accuracy.
  • For task success rate, the stereoscopic conditions (C5–C8) achieved 96.22%, 99.11%, 96.22%, and 97.55%, versus 39.33%, 47.55%, 51.33%, and 48.22% for the monocular conditions (C1–C4).
  • For workload and usability, the paper reports strong condition effects: perceived workload F(2.393, 57.437)=70.473, p<0.001, ηp²=0.746 and SUS usability F(3.930, 94.310)=97.064, p<0.001, ηp²=0.802.

## Methods

  • Full-factorial repeated-measures experiment with 8 interface conditions combining vision (monitor vs VRHMD), audio (off/on), and haptics (off/on), tested on a pick-and-place task in a virtual environment.
  • Sample: 25 participants from the University of Edinburgh, ages 21–44 (mean 26.36, SD 4.829), with 6 females and 19 males.
  • Main analyses used RM-ANOVA for normally distributed data, ART + RM-ANOVA for ordinal questionnaire data, Friedman/Wilcoxon for nonparametric time metrics, and a three-sigma rule for Bernoulli success proportions.

## Limitations

  • The paper states it did not test how auditory or somatosensory feedback might compensate for overloaded visual information.
  • The paper states it did not explicitly and strictly investigate monocular vs stereoscopic visual feedback by themselves.
  • Evident from the text: the study uses a static scenario and the authors note that conclusions about vection and motion sickness cannot be fully drawn from it, which may limit generalization to navigation or more dynamic teleoperation tasks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2020.2990080

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

FIGURE 1 Diagram of the simulation setup with all the software plugins used.
FIGURE 1 Diagram of the simulation setup with all the software plugins used.From the PDF
FIGURE 2 Hand control approach through direct joint angle re-targeting from our custom haptic glove to the final robotic hand.
FIGURE 2 Hand control approach through direct joint angle re-targeting from our custom haptic glove to the final robotic hand.From the PDF
FIGURE 3 Haptic glove (left module shown) in its final and first iteration with it's electronics and motors exposed in the latter.
FIGURE 3 Haptic glove (left module shown) in its final and first iteration with it's electronics and motors exposed in the latter.From the PDF
FIGURE 4 Image (left): All manipulation tasks illustrating the different three dimensional shapes, sizes as well as distances from 150, 300 and 600; green, yellow and red respectively. Tree (right): All 18 tasks broken down in type of object shapes (red), sizes (blue) and distances (green).
FIGURE 4 Image (left): All manipulation tasks illustrating the different three dimensional shapes, sizes as well as distances from 150, 300 and 600; green, yellow and red respectively. Tree (right): All 18 tasks broken down in type of object shapes (red), sizes (blue) and distances (green).From the PDF
FIGURE 5 Box plot illustration across all eight interface conditions of the mean perceived workload, with higher scoring equal to worse performance. Dots represent outliers.
FIGURE 5 Box plot illustration across all eight interface conditions of the mean perceived workload, with higher scoring equal to worse performance. Dots represent outliers.From the PDF
FIGURE 6 Box plot illustration across all eight interface conditions of the mean interface usability, with higher scoring equal to better performance. Dots represent outliers.
FIGURE 6 Box plot illustration across all eight interface conditions of the mean interface usability, with higher scoring equal to better performance. Dots represent outliers.From the PDF
FIGURE 7 Heat-map illustrating the proportion of task success rate going from lower to higher complexity, horizontal axis A.1.1 (left) to C.2.3 (right), across all the interface conditions C1 to C8, vertical axis.
FIGURE 7 Heat-map illustrating the proportion of task success rate going from lower to higher complexity, horizontal axis A.1.1 (left) to C.2.3 (right), across all the interface conditions C1 to C8, vertical axis.From the PDF
Image 8
From the PDF
Image 9
From the PDF
Image 10
From the PDF
FIGURE 8 Objective measurements represented in a bar graph in addition to standard error. From left to right, time-based metrics mean placement (opaque) and completion time (slightly transparent). Followed by spatial based metrics, specifically distance error, position and rotation accuracy. For all illustrated measurement results, all combinations between monocular and stereoscopic vision, i.e. C1, C2, C3, C4 with C5,C6,C7 and C8 are highly significant, p < 0 : 001 and as a result significance bars are not visualised. For the remaining combinations, significance bars are shown, with each indicating * p < 0 : 05, ** p < 0 : 01, *** p < 0 : 001 and finally n.s not significant.
FIGURE 8 Objective measurements represented in a bar graph in addition to standard error. From left to right, time-based metrics mean placement (opaque) and completion time (slightly transparent). Followed by spatial based metrics, specifically distance error, position and rotation accuracy. For all illustrated measurement results, all combinations between monocular and stereoscopic vision, i.e. C1, C2, C3, C4 with C5,C6,C7 and C8 are highly significant, p < 0 : 001 and as a result significance bars are not visualised. For the remaining combinations, significance bars are shown, with each indicating * p < 0 : 05, ** p < 0 : 01, *** p < 0 : 001 and finally n.s not significant.From the PDF
FIGURE 9 Example recordings of eight different participants during the manipulation experiment.
FIGURE 9 Example recordings of eight different participants during the manipulation experiment.From the PDF
FIGURE 10 Overall interface effectiveness through linear regression, across all measurements and across all tasks with an increasing task complexity from lower to higher. Width of the shapes represents the effectiveness, the wider the higher. Colouring also indicates the effectiveness increasing from red to green. The overall effectiveness is calculated linearly, specifically, the measurements are weighted 1 1 = Vmax where Vmax is the maximum limit of the measurement. The data points from the scatter plot have been line fitted through linear regression to visualize a cone-like illustration. The width of the cone represents the effectiveness while the height of the cone the effectiveness of the interface at the specific task complexity. The specific task complexity is discussed in section ''Manipulation Tasks of Varying Complexity''.
FIGURE 10 Overall interface effectiveness through linear regression, across all measurements and across all tasks with an increasing task complexity from lower to higher. Width of the shapes represents the effectiveness, the wider the higher. Colouring also indicates the effectiveness increasing from red to green. The overall effectiveness is calculated linearly, specifically, the measurements are weighted 1 1 = Vmax where Vmax is the maximum limit of the measurement. The data points from the scatter plot have been line fitted through linear regression to visualize a cone-like illustration. The width of the cone represents the effectiveness while the height of the cone the effectiveness of the interface at the specific task complexity. The specific task complexity is discussed in section ''Manipulation Tasks of Varying Complexity''.From the PDF
Image 14
From the PDF
Image 15
From the PDF
Image 16
From the PDF
Image 17
From the PDF

Verification

Claims the paper makes

6 Valid · 2 Not valid · 33 Not checked · 57 Own results and interpretations

60 more claims

Create a free account to see all 98 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 17
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Research works printed but not linked: 3

  1. simpson, 2013, Spatial audio display concepts supporting situation awareness for operators of unmanned aerial vehicles, Human Performance Situation Awareness and Automation Current Research and Trends, 2, 61
  2. richard, 1994, A comparison of haptic, visual and auditive force feedback for deformable virtual objects, Proc Internation Conf Autom Technol (ICAT), 49, 62
  3. aleotti, 2002, A Multi-modal User Interface for Remote Object Exploration in Teleopera-tion Systems

Cited by

See all 105

Metadata

Authors
Triantafyllidis, E., Mcgreavy, C., Gu, J., & Li, Z.
Affiliations
University of Edinburgh (GB)
Venue
IEEE Access
Published
2020
Volume
8
Pages
78213-78227
ISSN
2169-3536 (ISSN-L)
OpenAlex
W3019814707
Cited by
105
References
72
Citations by year
2026: 10 · 2025: 17 · 2024: 25 · 2023: 17 · 2022: 16 · 2021: 20
FWCI
6.52
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
EPSRC Future AI and Robotics for Space (EP/R026092/1)
Topics
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
  • Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
  • Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
Keywords
multimodal interfaces, teleoperation, haptic feedback, auditory feedback, stereo vision, virtual reality headsets, manipulation tasks, mental workload, system usability, task performance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.