CoChat
Sign Up
Cited by 57Interactive and Immersive DisplaysSpeech and dialogue systemsContext-Aware Activity Recognition Systems

Multimodal interfaces: Challenges and perspectives

Open access
Sebe, Nicu·2009·Journal of Ambient Intelligence and Smart Environments
DOI 10.3233/ais-2009-0003
Open this paper in CoChat — free accountOpen PDF

Abstract

The development of interfaces has been a technology-driven process. However, the newly developed multimodal interfaces are using recognition-based technologies that must interpret human-speech, gesture, gaze, movement patterns, and other behavioral c

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3233/ais-2009-0003

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The most informative channel for computer awareness of emotions is through facial expressions.

    “Sebe (2009) reports that the most informative channel for computer awareness of emotions, is through facial expressions.”

    From Towards multimodal emotion recognition in e-learning environments (2016) · cites this paper as Sebe, 2009

    The Introduction (page unavailable) identifies facial-expression analysis as an area of human–computer interaction, and the References list work on estimating emotions from facial expressions. Neither passage says facial expressions are the most informative channel for a computer. The Future directions and conclusion section (page unavailable) instead cautions against treating emotion cues as confined to a single channel. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • The accuracy of detecting one or more basic emotions is greatly improved when both visual and audio information are used in classification, leading to accuracy levels from 72% to 85%.

    “A recent review study by Sebe (2009) shows that the accuracy of detecting one or more basic emotions is greatly improved when both visual and audio information are used in classification, leading to accuracy levels from 72% to 85%.”

    From Towards multimodal emotion recognition in e-learning environments (2016) · cites this paper as Sebe, 2009

    Section 3.4 (Fusion; page not provided) discusses complementary audio and visual signals and joint processing, but does not report improved accuracy for basic-emotion classification. Searches for emotion-classification results and the 72% and 85% figures found no such results in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. J.K. Aggarwal and Q. Cai, "Human motion analysis: A re- view," CVIU, 73(3):428-440, 1999.
  2. M. Benali-Khoudja, M. Hafez, J.-M. Alexandre, and A. Kheddar, "Tactile interfaces: A state-of-the-art survey," Int. Symposium on Robotics, 2004.
  3. N.O. Bernsen, "Defining a taxonomy of output modalities from an HCI perspective," Computer Standards and Inter- faces, 18(6-7):537-553, 1997.
  4. M. Blattner and E. Glinert, "Multimodal integration," IEEE Multimedia, 3(4):14-24, 1996.
  5. R. Bolt, "Put-That-There: Voice and gesture at the graphics interface," Computer Graphics, 14(3):262-270, 1980.
  6. C. Borst and R. Volz, "Evaluation of a haptic mixed reality system for interactions with a virtual control panel," Pres- ence: Teleoperators and Virtual Environments, 14(6), 2005.
  7. L.S. Chen, Joint Processing of Audio-visual Information for the Recognition of Emotional Expressions in Human- computer Interaction, PhD thesis, UIUC, 2000.
  8. I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T.S. Huang, "Semi-supervised learning of classifiers: Theory, algorithms, and their applications to human-computer interaction," IEEE Trans. on PAMI, 22(12):1553-1567, 2004.
  9. P.R. Cohen and D.R. McGee, "Tangible multimodal inter- faces for safety-critical applications," Communications of the ACM, 47(1):41-46, 2004.
  10. Computer Vision and Image Understanding, Special Issue on Eye Detection and Tracking, 98(1), 2005.
  11. A. Dix, J. Finlay, G. Abowd, and R. Beale, Human- computer Interaction, Prentice Hall, 2003.
  12. Z. Duric, W. Gray, R. Heishman, F. Li, A. Rosenfeld, M. Schoelles, C. Schunn, and H. Wechsler, "Integrating per- ceptual and cognitive modeling for adaptive and intelligent human-computer interaction," Proc. of the IEEE, 90(7):1272-1289, 2002.
  13. A.T. Duchowski, "A breadth-first survey of eye tracking applications," Behavior Research Methods, Instruments, and Computing, 34(4):455-70, 2002.
  14. C. Fagiani, M. Betke, and J. Gips, "Evaluation of tracking methods for human-computer interaction," IEEE Workshop on Applications in Computer Vision, 2002.
  15. G. Fitzmaurice, H. Ishii, and W. Buxton, "Bricks: Laying the foundations for graspable user interfaces," ACM CHI, 1(442-449), 1995.
  16. D.M. Gavrila, "The visual analysis of human movement: A survey," CVIU, 73(1):82-98, 1999.
  17. K. Hook, "Designing and evaluating intelligent user inter- faces," Int. Conf. on Intelligent User Interfaces, 1999.
  18. W. Hu, T. Tan, L. Wang, and S. Maybank, "A survey on visual surveillance of object motion and behaviors", IEEE Trans. on Systems, Man, and Cybernetics, 34(3), 2004.
  19. R. Jacob, "The use of eye movements in human-computer interactions techniques: What you look at is what you get," ACM Trans. Information Systems, 9(3):152-169, 1991.
  20. A. Jaimes, "Human-centered multimedia: Culture, deploy- ment, and access", IEEE Multimedia, 13(1):12-19, 2006.
  21. A. Jaimes, D. Gatica-Perez, N. Sebe, and T.S. Huang, "Hu- man-centered Computing: Towards a Human Revolution," IEEE Computer, 40(5):30-34, 2007.
  22. A. Jaimes and N. Sebe, "Multimodal human computer inter- action: A survey," CVIU,108(1-2):116-134, 2007.
  23. R. Jain, "Folk computing," Comm. of the ACM, 46(4):27- 29, 2003.
  24. S. Kumar and P. Cohen, "Towards a fault-tolerant multi- agent system architecture," Int. Conf. on Autonomous Agents, 2000.
  25. P. Langley, "User modeling in adaptive interfaces," Int. Conf. on User Modeling, 1998.
  26. S. Marcel, "Gestures for multimodal interfaces: A Review," Technical Report IDIAP-RR 02-34, 2002.
  27. M. Maybury, Intelligent Multimedia Interfaces, AAAI/MIT Press, 1993.
  28. S. Meyer and A. Rakotonirainy, "A Survey of research on context-aware homes," Australasian Information Security Workshop Conference on ACSW Frontiers, 2003.
  29. T. B. Moeslund and E. Granum, "A survey of computer vi- sion-based human motion capture," CVIU, 81(3):231-258, 2001.
  30. I.R. Murray and J.L. Arnott, "Toward the simulation of emotion in synthetic speech: A review of the literature of human vocal emotion," J. of the Acoustic Society of Amer- ica, 93(2):1097-1108, 1993.
  31. Z. Obrenovic and D. Starcevic, "Modeling multimodal hu- man-computer interaction," IEEE Computer, pp. 65-72, September, 2004.
  32. P.Y. Oudeyer, "The production and recognition of emotions in speech: Features and algorithms," Int. J. of Human- Computer Studies, 59(1-2):157-183, 2003.
  33. A. Oulasvirta and A. Salovaara, "A cognitive meta-analysis of design approaches to interruptions in intelligent environ- ments," ACM CHI, 2004.
  34. S.L. Oviatt, "Mutual disambiguation of recognition errors in a multimodal architecture," ACM CHI, 1999.
  35. S.L. Oviatt, "Ten myths of multimodal interaction," Comm. of the ACM, 42(11):74-81, 1999.
  36. S.L. Oviatt, "User-centered Modeling and Evaluation of Multimodal Interfaces," Proc. IEEE, 91(9):1457-1468, 2004.
  37. S.L. Oviatt, T. Darrell, and M. Flickner, eds. Comm. of the ACM, Special Issue on Multimodal interfaces that flex, adapt, and persist, 47(1), 2004.
  38. S.L. Oviatt and P. Cohen, "Multimodal interfaces that proc- ess what comes naturally," Comm. of the ACM, 43(3):45- 48, 2000.
  39. S.L. Oviatt, R. Lunsford, and R. Coulston, "Individual dif- ferences in multimodal integration patterns: What are they and why do they exist?" ACM CHI, 2005.
  40. H. Pan, Z.P. Liang, T.J. Anastasio, and T.S. Huang. "Ex- ploiting the dependencies in information fusion," CVPR, vol. 2:407-412, 1999.
  41. T. Partala, V. Surakka, and T. Vanhala, "Real-time estima- tion of emotional experiences from facial expressions," In- teracting with Computers, 18(2):208-226, 2006.
  42. V.I. Pavlovic, R. Sharma and T.S. Huang, "Visual interpre- tation of hand gestures for human-computer interaction: A review", IEEE Trans. on PAMI, 19(7):677-695, 1997.
  43. A. Pentland, "Looking at people," Comm. of the ACM, 43(3):35-44, 2000.
  44. M. Porta, "Vision-based user interfaces: methods and appli- cations," Int. J. Human-Computer Studies, 57(1):27-73, 2002.
  45. G. Potamianos, C. Neti, J. Luettin, and I. Matthews, "Au- dio-visual automatic speech recognition: An overview," Is- sues in Visual and Audio-Visual Speech Processing, MIT Press, 2004.
  46. L.M. Reeves, J-C. Martin, M. McTear, T. Raman, K. Stan- ney, H. Su, Q. Wang, J. Lai, J. Larson, S. Oviatt, T. Balaji, S. Buisine, P. Collings, P. Cohen, and B. Kraal, "Guidelines for multimodal user interface design," Comm. of the ACM, 47(1):57-69, 2004.
  47. E. Schapira and R. Sharma, "Experimental evaluation of vi- sion and speech based multimodal interfaces," Workshop on Perceptive User Interfaces, pp. 1-9, 2001.
  48. T. Selker, "Visual attentive interfaces," BT Technology Journal, 22(4):146-150, 2004.
  49. B. Shneiderman, Leonardo's Laptop: Human Needs and the New Computing Technologies, MIT Press, 2002.
  50. O. Stock, and M. Zancanaro, (eds.). Multimodal Intelligent Information Presentation. Series Text, Speech and Lan- guage Technology. Kluwer, pp. 325-340, 2005.
  51. M. Turk and M. Kölsch, "Perceptual interfaces," G. Medi- oni and S.B. Kang, eds., Emerging Topics in Computer Vi- sion, Prentice Hall, 2004.
  52. M. Turunen, J. Hakulinen, K.-J. Räihä, E.-P. Salonen, A. Kainulainen, P. Prusi, "An architecture and applications for speech-based accessibility systems," IBM Systems, 44(3):485-504, 2005.
  53. R. Vertegaal, ed., "Attentive user interfaces: Special issue," Comm. of the ACM, 46(3), 2003.
  54. J.J.L. Wang and S. Singh, "Video analysis of human dy- namics -A survey," Real-Time Imaging, 9(5):321-346, 2003.
  55. M. Weiser, "Some computer science issues in ubiquitous computing," Comm. of the ACM, 36(7):74-83, 1993.
  56. M.-H. Yang, D. Kriegman, and N. Ahuja, "Detecting faces in images: A survey," IEEE Trans. on PAMI, 24(1):34-58, 2002.
  57. H. Yoshikawa, "Modeling humans in human-computer in- teraction," in Human-computer Interaction Handbook: Fun- damentals, Evolving Technologies and Emerging Applica- tions, pp. 118-146, 2002.
  58. W. Zhao, R. Chellappa, A. Rosenfeld, and J. Phillips, "Face recognition: A literature survey," ACM Computing Surveys, 12:399-458, 2003.

Cited by

Sorted by citations

Authors
Sebe, N.
Affiliations
University of Trento (IT)
Corresponding author
Sebe, Nicu
ORCID
Sebe, Nicu
Venue
Journal of Ambient Intelligence and Smart Environments
Publisher
SAGE Publications
Published
2009
Volume
1
Issue
1
Pages
23-30
ISSN
1876-1364 (ISSN-L)
DOI
10.3233/ais-2009-0003
OpenAlex
W2103524533
Cited by
57
References
3
Citations by year
2025: 2 · 2024: 2 · 2022: 3 · 2021: 1 · 2020: 3 · 2019: 3 · 2018: 1 · 2017: 2 · 2016: 6 · 2015: 7 · 2014: 5 · 2013: 3 · 2012: 9
FWCI
10.13
Type
journal article
Language
en
Open access
Locations
  • Journal of Ambient Intelligence and Smart Environments (publishedVersion)
  • Institutional Research Information System (Università degli Studi di Trento) (submittedVersion)
  • INFM-OAR (INFN Catania) (submittedVersion, other-oa)
  • Zenodo (CERN European Organization for Nuclear Research) (other-oa)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
Keywords
multimodal interfaces, speech recognition, gesture recognition, gaze recognition, movement pattern recognition, human-computer interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review