Mutual disambiguation of recognition errors in a multimodel architecture
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
As a new generation of multimodal/media systems begins to define itself, researchers are attempting to learn how to combine different modes into strategically integrated whole systems. In theory, well designed multimodal systems should be able to integrate complementary modalities in a manner that supports mutual disambiguation (MD) of errors and leads to more robust performance. In this study, over 2,000 multimodal utterances by both native and accented speakers of English were processed by a multimodal system, and then logged and analyzed. The results confirmed that multimodal systems can indeed support significant levels of MD, and also higher levels of MD for the more challenging accented users. As a result, although speech recognition as a stand-alone performed far more poorly for accented speakers, their multimodal recognition rates did not differ from those of native speakers. Implications are discussed for the development of future multimodal architectures that can perform in a more robust and stable manner than individual recognition technologies. Also discussed is the design of interfaces that support diversity in tangible ways, and that function well under challenging real-world usage conditions,
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
“Put-that-there”: Voice and gesture at the graphics interface
Bolt, R. A. (1980) · Proceedings of the 7th annual conference on Computer graphics and interactive techniques - SIGGRAPH '80 · doi:10.1145/800250.807503 · Cited by 1,445
“Put-that-there”: Voice and gesture at the graphics interface
Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921
Ten myths of multimodal interaction
Oviatt, S. (1999) · Communications of the ACM · doi:10.1145/319382.319398 · Cited by 689
Self-citationThe logic of typed feature structures
Carpenter, B. (1992) · Cambridge University Press eBooks · Cited by 607
QuickSet: multimodal interaction for distributed applications
Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., & Clow, J. (1997) · Proceedings of the fifth ACM international conference on Multimedia - MULTIMEDIA '97 · doi:10.1145/266180.266328 · Cited by 466
Self-citationIntegration and synchronization of input modes during multimodal human-computer interaction
Oviatt, S., DeAngeli, A., & Kuhn, K. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258821 · Cited by 360
Self-citationUnification-based multimodal integration
Johnston, M., Cohen, P. R., McGee, D., Oviatt, S. L., Pittman, J. A., & Smith, I. (1997) · Proceedings of the 35th annual meeting on Association for Computational Linguistics - · doi:10.3115/976909.979653 · Cited by 174
Self-citationSynergistic use of direct manipulation and natural language
Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C., & Sullivan, J. W. (1989) · Proceedings of the SIGCHI conference on Human factors in computing systems Wings for the mind - CHI '89 · doi:10.1145/67449.67494 · Cited by 142
Intelligent multi-media interface technology
Neal, J. G., & Shapiro, S. C. (1991) · Intelligent User Interfaces · doi:10.1145/107215.128690 · Cited by 112
Linguistic Adaptations During Spoken and Multimodal Error Resolution
Oviatt, S., Bernard, J., & Levow, G.-A. (1998) · Language and Speech · doi:10.1177/002383099804100409 · Cited by 80
Self-citation
Cited by
Multimodal human–computer interaction: A survey
Jaimes, A., & Sebe, N. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.019 · Cited by 965
Ten myths of multimodal interaction
Oviatt, S. (1999) · Communications of the ACM · doi:10.1145/319382.319398 · Cited by 689
Self-citationOviatt, S. (2007) · Human Factors and Ergonomics · doi:10.1201/9781410615862.ch21 · Cited by 485
Self-citationOviatt, S., Cohen, P., Wu, L., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., et al. (2000) · Human–Computer Interaction · doi:10.1207/s15327051hci1504_1 · Cited by 375
Self-citationHuman-centered design meets cognitive load theory: designing interfaces that help people think
Oviatt, S. (2006) · Proceedings of the 14th ACM international conference on Multimedia · doi:10.1145/1180639.1180831 · Cited by 315
Self-citationTaming recognition errors with a multimodal interface
Oviatt, S. (2000) · Communications of the ACM · doi:10.1145/348941.348979 · Cited by 299
Self-citationMultimodal human discourse: gesture and speech
Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C., McCullough, K. E., & Ansari, R. (2002) · ACM Transactions on Computer-Human Interaction · doi:10.1145/568513.568514 · Cited by 291
When do we interact multimodally?: cognitive load and multimodal communication patterns
Oviatt, S., Coulston, R., & Lunsford, R. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027957 · Cited by 285
Self-citationPerceptual user interfaces: multimodal interfaces that process what comes naturally
Oviatt, S., & Cohen, P. (2000) · Communications of the ACM · doi:10.1145/330534.330538 · Cited by 278
Self-citationHuman Values, Ethics, and Design
(2002) · The Human-Computer Interaction Handbook · doi:10.1201/9781410606723-48 · Cited by 270
Suede: a Wizard of Oz prototyping tool for speech user interfaces
Klemmer, S. R., Sinha, A. K., Chen, J., Landay, J. A., Aboobaker, N., & Wang, A. (2000) · Proceedings of the 13th annual ACM symposium on User interface software and technology - UIST '00 · doi:10.1145/354401.354406 · Cited by 230
Multimodal error correction for speech user interfaces
Suhm, B., Myers, B., & Waibel, A. (2001) · ACM Transactions on Computer-Human Interaction · doi:10.1145/371127.371166 · Cited by 198
Multimodal integration-a statistical view
Wu, L., Oviatt, S. L., & Cohen, P. R. (1999) · IEEE Transactions on Multimedia · doi:10.1109/6046.807953 · Cited by 190
Self-citationGeneration and evaluation of user tailored responses in multimodal dialogue
Walker, M. A., Whittaker, S. J., Stent, A., Maloor, P., Moore, J., Johnston, M., & Vasireddy, G. (2004) · Cognitive Science · doi:10.1016/j.cogsci.2004.06.002 · Cited by 125
Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511599 · Cited by 119
“One goal of a well-designed multimodal system is to integrate complementary input modes to create a synergistic blend, permitting each mode’s strengths to overcome weaknesses in the others and support "mutual compensation" of feedback errors [71].”
BackgroundMultimodal interfaces: Challenges and perspectives
Sebe, N. (2009) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-2009-0003 · Cited by 57
Bourguet, M.-L. (2006) · Signal Processing · doi:10.1016/j.sigpro.2006.02.047 · Cited by 30
Effective error recovery strategies for multimodal form-filling applications
Sturm, J., & Boves, L. (2005) · Speech Communication · doi:10.1016/j.specom.2004.11.007 · Cited by 29
Vouch: multimodal touch-and-voice input for smart watches under difficult operating conditions
Lee, J., Lee, C., & Kim, G. J. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-017-0246-y · Cited by 16
Multimodal interaction, interfaces, and analytics
Oviatt, S. (2022) · Handbook of Human Computer Interaction · doi:10.1007/978-3-319-27648-9_22-1 · Cited by 12
Self-citation
Metadata
- Authors
- Oviatt, S.
- Venue
- Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99
- Publisher
- ACM Press
- Published
- 1999
- Pages
- 576-583
- OpenAlex
- W1977011589
- Cited by
- 315
- References
- 13
- FWCI
- 15.28
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.