Cited by 507Hand Gesture Recognition SystemsAdvanced Sensor and Energy Harvesting MaterialsTactile and Sensory Interactions

The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.

AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

Open access
Wen, Feng, Zhang, Zixuan, He, Tianyiyi, Lee, Chengkuo·2021·Nature Communications

Abstract

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper presents an AI-enabled sign language recognition and communication system using a 15-channel triboelectric smart glove, deep learning, and a VR interface. It recognizes 50 words and 20 sentences, and with segmentation can reconstruct sentences and recognize new/never-seen sentences at an average correct rate of 86.67%.

Key findings

  • Nonsegmentation CNN achieved 91.3% accuracy for 50 words and 95% for sentences; the abstract/discussion report 20 sentences, while Results and Fig. 3j describe 17 sentences.
  • Segmentation with a hierarchy classifier achieved 82.81% accuracy for word fragments and 85.58% average correct rate for sentence reconstruction; the single classifier achieved 81.9% and 79.41%, respectively.
  • New/never-seen sentences created by recombining learned word units in new orders were recognized at 86.67% average correct rate with the hierarchy classifier, versus 60% with the single classifier.
  • The glove uses 15 triboelectric sensors; hand motion accounts for 43% of sign language motions, with finger bending (56%), wrist motion (18%), fingertip touch (16%), and palm interaction (10%) among hand motions.
  • The VR interface enables bidirectional communication: sign language is translated into text/audio and sent via TCP/IP to a nonsigner, who can type responses; a greeting conversation between avatars Lily and Mary is demonstrated.

Methods

  • Triboelectric sensors were fabricated from Ecoflex and wrinkled nitrile with conductive textile electrodes, encapsulated, and stitched/glued onto a cotton glove; 15 sensors were placed on fingers, wrists, right index/middle fingertips, and left palm.
  • For nonsegmentation, 100 samples per word and sentence were collected, with 80% training and 20% testing, yielding 5000 word samples and 2000 sentence samples. For segmentation, 50 samples per sentence for 17 sentences (850 total) were split with a 200-point sliding window at 50-point steps, producing 13 elements per 800-point sentence; 60%/20%/20% were used for training/validation/testing, and 5 samples for 3 new sentences tested new-sentence recognition.
  • Main analysis used an optimized 1D CNN (5 kernels, 64 filters, 4 convolutional layers), PCA and t-SNE for visualization, single and hierarchy classifiers for segmentation, and a Unity VR interface with TCP/IP client–server communication.

Limitations

  • No dedicated Limitations section is present. The paper states that the nonsegmentation method cannot distinguish new/never-seen sentences even when they contain the same word elements in a different order, and it introduces recognition time latency; the segmentation method has compromised recognition accuracy.
  • The paper states the demonstration is limited to 50 words and 20 sentences from an American Sign Language phrase book and says the database can be broadened; new-sentence recognition reaches 86.67%, not perfect, and the single classifier is lower at 60%.
  • Evident from the text: the VR communication demonstration is a greeting scenario between two avatars under an identical LAN, so broader real-world deployment and larger user populations are not tested in the paper.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-021-25637-w

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 The glove configuration and sensor characterization. a Schematics of the sign language recognition and communication system. b Proportion of different motions that are commonly used in sign language helps determine sensor position on gloves. c Sensor position on gloves based on hand motion analysis in b. Detailed area information of the sensor on each position can be found in Supplementary Fig. 1. d Materials of the triboelectric sensor. e–h Voltage output dependence on key parameters, including sensor area, force, bending degree, and bending speed. The hand, head, and phone images are created by the authors via Blender.
Fig. 1 The glove configuration and sensor characterization. a Schematics of the sign language recognition and communication system. b Proportion of different motions that are commonly used in sign language helps determine sensor position on gloves. c Sensor position on gloves based on hand motion analysis in b. Detailed area information of the sensor on each position can be found in Supplementary Fig. 1. d Materials of the triboelectric sensor. e–h Voltage output dependence on key parameters, including sensor area, force, bending degree, and bending speed. The hand, head, and phone images are created by the authors via Blender.CC BY · PubMed Central
Fig. 2 The data analysis for signals of 50 words and 20 sentences. a Part of representatives among 50 words or gestures (showing 19 gestures here), in which the opaque and translucent gesture images show the starting and final state of the gesture, respectively. The rest of 31 gesture photo and their corresponding triboelectric signals can be found in Supplementary Fig. 2. b Triboelectric voltage output of 19 words (top), and the similarity and correlation analysis based on the word signals (bottom). The high correlation coefficient of ‘Get’ and ‘Must’ shows a high similarity between these two gesture signals, indicating a high possibility for wrong classification. c Correlation coefficient matrix of signals of 50 words. d Correlation coefficient distribution curve of 50 words. e Correlation coefficient matrix of 20 sign language sentences. f Correlation coefficient distribution curve of 20 sentences. g Voltage output of 20 sentences. Photo credit: Feng Wen, National University of Singapore. Source data65 of Figs. 2c, e are provided in Harvard Dataverse.
Fig. 2 The data analysis for signals of 50 words and 20 sentences. a Part of representatives among 50 words or gestures (showing 19 gestures here), in which the opaque and translucent gesture images show the starting and final state of the gesture, respectively. The rest of 31 gesture photo and their corresponding triboelectric signals can be found in Supplementary Fig. 2. b Triboelectric voltage output of 19 words (top), and the similarity and correlation analysis based on the word signals (bottom). The high correlation coefficient of ‘Get’ and ‘Must’ shows a high similarity between these two gesture signals, indicating a high possibility for wrong classification. c Correlation coefficient matrix of signals of 50 words. d Correlation coefficient distribution curve of 50 words. e Correlation coefficient matrix of 20 sign language sentences. f Correlation coefficient distribution curve of 20 sentences. g Voltage output of 20 sentences. Photo credit: Feng Wen, National University of Singapore. Source data65 of Figs. 2c, e are provided in Harvard Dataverse.CC BY · PubMed Central
Fig. 3 Word and sentence recognition based on the nonsegmentation method. a–c Optimization of CNN structure parameters based on accuracy performance, including Kernel size, the number of filters, and the number of convolutional layers. Boxplots indicate median (middle line), 25th, 75th percentile (box) and 5th and 95th percentile (whiskers) as well as outliers (single points). d Final structure of CNN after optimization. e–f Cluster results of word signal from CNN input layer and output layer. g Confusion map of recognizing 50 words. h–i Cluster results of sentence signal from CNN input layer and output layer. j Confusion map of recognizing 17 sentences. Source data65 of Figs. 3g, j are provided in Harvard Dataverse.
Fig. 3 Word and sentence recognition based on the nonsegmentation method. a–c Optimization of CNN structure parameters based on accuracy performance, including Kernel size, the number of filters, and the number of convolutional layers. Boxplots indicate median (middle line), 25th, 75th percentile (box) and 5th and 95th percentile (whiskers) as well as outliers (single points). d Final structure of CNN after optimization. e–f Cluster results of word signal from CNN input layer and output layer. g Confusion map of recognizing 50 words. h–i Cluster results of sentence signal from CNN input layer and output layer. j Confusion map of recognizing 17 sentences. Source data65 of Figs. 3g, j are provided in Harvard Dataverse.CC BY · PubMed Central
Fig. 4 Word and sentence recognition based on segmentation method, which brings the feasibility of new/never-seen sentence recognition. a Label table (W01–W20) of 19 words (they are among the total 50 words) that present in 20 sentences (using Y1–Y17 and New1–New3 denote). b Schematic diagram of sentence signal segmentation, ‘The dog scared me’ is taken as an example. c Summary table of sentences with category remarks, comprised words, label series, and unique labeled number order. The same word using the same color to mark. d Schematic diagram of single classifier. e Confusion map of split word element recognition (accuracy 81.9%) based on single classifier. f With successful recognition of each element in sentences, the sentences can be inversely reconstructed and recognized at an average correct rate of 79.41% within single classifier. The dark green means right recognition and light blue means wrong prediction. g Schematic diagram of the hierarchy classifier. h Confusion map of segmented word element recognition (accuracy 82.8%) based on hierarchy classifer. i With successful recognition of each element in sentences, the sentence can be inversely reconstructed and recognition at an average correct rate of 85.58% within hierarchy classifier. j Recognition process of three new sentences that the CNN model did not learn before, taking ‘I lost my dog’ as an example. The detailed recognition results can be found in Supplementary Table 2. Source data65 of Figs. 4e, h are provided in Harvard Dataverse.
Fig. 4 Word and sentence recognition based on segmentation method, which brings the feasibility of new/never-seen sentence recognition. a Label table (W01–W20) of 19 words (they are among the total 50 words) that present in 20 sentences (using Y1–Y17 and New1–New3 denote). b Schematic diagram of sentence signal segmentation, ‘The dog scared me’ is taken as an example. c Summary table of sentences with category remarks, comprised words, label series, and unique labeled number order. The same word using the same color to mark. d Schematic diagram of single classifier. e Confusion map of split word element recognition (accuracy 81.9%) based on single classifier. f With successful recognition of each element in sentences, the sentences can be inversely reconstructed and recognized at an average correct rate of 79.41% within single classifier. The dark green means right recognition and light blue means wrong prediction. g Schematic diagram of the hierarchy classifier. h Confusion map of segmented word element recognition (accuracy 82.8%) based on hierarchy classifer. i With successful recognition of each element in sentences, the sentence can be inversely reconstructed and recognition at an average correct rate of 85.58% within hierarchy classifier. j Recognition process of three new sentences that the CNN model did not learn before, taking ‘I lost my dog’ as an example. The detailed recognition results can be found in Supplementary Table 2. Source data65 of Figs. 4e, h are provided in Harvard Dataverse.CC BY · PubMed Central
Fig. 5 The demonstration of communication between the speech impaired and the nonsigner. a Flow chart of the sign language recognition and communication system, which allows the signer to use sign language and nonsigner to types directly to engage in the interaction. The delivered sign language by the signer is recognized and translated into text and speech by AI block. Based on TCP/IP, the client (controlled by signer Lily) in VR interface receives the recognition results and transmits to the sever (operated by the nonsigner Mary). The nonsigner types on the chat box to respond to the signer. b (i–v) Communication/conversation process in VR interface between the speech-disordered user Lily and nonsigner Mary based on the sign language recognition and communication system. The red rectangle indicates the corresponding reaction of these two users. These photos are of one of the authors. c Conversation summary of b. The hand image is created by the authors via Blender. Photo credit: Feng Wen, National University of Singapore.
Fig. 5 The demonstration of communication between the speech impaired and the nonsigner. a Flow chart of the sign language recognition and communication system, which allows the signer to use sign language and nonsigner to types directly to engage in the interaction. The delivered sign language by the signer is recognized and translated into text and speech by AI block. Based on TCP/IP, the client (controlled by signer Lily) in VR interface receives the recognition results and transmits to the sever (operated by the nonsigner Mary). The nonsigner types on the chat box to respond to the signer. b (i–v) Communication/conversation process in VR interface between the speech-disordered user Lily and nonsigner Mary based on the sign language recognition and communication system. The red rectangle indicates the corresponding reaction of these two users. These photos are of one of the authors. c Conversation summary of b. The hand image is created by the authors via Blender. Photo credit: Feng Wen, National University of Singapore.CC BY · PubMed Central

Verification

Claims the paper makes

35 Own result · 13 Own interpretation · 8 No basis given · 3 Not checked yet · 1 Own method

Verify all 3 references — free account

Show all 33 claims

  • It may state two separate claims as one. 23
  • Probably not a substantial claim. 6
27 of 27
  • The paper relies on this without support.

    New/never-seen sentences created by new-order word element recombination are valid sign language sentences that a signer would use.

    “new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%.”
  • The paper relies on this without support.

    The 50 gestures and 20 sentences selected are representative of daily sign language communication.

    “we select 50 gestures that are daily used in the signer’s life as demonstration.”
    “20 daily sentences are sorted out for subsequent study.”
  • The paper relies on this without support.

    Sign language is not as prevalent as spoken language and is difficult for nonsigners to understand without prior learning.

    “sign language is not as prevalent as the speaking language and difficult for nonsigners to understand without prior learning.”
  • The paper relies on this without support.

    Most current sign language translation systems simply display recognition results on visualized interfaces such as mobile phones and portable displays.

    “Most current sign language translation systems simply display the recognition result on visualized interfaces such as mobile phones and portable displays.”
  • The paper relies on this without support.

    Most works do not allow interaction between signers and nonsigners.

    “since most of works do not allow the interaction between signers and nonsigners.”
  • The paper relies on this without support.

    There is an acute demand for effective two-way communication between signers and nonsigners.

    “However, the acute demand of effective two-way communication between the signers and nonsigners is neglected”
  • The paper relies on this without support.

    The emerging VR interface is an ideal platform for advanced sign language interpretation, visualization, and communication interface.

    “which would be an ideal platform for an advanced sign language interpretation, visualization, and communication interface.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Cited by

Sorted by citations

See all 507

Metadata

Authors
Wen, F., Zhang, Z., He, T., & Lee, C.
Affiliations
National University of Singapore (SG)
Corresponding author
Lee, Chengkuo
Venue
Nature Communications
Published
2021-09-10
Volume
12
Issue
1
Pages
5378-5378
ISSN
2041-1723 (ISSN-L)
OpenAlex
W3197059061
Cited by
494
References
73
Citations by year
2026: 58 · 2025: 121 · 2024: 121 · 2023: 107 · 2022: 78 · 2021: 9
FWCI
48.31
Type
journal article
Language
en
Open access
Locations
  • Nature Communications (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • National University of Singapore (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Key Research and Development Program of China (2019YFB2004800, R‐2020‐S‐002)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Engineering › Biomedical Engineering › Advanced Sensor and Energy Harvesting Materials
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
sign language recognition, sentence recognition, virtual reality communication, two-way communication, hearing loss, word segmentation, gesture sensing, deep learning
MeSH terms
Virtual Reality, Wearable Electronic Devices, Deep Learning, Communication Devices for People with Disabilities, Communication Aids for Disabled, Deafness, Gestures, Humans, Sign Language

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review