Cited by 507Hand Gesture Recognition SystemsAdvanced Sensor and Energy Harvesting MaterialsTactile and Sensory Interactions

AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

Open access
Wen, Feng, Zhang, Zixuan, He, Tianyiyi, Lee, Chengkuo·2021·Nature Communications

Abstract

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper presents an AI-enabled sign language recognition and communication system using a 15-channel triboelectric smart glove, deep learning, and a VR interface. It recognizes 50 words and 20 sentences, and with segmentation can reconstruct sentences and recognize new/never-seen sentences at an average correct rate of 86.67%.

Key findings

  • Nonsegmentation CNN achieved 91.3% accuracy for 50 words and 95% for sentences; the abstract/discussion report 20 sentences, while Results and Fig. 3j describe 17 sentences.
  • Segmentation with a hierarchy classifier achieved 82.81% accuracy for word fragments and 85.58% average correct rate for sentence reconstruction; the single classifier achieved 81.9% and 79.41%, respectively.
  • New/never-seen sentences created by recombining learned word units in new orders were recognized at 86.67% average correct rate with the hierarchy classifier, versus 60% with the single classifier.
  • The glove uses 15 triboelectric sensors; hand motion accounts for 43% of sign language motions, with finger bending (56%), wrist motion (18%), fingertip touch (16%), and palm interaction (10%) among hand motions.
  • The VR interface enables bidirectional communication: sign language is translated into text/audio and sent via TCP/IP to a nonsigner, who can type responses; a greeting conversation between avatars Lily and Mary is demonstrated.

Methods

  • Triboelectric sensors were fabricated from Ecoflex and wrinkled nitrile with conductive textile electrodes, encapsulated, and stitched/glued onto a cotton glove; 15 sensors were placed on fingers, wrists, right index/middle fingertips, and left palm.
  • For nonsegmentation, 100 samples per word and sentence were collected, with 80% training and 20% testing, yielding 5000 word samples and 2000 sentence samples. For segmentation, 50 samples per sentence for 17 sentences (850 total) were split with a 200-point sliding window at 50-point steps, producing 13 elements per 800-point sentence; 60%/20%/20% were used for training/validation/testing, and 5 samples for 3 new sentences tested new-sentence recognition.
  • Main analysis used an optimized 1D CNN (5 kernels, 64 filters, 4 convolutional layers), PCA and t-SNE for visualization, single and hierarchy classifiers for segmentation, and a Unity VR interface with TCP/IP client–server communication.

Limitations

  • No dedicated Limitations section is present. The paper states that the nonsegmentation method cannot distinguish new/never-seen sentences even when they contain the same word elements in a different order, and it introduces recognition time latency; the segmentation method has compromised recognition accuracy.
  • The paper states the demonstration is limited to 50 words and 20 sentences from an American Sign Language phrase book and says the database can be broadened; new-sentence recognition reaches 86.67%, not perfect, and the single classifier is lower at 60%.
  • Evident from the text: the VR communication demonstration is a greeting scenario between two avatars under an identical LAN, so broader real-world deployment and larger user populations are not tested in the paper.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-021-25637-w

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 3 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The mean correlation coefficient for 'Get' compared with its own database is 0.53, indicating high similarity.

    “the mean correlation coefficient 0.53 > 0.3 (threshold value for high similarity) by extracting the average value of 100 data samples and correlating it with the new input, which means the similarity is high between different ‘Get’ data samples.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The nonsegmentation deep learning model achieves a recognition accuracy of 95% for 17 sentences.

    “Therefore, a high sentence recognition accuracy of 95% is obtained, as indicated in Fig. 3j.”
    • Also stated in: § Word and sentence recognition upon nonsegmentation method · plain
      “j Confusion map of recognizing 17 sentences.”
    Quote in the paperSubstantial claim
  • Findingown data

    The system successfully realizes the recognition of 50 words and 20 sentences.

    “The system successfully realizes the recognition of 50 words and 20 sentences.”
    Quote in the paperSubstantial claim
  • Findingown data

    The segmentation method with hierarchy classifier achieves an average correct rate of 85.58% for reconstructing and recognizing sentences in the dataset.

    “the sentence can be inversely reconstructed and recognition at an average correct rate of 85.58% within hierarchy classifier.”
    • Also stated in: § Introduction · plain
      “the whole sentence’s information can be reconstructed and recognized with an accuracy of 85.58% upon established correlation between word units and sentences.”
    Quote in the paperSubstantial claim
  • Findingown data

    The single classifier achieves an average correct rate of 79.41% for reconstructing and recognizing sentences.

    “showing an insufficient average correct rate of 79.41%.”
    Quote in the paperSubstantial claim
  • May combine two claims. 23
  • Classified as probably not a substantial claim. 6
  • May combine two claims.

    Findingown data

    The hierarchy classifier improves sentence recognition average correct rate to 85.58% with reduced false area.

    “Eventually, it positively contributes to the sentence recognition, which is indicated by the improved average correct rate of 85.58% with a reduced false area in Fig. 4i.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The voltage output increases when sensor area, force, bending degree, and bending speed increase.

    “the voltage output increases when sensor area, force, bending degree, and bending speed increase.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    The mean correlation coefficient for 'Must' compared with 'Get' database is 0.37, indicating high signal similarity between 'Get' and 'Must'.

    “The mean correlation coefficient of 0.37 is larger than 0.3, indicating the high signal similarity between ‘Get’ and ‘Must’ as well”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Hand motions fall into four categories: finger bending (56%), wrist motion (18%), touch with fingertips (16%), interaction with palm (10%).

    “hand motions fall into four categories, including finger bending (56%), wrist motion (18%), touch with fingertips (16%), and interaction with palm (10%).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingreasoning

    The recognition latency is significantly reduced due to small-distance sliding (50 data points per sliding) as the recognition process will be triggered with each sliding.

    “the recognition latency is significantly reduced due to small-distance sliding (50 data points per sliding) as the recognition process will be triggered with each sliding.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The 3rd and 5-10th samples of sentence Y01 are wrongly classified with a poor correct rate of 30%.

    “the 3rd and 5–10th samples of sentence Y01 are wrongly classified with a poor correct rate of 30%.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    The t-SNE visualization shows poor feature clustering before CNN and desirable classification after CNN with clear boundaries among 50 classes.

    “It shows the performance of feature clustering is poor with category overlaps before going through the CNN network. After undergoing the feature extraction and classification of CNN, the visualization result realized by t-SNE in Fig. 3f indicates a desirable classification performance of the developed CNN model with clear boundaries among these 50 classes with less overlap.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingreasoning

    The deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences.

    “Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The data analysis shows high similarities of word-to-word and sentence-to-sentence signals.

    “the data analysis proclaims the high similarities of word-to-word and sentence-to-sentence signals.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingreasoning

    The system projects sign language recognition results into virtual space and translates them into text and audio, allowing remote and bidirectional communication between signers and non-signers.

    “the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingnone given

    The system enables separate and independent recognition of words and sentences with high accuracies of 91.3% and 95% within nonsegmentation frame.

    “The system enables the separate and independent recognition of words (i.e., single gestures) and sentences (i.e., continuous multiple gestures) with high accuracies of 91.3% and 95% within nonsegmentation frame.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The hierarchy classifier achieves precise identification of 5 samples of New1, 3 samples of New2, and 5 samples of New3.

    “In comparison, the hierarchy classifier achieves precise identification of 5 samples of New1, 3 samples of New2, and 5 samples of New3.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The hierarchy classifier achieves an average correct rate of 86.67% for new sentence translation, superior to 60% of the single classifier.

    “In other words, the hierarchy classifier achieves an average correct rate of 86.67% (see Supplementary Fig. 6b), superior to 60% of the single classifier.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The single classifier correctly predicts 2 samples of New1, 3 samples of New2, 4 samples of New3, average correct rate 60%.

    “Moving forward to the translation (i.e., recognition) stage (see Supplementary Fig. 6a), the single classifier correctly predicts 2 samples of New1, 3 samples of New2, 4 samples of New3 (average correct rate of 60%).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Both classifiers render label series prediction for the total 15 inputs of the 3 new sentences, although the CNN model never sees or learns the true label series before.

    “As provided in Supplementary Table 2, both classifiers render the label series prediction for the total 15 inputs of these 3 new sentences, although the CNN model never sees or learns the true label series before.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The speech-impaired user Lily performs sign language 'How are you?', which is recognized and converted to text and audio by the deep learning model.

    “In the first step Fig. 5b(i), the speech-impaired user Lily performs sign language ‘How are you?’, which is recognized and converted to text and audio by the deep learning model.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The nonsegmentation approach possesses better recognition accuracy for words or sentences but has apparent recognition time latency and incapability of new sentence recognition.

    “Overall, the nonsegmentation approach possesses better performance in the aspect of recognition accuracy either for words or sentences but with the shortcoming of apparent recognition time latency and incapability of new sentence recognition.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The VR interface for two-way remote communication, linked with the AI front end for sign language recognition, shows a potential prototype of a future intelligent sign language recognition and communication system.

    “The VR interface for two-way remote communication, linked with the AI front end for sign language recognition, shows a potential prototype of a future intelligent sign language recognition and communication system.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    The long-length signal of sentences provides more distinguishable features than single-word signals, contributing to high accuracy.

    “The long-length signal of sentence provides more distinguishable features than the single-word signal. Therefore, a high sentence recognition accuracy of 95% is obtained”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    The hierarchy classifier improves both the accuracy of recognizing word elements and that of identifying ever-seen/trained sentences.

    “Overall, the hierarchy classifier improves both the accuracy of recognizing word elements and that of identifying ever-seen/trained sentences.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationnone given

    The single classifier may not be effective enough when reconstructing and recognizing sentences that existed in the dataset, which may hinder the identification of new/never-seen sentences.

    “the single classifier may not be effective enough when reconstructing and recognizing sentences that existed in the dataset, which may hinder the identification of new/never-seen sentences.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The VR communication interface linked with AI allows interaction of the speech/hearing disordered and healthy people closely and even remotely, providing a promising platform for immense interactions between two population groups.

    “The VR communication interface linked with AI allows interaction of the speech/hearing disordered and healthy people closely and even remotely, providing a promising platform for the immense interactions between two population groups.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Most current sign language translation systems simply display recognition results on visualized interfaces such as mobile phones and portable displays.
    • Rests on an assumption: Most works do not allow interaction between signers and nonsigners.
    • Rests on an assumption: There is an acute demand for effective two-way communication between signers and nonsigners.
    • Rests on an assumption: The emerging VR interface is an ideal platform for advanced sign language interpretation, visualization, and communication interface.
  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The embedded VR interface can act as the bridge of user terminals, transmitting messages back and forth, configuring a closed-loop communication system with gloves and AI component.

    “Finally, the embedded VR interface can act as the bridge of user terminals, transmitting messages back and forth, configuring a closed-loop communication system with gloves and AI component.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Most current sign language translation systems simply display recognition results on visualized interfaces such as mobile phones and portable displays.
    • Rests on an assumption: Most works do not allow interaction between signers and nonsigners.
    • Rests on an assumption: There is an acute demand for effective two-way communication between signers and nonsigners.
    • Rests on an assumption: The emerging VR interface is an ideal platform for advanced sign language interpretation, visualization, and communication interface.
  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The intelligent system provides a reliable and appealing platform for seamless and immense interactions between nonsigners and signers either remotely or proximately.

    “Besides, the intelligent system consist of smart gloves, AI-assisted recognition, and TCP/IP enabled server-client LAN communication in virtual space, providing a reliable and appealing platform for the seamless and immense interactions between nonsigners and signers either remotely or proximately.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Most current sign language translation systems simply display recognition results on visualized interfaces such as mobile phones and portable displays.
    • Rests on an assumption: Most works do not allow interaction between signers and nonsigners.
    • Rests on an assumption: There is an acute demand for effective two-way communication between signers and nonsigners.
    • Rests on an assumption: The emerging VR interface is an ideal platform for advanced sign language interpretation, visualization, and communication interface.
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The authors affirm that the participant provide informed consent for publication of images and videos.

    “The authors affirm that the participant provide informed consent for publication of the images in Figs. 2a, 5b, Supplementary Figs. 2a, 8a, and the videos in Supplementary Movies 1, 2, 3.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    Few works have employed AI-enabled advanced data analytics to achieve highly precise sign language translation, but they are still limited to classification of discrete words that cannot meet practical communication requirements.

    “Although few works have employed AI-enabled advanced data analytics to achieve highly precise sign language translation, they are still limited to the classification of discrete words that cannot meet the requirement of practical communication in the signer’s daily lives.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Supplementary Table 3

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The small-distance sliding induced recognition brings the potential of real-time recognition or translation of sentences.

    “Besides, the small-distance sliding induced recognition brings the potential of the real-time recognition or translation of sentences.”
    Quote in the paperNot a substantial claim
27 of 27
  • The paper relies on this without support.

    New/never-seen sentences created by new-order word element recombination are valid sign language sentences that a signer would use.

    “new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%.”
  • The paper relies on this without support.

    The 50 gestures and 20 sentences selected are representative of daily sign language communication.

    “we select 50 gestures that are daily used in the signer’s life as demonstration.”
    “20 daily sentences are sorted out for subsequent study.”
  • The paper relies on this without support.

    Sign language is not as prevalent as spoken language and is difficult for nonsigners to understand without prior learning.

    “sign language is not as prevalent as the speaking language and difficult for nonsigners to understand without prior learning.”
  • The paper relies on this without support.

    Most current sign language translation systems simply display recognition results on visualized interfaces such as mobile phones and portable displays.

    “Most current sign language translation systems simply display the recognition result on visualized interfaces such as mobile phones and portable displays.”
  • The paper relies on this without support.

    Most works do not allow interaction between signers and nonsigners.

    “since most of works do not allow the interaction between signers and nonsigners.”
  • The paper relies on this without support.

    There is an acute demand for effective two-way communication between signers and nonsigners.

    “However, the acute demand of effective two-way communication between the signers and nonsigners is neglected”
  • The paper relies on this without support.

    The emerging VR interface is an ideal platform for advanced sign language interpretation, visualization, and communication interface.

    “which would be an ideal platform for an advanced sign language interpretation, visualization, and communication interface.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Nonsegmentation sentence accuracy is attributed to both 20 and 17 sentences

    Minor

    The paper attributes its 95% nonsegmentation accuracy to 20 sentences, but the results section says that accuracy was obtained for 17 sentences. This changes the scope of a headline recognition result.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Wen, F., Zhang, Z., He, T., & Lee, C.
Affiliations
National University of Singapore (SG)
Corresponding author
Lee, Chengkuo
Venue
Nature Communications
Published
2021-09-10
Volume
12
Issue
1
Pages
5378-5378
ISSN
2041-1723 (ISSN-L)
OpenAlex
W3197059061
Cited by
494
References
73
Citations by year
2026: 58 · 2025: 121 · 2024: 121 · 2023: 107 · 2022: 78 · 2021: 9
FWCI
48.31
Type
journal article
Language
en
Open access
Locations
  • Nature Communications (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • National University of Singapore (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Key Research and Development Program of China (2019YFB2004800, R‐2020‐S‐002)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Engineering › Biomedical Engineering › Advanced Sensor and Energy Harvesting Materials
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
sign language recognition, sentence recognition, virtual reality communication, two-way communication, hearing loss, word segmentation, gesture sensing, deep learning
MeSH terms
Virtual Reality, Wearable Electronic Devices, Deep Learning, Communication Devices for People with Disabilities, Communication Aids for Disabled, Deafness, Gestures, Humans, Sign Language

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review