A Deformable Interface for Human Touch Recognition Using Stretchable Carbon Nanotube Dielectric Elastomer Sensors and Deep Neural Networks
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
This article presents a machine learning approach to map outputs from an embedded array of sensors distributed throughout a deformable body to continuous and discrete virtual states, and its application to interpret human touch in soft interfaces. We integrate stretchable capacitors into a rubber membrane, and use a passive addressing scheme to probe sensor arrays in real time. To process the signals from this array, we feed capacitor measurements into convolutional neural networks that classify and localize touch events on the interface. We implement this concept with a device called OrbTouch. To modularize the system, we use a supervised learning approach wherein a user defines a set of touch inputs and trains the interface by giving it examples; we demonstrate this by using OrbTouch to play the popular game Tetris. Our regression model localizes touches with mean test error of 0.09 mm, whereas our classifier recognizes five gestures with a mean test error of 1.2%. In a separate demonstration, we show that OrbTouch can discriminate between 10 different users with a mean test error of 2.4%. At test time, we feed the outputs of these models into a debouncing algorithm to provide a nearly error-free experience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“We initially considered other approaches to capture temporal information, including long-short term memory (LSTM) units, as well as CNN feature extractors in combination with LSTMs (Ordóñez and Roggen 2016), however we found that gestures, and even short sequences of gestures, occur over relatively short time horizons.”
MethodWearable Electronics and Smart Textiles: A Critical Review
Stoppa, M., & Chiolerio, A. (2014) · Sensors · doi:10.3390/s140711957 · Cited by 2,107
Stretchable, Transparent, Ionic Conductors
Keplinger, C., Sun, J.-Y., Foo, C. C., Rothemund, P., Whitesides, G. M., & Suo, Z. (2013) · Science · doi:10.1126/science.1240228 · Cited by 1,754
Stretchable silicon nanoribbon electronics for skin prosthesis
Kim, J., Lee, M., Shim, H. J., Ghaffari, R., Cho, H. R., Son, D., Jung, Y. H., Soh, M., Choi, C., Jung, S., et al. (2014) · Nature Communications · doi:10.1038/ncomms6747 · Cited by 1,435
Highly stretchable electroluminescent skin for optical signaling and tactile sensing
Larson, C., Peele, B., Li, S., Robinson, S., Totaro, M., Beccai, L., Mazzolai, B., & Shepherd, R. (2016) · Science · doi:10.1126/science.aac5082 · Cited by 1,393 · cited 1×
“bber with internal uidic circuits. Additionally, recent work in stretchable electronics has introFirst author duced materials that enable soft devices to sense deformation and change their appearance [12]. Conductive leads and electrodes, for example, can be made stretchable though the use of carbon nanotube (CNT) lms…”
Self-citation
Research works printed but not linked: 34
- 10.1038/nature14543 · DOI · Paper page
- Follmer S, , Leithinger D, , Olwal A, et al. Jamming user interfaces: Programmable particle stiffness and sensing for malleable and shape-changing devices. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. Cambridge, MA: ACM, 2012, pp. 519–528.
- 10.1073/pnas.1003250107 · DOI · Paper page
- 10.1109/TVCG.2016.2525788 · DOI
- 10.1109/TOH.2015.2423492 · DOI · Paper page
- Lee S-S, , Kim S, , Jin B, et al. How users manipulate deformable displays as input devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Cambridge, MA: ACM, 2010, pp. 1647–1656.
- Rasmussen MK, , Pedersen EW, , Petersen MG, et al. Shape-changing interfaces: A review of the design space and open research questions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Atlanta, GA: ACM, 2012, pp. 735–744.
- Ogata M, , Sugiura Y, , Makino Y, et al. Senskin: Adapting skin as a soft interface. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. St. Andrews, Scotland: ACM, 2013, pp. 539–544.
- 10.1145/2556288.2557239 · DOI
- Pai. DK, VanDerLoo EW, Sadhukhan S, et al. The tango: A tangible tangoreceptive whole-hand human interface. In Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2005. World Haptics 2005. First Joint, IEEE, Pisa, Italy, 2005, pp. 141–147.
- Han S, , Park J. Grip-ball: A spherical multi-touch interface for interacting with virtual worlds. In Consumer Electronics (ICCE), 2013 IEEE International Conference. Las Vegas, NV: IEEE, 2013, pp. 600–601.
- 10.1145/1753846.1753864 · doi:10.1145/1753846.1753864
- 10.1109/VR.2013.6549433 · DOI
- 10.1145/1518701.1518749 · DOI · Paper page
- Steimle J, , Jordt A, , Maes P. Flexpad: Highly flexible bending interactions for projected handheld displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Paris, France: ACM, 2013, pp. 237–246.
- 10.1126/science.1182383 · DOI · Paper page
- 10.1038/nn.2973 · DOI · Paper page
- 10.1088/0960-1317/20/12/125029 · DOI · Paper page
- 10.1126/science.1121401 · DOI · Paper page
- 10.1038/nnano.2011.36 · DOI · Paper page
- 10.1038/nnano.2011.184 · DOI · Paper page
- Shepherd R Peele B Murray BM et al. Stretchable transducers for kinesthetic interactions in virtual reality. In ACM SIGGRAPH 2017 Emerging Technologies. ACM 2017 p. 21. · DOI
- 10.1109/LRA.2018.2799743 · DOI · Paper page
- 10.1021/acsnano.5b01613 · DOI · Paper page
- He. K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, 2016, pp. 770–778.
- Mnih V Kavukcuoglu K Silver D et al. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013. · arXiv
- Mikolov T Chen K Corrado G et al. Efficient estimation of word representations in vector space. arxiv: 1301.3781 2013. · arXiv
- Bengio. S, Vinyals O, Jaitly N, et al. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal, Canada, 2015, pp. 1171–1179.
- 10.1098/rsta.1952.0013 · doi:10.1098/rsta.1952.0013
- Kingma D Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014. · arXiv
- Abadi M Agarwal A Barham P et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv: 1603.04467 2016. · arXiv
- Nesterov Y. A method of solving a convex programming problem with convergence rate o (1/k2). Soviet Math Doklady, 1983; 27:372–376.
- 10.1109/TRO.2011.2171093 · DOI · Paper page
- 10.1002/smll.200600061 · DOI · Paper page
Cited by
Electronic skins and machine learning for intelligent soft robots
Shih, B., Shah, D., Li, J., Thuruthel, T. G., Park, Y.-L., Iida, F., Bao, Z., Kramer-Bottiglio, R., & Tolley, M. T. (2020) · Science Robotics · doi:10.1126/scirobotics.aaz9239 · Cited by 715
“(60) used convolutional neural networks to learn deformations on a sensor array that can interpret human touch in soft interfaces.”
BackgroundSoft magnetic skin for super-resolution tactile sensing with force self-decoupling
Yan, Y., Hu, Z., Yang, Z., Yuan, W., Song, C., Pan, J., & Shen, Y. (2021) · Science Robotics · doi:10.1126/scirobotics.abc8801 · Cited by 556
Review of machine learning methods in soft robotics
Kim, D., Kim, S.-H., Kim, T., Kang, B. B., Lee, M., Park, W., Ku, S., Kim, D., Kwon, J., Lee, H., et al. (2021) · PLOS ONE · doi:10.1371/journal.pone.0246102 · Cited by 256
Machine Learning for Soft Robotic Sensing and Control
Chin, K., Hellebrekers, T., & Majidi, C. (2020) · Advanced Intelligent Systems · doi:10.1002/aisy.201900171 · Cited by 215
Soft optoelectronic sensory foams with proprioception
Van Meerbeek, I. M., De Sa, C. M., & Shepherd, R. F. (2018) · Science Robotics · doi:10.1126/scirobotics.aau2489 · Cited by 203
“Such an approach has been used on sensor systems to fabricate devices such as a gesture recognition device, a pressure sensor, and a robotic skin ( 24 – 27 ).”
MethodSelf-citation
Metadata
- Authors
- Larson, C., Spjut, J., Knepper, R., & Shepherd, R.
- Venue
- Soft Robotics
- Publisher
- SAGE Publications
- Published
- 2019-10-01
- Volume
- 6
- Issue
- 5
- Pages
- 611-620
- ISSN
- 2169-5172 (ISSN-L)
- 2169-5180
- OpenAlex
- W2795512733
- Cited by
- 51
- References
- 7
- FWCI
- 1.99
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.