A Multitask Grocery Assist System for the Visually Impaired: Smart glasses, gloves, and shopping carts provide auditory and tactile feedback
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
According to the World Health Organization, "285 million people are estimated to be visually impaired worldwide" [1]. Several technologies such as automatic text readers, Braille note makers, and navigation assistance canes have been developed to assist the visually impaired. Concurrent advances in computer vision and hardware technologies provide opportunities for a visual-assistance system that can be used in multiple contexts. As part of the Visual Cortex on Silicon program, we have been developing interfaces, algorithms, and hardware platforms to assist the visually impaired with a focus on grocery shopping. This article describes the various features that we have incorporated into this visual-assistance system so that it can be used in multiple contexts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“PVI need to take many good quality photos that is hard for them”
From Making Shopping Easy for People with Visual Impairment Using Mobile Assistive Technologies (2019) · cites this paper as [83]
In the citing paper
“Inconsistent performance in real-world environments because of factors such as motion blur”
From Making Shopping Easy for People with Visual Impairment Using Mobile Assistive Technologies (2019) · cites this paper as [83]
In the citing paper
“Most of them can be installed easily on smartphones”
From Making Shopping Easy for People with Visual Impairment Using Mobile Assistive Technologies (2019) · cites this paper as [83]
In the citing paper
“Cost-effective as they need little or no infrastructure”
From Making Shopping Easy for People with Visual Impairment Using Mobile Assistive Technologies (2019) · cites this paper as [83]
In the citing paper
“Use extensive computation power”
From Making Shopping Easy for People with Visual Impairment Using Mobile Assistive Technologies (2019) · cites this paper as [83]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 1×
“Another important insight, recently unearthed by [5], is that the accuracies of CNNs can saturate after a few million images of training data.”
BackgroundLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip
Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, P., Nam, G.-J., et al. (2015) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2015.2474396 · Cited by 1,687
The neural basis of multisensory integration in the midbrain: Its organization and maturation
Stein, B. E., Stanford, T. R., & Rowland, B. A. (2009) · Hearing Research · doi:10.1016/j.heares.2009.03.012 · Cited by 209
RedEye: analog ConvNet image sensor architecture for continuous mobile vision
LiKamWa, R., Hou, Y., Gao, J., Polansky, M., & Zhong, L. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001164 · Cited by 189
CAPI: A Coherent Accelerator Processor Interface
Stuecheli, J., Blaner, B., Johns, C. R., & Siegel, M. S. (2015) · IBM Journal of Research and Development · doi:10.1147/jrd.2014.2380198 · Cited by 164
RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision
LiKamWa, R., Hou, Y., Gao, Y., Polansky, M., & Zhong, L. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.31 · Cited by 61
An Information Theoretic Model of Saliency and Visual Search
Bruce, N. D. B., & Tsotsos, J. K. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-77343-6_11 · Cited by 36
Enabling New Computation Paradigms with HyperFET - An Emerging Device
Tsai, W.-Y., Li, X., Jerry, M., Xie, B., Shukla, N., Liu, H., Chandramoorthy, N., Cotter, M. J., Raychowdhury, A., Chiarulli, D. M., et al. (2016) · IEEE Transactions on Multi-Scale Computing Systems · doi:10.1109/tmscs.2016.2519022 · Cited by 34
Self-citation
Research works printed but not linked: 3
- 0, Robust Multi-Person Tracking From Mobile Platforms
- 0, INRIA Person Dataset
- 2014, Visual Impairment and Blindness
Cited by
Deep Learning for IoT Big Data and Streaming Analytics: A Survey
Mohammadi, M., Al-Fuqaha, A., Sorour, S., & Guizani, M. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2844341 · Cited by 1,471
Deep learning-based object detection in augmented reality: A systematic review
Ghasemi, Y., Jeong, H., Choi, S. H., Park, K.-B., & Lee, J. Y. (2022) · Computers in Industry · doi:10.1016/j.compind.2022.103661 · Cited by 188
A Review on Deep Learning Techniques for IoT Data
Lakshmanna, K., Kaluri, R., Gundluru, N., Alzamil, Z. S., Rajput, D. S., Khan, A. A., Haq, M. A., & Alhussen, A. (2022) · Electronics · doi:10.3390/electronics11101604 · Cited by 154
Santra, B., & Mukherjee, D. P. (2019) · Image and Vision Computing · doi:10.1016/j.imavis.2019.03.005 · Cited by 102
Budrionis, A., Plikynas, D., Daniušis, P., & Indriulionis, A. (2020) · Assistive Technology · doi:10.1080/10400435.2020.1743381 · Cited by 117
“Alternative mechanisms in the form of vibrating bracelets (Scheggi et al., 2014), gloves with micro-motors (Advani et al., 2017; Poggi & Mattoccia, 2016), vibrating belts (H.-C. Wang et al., 2017) and other means of tactile feedback (Peiris et al., 2016) have been reported, demonstrating the…”
BackgroundChang, W., Chen, L.-B., Sie, C.-Y., & Yang, C.-H. (2020) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2020.3037065 · Cited by 80
“Advani et al. [15] developed a smart glasses/gloves-based multitask grocery assistive system for visually impaired consumers that was capable of product recognition, person detection, and food recognition.”
BackgroundA fog-computing architecture for preventive healthcare and assisted living in smart ambients
Cerina, L., Notargiacomo, S., Paccanit, M. G., & Santambrogio, M. D. (2017) · IEEE International Forum on Research and Technology for Society and Industry (RTSI) · doi:10.1109/rtsi.2017.8065939 · Cited by 70
“An interesting approach is described in [13], aimed at the assistance of visually impaired subjects during grocery shopping: the system uses Fog devices to locate the user inside the aisles and to offload the video processing necessary to assist the subjects.”
BackgroundWolniak, R., Stecuła, K., & Aydın, B. (2024) · Foods · doi:10.3390/foods13182948 · Cited by 77
“Advani et al. [155] detail the development of the Third-Eye prototype, focusing on grocery shopping.”
BackgroundSensing and Navigation of Wearable Assistance Cognitive Systems for the Visually Impaired
Li, G., Xu, J., Li, Z., Chen, C., & Kan, Z. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3146828 · Cited by 59
“According to the World Health Organization, 285 million people are living with visual impairment, which makes the blind and visually impaired (BVI) difficult to travel freely [1].”
BackgroundMaking Shopping Easy for People with Visual Impairment Using Mobile Assistive Technologies
Elgendy, M., Sik-Lanyi, C., & Kelemen, A. (2019) · Applied Sciences · doi:10.3390/app9061061 · Cited by 48
“used smartglasses to accurately locate and identify objects using CV techniques [41,83].”
MethodDargazany, A. R., Stegagno, P., & Mankodiya, K. (2018) · Mobile Information Systems · doi:10.1155/2018/8125126 · Cited by 57
“[4] build a multitask AI visual-assistance system for assisting visually impaired people in grocery shopping using smart glass, smart glove, and shopping carts for providing auditory and tactile feedback.”
BackgroundInfluentialBoldú, R., Dancu, A., Matthies, D. J. C., Buddhika, T., Siriwardhana, S., & Nanayakkara, S. C. (2018) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3264904 · Cited by 48
“al [51] created a system composed of smart glasses with an attached camera, that can guide the user towards the desired product through commands (left, right, forward, and back).”
BackgroundA Deep Learning Framework for Grocery Product Detection and Recognition
Selvam, P., & Sundar, K. J. A. (2022) · Food Analytical Methods · doi:10.1007/s12161-022-02384-2 · Cited by 38
“Karlinsky et al. (Karlinsky et al. 2017) and Zientara et al. (Advani et al. 2017) calculated a homography matrix to identify the grocery products in shelf images by matching SURF key points of product images with corresponding rack images.”
BackgroundMethodLee, C.-W., Chondro, P., Ruan, S.-J., Christen, O., & Naroska, E. (2018) · IEEE Consumer Electronics Magazine · doi:10.1109/mce.2018.2797741 · Cited by 25
“Several studies in [10]–[13] have developed positioning systems based on the matched features on corresponding frames.”
BackgroundDisabled People’s Vulnerability in the European Single Market: The Case of Consumer Information
Eskytė, I. (2019) · Journal of Consumer Policy · doi:10.1007/s10603-019-09422-3 · Cited by 31
Product identification in retail stores by combining faster r-cnn and recurrent neural network
Ghosh, R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15633-1 · Cited by 16
“Zientara et al. [6] presented a smart glass based product detection system that locates the products in the rack images through deriving the saliency map of the rack images.”
Dourado, A. M. B., & Pedrino, E. C. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103046 · Cited by 18
An Overview of Machine Learning and 5G for People with Disabilities
Domingo, M. C. (2021) · Sensors · doi:10.3390/s21227572 · Cited by 18
“Smart glasses (with a camera and a headset) and haptic gloves (with vibration motors and cameras, as well) analyze images to accurately locate and identify objects; they use computer vision technology to transfer real-time visual information into auditory information to assist the visually impaired do their daily acti…”
BackgroundConglomeration of deep neural network and quantum learning for object detection: Status quo review
Sinha, P. K., & Marimuthu, R. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111480 · Cited by 16
Smart Shopping Cart for Visually Impaired Individuals
Shete, V., Shaikh, A., Sawant, R., Singh, C., Ponmalar, P. S., & Harikrishnan, R. (2022) · Review of Computer Engineering Research · doi:10.18488/76.v9i2.3083 · Cited by 12
“Not only must the interface be user-friendly, but it must also be adaptable to various contexts [5].”
BackgroundInfluentialChen, L.-B., Pai, W.-Y., Chen, W.-H., & Huang, X.-R. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3484576 · Cited by 10
“In recent years, many related works [3], [4], [5], [6], [7], [8], [9], [10], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30] on walking guide assistance for visually impaired people have been presented.”
A Systematic Review on Product Recognition for Aiding Visually Impaired People
Machado, A., Veras, R. D. M. S., Aires, K. R. T., & Neto, L. D. S. B. (2021) · IEEE Latin America Transactions · doi:10.1109/tla.2021.9448542 · Cited by 12
“Em Advani et al. [48], o protótipo Third-Eye foi apresen-tado para auxiliar pessoas com deficiências visuais a realizar compras de supermercado, por meio da combinação de acele-radores de hardwares, algoritmos de visão computacional, com uma câmera cada, um par de óculos e uma luva wearable .”
BackgroundDeep learning and metaheuristics application in internet of things: A literature review
Khelili, M. A., Slatnia, S., Kazar, O., Merizig, A., & Mirjalili, S. (2023) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2023.104792 · Cited by 10
Finger Bending Sensing Based on Series-Connected Fiber Bragg Gratings
Lin, Q., Yao, K., Zhao, N., Jiao, Y., Wang, Z., Tian, B., Zhao, L., Peng, G., & Jiang, Z. (2022) · Materials · doi:10.3390/ma15103472 · Cited by 7
“[2] used the smart glove and glasses concept to develop a system to assist the visually impaired with a focus on grocery shopping.”
MethodSmart Store Assistor for Visually Impaired
Devipriya, D., Sri, V. S., & Mamatha, I. (2018) · International Conference on Advances in Computing, Communications and Informatics (ICACCI) · doi:10.1109/icacci.2018.8554676 · Cited by 7
Indoor Navigation Aid Systems for the Blind and Visually Impaired Based on Depth Sensors
Merchán, F., Poveda, M., Cáceres-Hernández, D., & Sánchez-Galán, J. E. (2020) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-7998-6522-3.ch007 · Cited by 5
A Large Vision-Language Model based Environment Perception System for Visually Impaired People
Chen, Z., Liu, Z., Wang, K., Wang, K., & Lian, S. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801813 · Cited by 9
“Advani et al. [27] proposed to use of smart glasses and data glove to retrieve the product information in grocery shopping.”
Scalable Object Detection in Mixed Reality Using Incremental Re-Training and One-Shot 3D Annotation
Taheritajar, A., Benson, J., Gibson, A., Wilburn, B., Zhao, J. L., & Orlosky, J. (2025) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar67309.2025.00040 · Cited by 1
Jiang, Y., Zhang, Z., Xu, J., Zheng, Q., Guo, Q., Wang, Q., Wang, Q., & Liu, G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790604 · Cited by 3
“Meanwhile, research on assistive technologies for people with visual impairments has primarily focused on individual functional compensation [11, 77], such as scene recognition, navigation, or emotion detection [2, 22, 37, 48, 58].”
Blind Lane Detection and Following for Assistive Navigation of Vision Impaired People
Zhicheng, H., Gu, J. J., & Feng, Y. (2023) · International Conference on Advanced Robotics and Mechatronics (ICARM) · doi:10.1109/icarm58088.2023.10218843 · Cited by 2
“Similarly, the issue of mobility for individuals with visual impairments has also been receiving increasing attention [2].”
BackgroundA Survey on Identification of Grocery Store Items Using Deep Learning in Retail Store
Savani, N., & Lunagaria, M. (2022) · Smart innovation, systems and technologies · doi:10.1007/978-981-19-3571-8_52
ARPOV: Expanding Visualization of Object Detection in AR with Panoramic Mosaic Stitching
McGowan, E., Brewer, E., & Silva, C. T. (2024) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi62404.2024.10716346
“2) A novel technique for improving panoramic mosaic quality by clustering the frames’ corresponding homography matrices by their singular values and filtering outliers.”
Method
Metadata
- Authors
- Advani, S., Zientara, P., Shukla, N., Okafor, I., Irick, K., Sampson, J., Datta, S., & Narayanan, V.
- Affiliations
- Pennsylvania State University (US)
- University of Notre Dame (US)
- Venue
- IEEE Consumer Electronics Magazine
- Published
- 2017-01
- Volume
- 6
- Issue
- 1
- Pages
- 73-81
- ISSN
- 2162-2248 (ISSN-L)
- 2162-2256
- OpenAlex
- W2561304097
- Cited by
- 32
- References
- 14
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 5 · 2023: 4 · 2022: 7 · 2021: 2 · 2020: 3 · 2019: 3 · 2018: 5 · 2017: 1
- FWCI
- 9.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Consumer Electronics Magazine (publishedVersion)
- Funders
- National Science Foundation
- University of Notre Dame
- Pennsylvania State University
- University of South Florida
- Topics
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- visually impaired people, smart glasses, smart glove, shopping cart, auditory feedback, tactile feedback, assistive technology, multimodal interfaces
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.