CoChat
Sign Up
Cited by 26Advanced Neural Network ApplicationsVisual Attention and Saliency DetectionGaze Tracking and Assistive Technology

Human gaze-aware attentive object detection for ambient intelligence

Open access
Cho, Dae-Yong, Kang, Min-Koo·2021·Engineering Applications of Artificial Intelligence
DOI 10.1016/j.engappai.2021.104471
Open this paper in CoChat — free accountOpen PDF

Abstract

Understanding human behavior and the surrounding environment is essential for realizing ambient intelligence (AmI), for which eye gaze and object information are reliable cues. In this study, the authors propose a novel human gaze-aware attentive object detection framework as an elemental technology for AmI. The proposed framework detects users’ attentive objects and shows more precise and robust performance against object-scale variations. A novel Adaptive-3D-Region-of-Interest (Ada-3D-RoI) scheme is designed as a front-end module, and scalable detection network structures are proposed to maximize cost-efficiency. The experiments show that the detection rate is improved up to 97.6% on small objects (14.1% on average), and it is selectively tunable with a tradeoff between accuracy and computational complexity. In addition, the qualitative results demonstrate that the proposed framework detects a user’s single object-of-interest only, even when the target object is occluded or extremely small. Complementary matters for follow-up study are presented as suggestions to extend the results of the proposed framework to further practical AmI applications. This study will help develop advanced AmI applications that demand a higher-level understanding of scene context and human behavior such as human–robot symbiosis, remote-/autonomous control, and augmented/mixed reality.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.engappai.2021.104471

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI algorithms can enable robots and AGVs to make real-time decisions based on sensor data, allowing them to adapt to changing conditions in a manufacturing environment.

    “Real-time decision making: AI algorithms can enable robots and AGVs to make real-time decisions based on sensor data, allowing them to adapt to changing conditions in a manufacturing environment [77] .”

    From Artificial intelligence, machine learning and deep learning in advanced robotics, a review (2023) · cites this paper as [77]

    The Introduction discusses ambient intelligence recognizing users and their surroundings, and the Conclusion suggests human-robot symbiosis and autonomous control as possible applications. Neither passage addresses AGVs making real-time decisions from sensor data or adapting to changing manufacturing conditions. Searches for AGVs, manufacturing, and real-time decisions found no passage supporting that claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L., 2018. Bottom-up and top-down attention for image captioning and visual question answering. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition.
  2. Barz, M., Sonntag, D., 2016. Gaze-guided object classification using deep neural networks for attention-based computing. In: Proc. 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing.
  3. Beugher, S.D., Ichiche, Y., Brone, G., Goedeme, T., Automatic analysis of eye-tracking data using object detection algorithms. In: Proc. 2012 ACM Conference on Ubiquitous Computing.
  4. Brais, Bosquet, Manuel, Mucientes, Brea, Victor M., 2020. STDnet: Exploiting high resolution feature maps for small object detection. Eng. Appl. Artif. Intell. 91, https://doi.org/10.1016/j.engappai.2020.103615. · DOI
  5. Canny, J., 1986. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. PAMI 8 (6), 679-698.
  6. Cheng, M.M., Zhang, Z., Lin, W.Y., Torr, P., 2014. BING: binarized normed gradients for objectness estimation at 300fps. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition.
  7. Gui, J., Liu, Y., Qiu, Q., Huang, J., Liu, C., Cao, Z., Y, Chen., 2019. A novel robotic guidance system with eye gaze tracking control for needle based interventions. IEEE Trans. Cognit. Dev. Syst. Early Access 1, https://doi.org/10.1109/TCDS.2019. 2959071. · DOI
  8. Hariharan, B., Arbelaez, P., Girshick, R., Malik, J., 2016. Object instance segmentation and fine-grained localization using hypercolumns. IEEE Trans. Pattern Anal. Mach. Intell. 39 (4), 627-639. https://doi.org/10.1109/TPAMI.2016.2578328. · DOI · Paper page
  9. Kassner, M., Patera, W., Bulling, A., 2014. Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In: Proc. 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. https://doi.org/10.1145/2638728.2641695. · DOI
  10. Kim, J.H., Choi, S.J., Jeong, J.W., 2019. Watch & do: a smart IoT interaction system with object detection and gaze estimation. IEEE Trans. Consum. Electron. 65 (2), 195-204. https://doi.org/10.1109/TCE.2019.2897758. · DOI
  11. Kuo, T.-S., Shih, J.-T, Chung, S.-L, Chen, H.H., 2018. Depth from Gaze. In: IEEE International Conference on Image Processing. ICIP, pp. 2910-2914. https://doi. org/10.1109/ICIP.2018.8451156. · DOI
  12. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C., 2016. SSD: single shot MultiBox detector. In: Proc. European Conference on Computer Vision.
  13. Liu, M., Li, Y., Liu, H., 2020. 3D gaze estimation for head-mounted eye tracking system with auto-calibration method. IEEE Access 8, 104207-104215. https://doi.org/10. 1109/ACCESS.2020.2999633. · DOI
  14. Lungaro, P., Sjoberg, R., Valero, A.J.F., Mittal, A., Tollmar, K., 2018. Gaze-aware streaming solutions for the next generation of mobile VR experiences. IEEE Trans. Vis. Comput. Graphics 24 (4), 1535-1544. https://doi.org/10.1109/TVCG.2018. 2794119. · DOI
  15. Orlosky, J., Toyama, T., Kiyokawa, K., Sonntag, D., 2015. ModulAR: eye-controlled vision augmentations for head mounted displays. IEEE Trans. Vis. Comput. Graphics 21 (11), 1259-1268. https://doi.org/10.1109/TVCG.2015.2459852. · DOI
  16. Park, H.M., Cho, D.Y., Yoon, K.J., 2018. Greedy refinement of object proposals via boundary-aligned minimum bounding box search. IET Comput. Vis. 12 (3), 357-363. https://doi.org/10.1049/iet-cvi.2017.0208. · DOI
  17. Redmon, J., Farhadi, A., 2018. YOLOv3: An incremental improvement. arXiv:1804. 02767.
  18. Simonyan, K., Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. · arXiv
  19. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., 2015. Going deeper with convolutions. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition.
  20. Teng, G., He, Y., Zhao, H., Liu, D., Xiao, J., R, KU.M.A.R., 2019. Design and development of human computer interface using electrooculogram with deep learning. In: Artificial Intelligence in Medicine. 101765. https://doi.org/10.1016/ j.artmed.2019.101765. · DOI
  21. Wentao, D., Liu, Y., Raffaele, G., Giancarlo, F., 2021. ANFIS fusion algorithm for eye movement recognition via soft multi-functional electronic skin. Inf. Fusion 71, 99-108. https://doi.org/10.1016/j.inffus.2021.02.003. · DOI · Paper page
  22. Wu, Q., Shen, C., Wang, P., Dick, A., Hengel, A., 2017. Image captioning and visual question answering based on attributes and external knowledge. IEEE Trans. Pattern Anal. Mach. Intell. 40 (6), 1367-1381. https://doi.org/10.1109/TPAMI. 2017.2708709. · DOI
  23. Xiaobing, Han, Tiantian, He, Yew-Soon, Ong, Yanfei, Zhong, 2020. Precise object detection using adversarially augmented local/global feature fusion. Eng. Appl. Artif. Intell. 94, 103710. https://doi.org/10.1016/j.engappai.2020.103710. · DOI
  24. Zitnick, C.L., Dollár, P., 2014. Edge boxes: locating object proposals from edges. In: Proc. European Conference on Computer Vision.

Cited by

Sorted by citations

Authors
Cho, D.-Y., & Kang, M.-K.
Affiliations
Korea Institute of Science and Technology (KR)
Corresponding author
Kang, Min-Koo
ORCID
Venue
Engineering Applications of Artificial Intelligence
Publisher
Elsevier BV
Published
2021-11
Volume
106
Pages
104471
ISSN
0952-1976 (ISSN-L)
DOI
10.1016/j.engappai.2021.104471
OpenAlex
W3204589851
Cited by
26
References
4
Citations by year
2026: 3 · 2025: 3 · 2024: 9 · 2023: 9 · 2022: 2
FWCI
1.74
Type
journal article
Language
en
Open access
Licence
Locations
Engineering Applications of Artificial Intelligence (publishedVersion, cc-by-nc-nd)
Funders
  • Ministry of Trade, Industry and Energy (17-CM-DP-20, 21-CM-BD-05)
  • Defense Acquisition Program Administration
  • Institute of Civil-Military Technology Cooperation
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
  • Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
Keywords
ambient intelligence, human gaze, gaze, small object detection, occluded object detection, human behavior understanding, human-robot symbiosis, augmented reality, mixed reality, computational efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review