CoChat
Sign Up
PreprintCited by 4Emotion and Mood RecognitionGaze Tracking and Assistive TechnologyHuman-Automation Interaction and Safety

The DOI matches its DataCite record.

Multimodal Observation and Interpretation of Subjects Engaged in Problem Solving

Open access
Guntz, Thomas, Balzarini, Raffaella, Vaufreydaz, Dominique, Crowley, James L.·2017·arXiv
DOI 10.48550/arxiv.1710.04486arXiv 1710.04486
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper we present the first results of a pilot experiment in the capture and interpretation of multimodal signals of human experts engaged in solving challenging chess problems. Our goal is to investigate the extent to which observations of eye-gaze, posture, emotion and other physiological signals can be used to model the cognitive state of subjects, and to explore the integration of multiple sensor modalities to improve the reliability of detection of human displays of awareness and emotion. We observed chess players engaged in problems of increasing difficulty while recording their behavior. Such recordings can be used to estimate a participant's awareness of the current situation and to predict ability to respond effectively to challenging situations. Results show that a multimodal approach is more accurate than a unimodal one. By combining body posture, visual attention and emotion, the multimodal approach can reach up to 93% of accuracy when determining player's chess expertise while unimodal approach reaches 86%. Finally this experiment validates the use of our equipment as a general and reproducible tool for the study of participants engaged in screen-based interaction and/or problem solving.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1710.04486

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 27

  1. [1][1] R. El Kaliouby and P. Robinson, 'Real-time inference of complex mental states from facial expressions and head gestures,' in Real-time vision for humancomputer interaction . Springer, 2005, pp. 181-200. -
  2. [2][2] T. Baltruˇ saitis, D. McDuff, N. Banda, M. Mahmoud, R. El Kaliouby, P. Robinson, and R. Picard, 'Real-time inference of mental states from facial expressions and upper body gestures,' in Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on . IEEE, 2011, pp. 909-914. -
  3. [3][3] T. Baltruaitis, P. Robinson, and L. P. Morency, 'Openface: An open source facial behavior analysis toolkit,' in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) , March 2016, pp. 1-10. -
  4. [4][4] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, 'Realtime multi-person 2d pose estimation using part affinity fields,' in CVPR , 2017. -
  5. [5][5] T. Simon, H. Joo, I. Matthews, and Y. Sheikh, 'Hand keypoint detection in single images using multiview bootstrapping,' in CVPR , 2017. -
  6. [6][6] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, 'Convolutional pose machines,' in CVPR , 2016. -
  7. [8][8] P. Ekman and W. V. Friesen, 'Nonverbal leakage and clues to deception,' Psychiatry , vol. 32, no. 1, pp. 88-106, 1969. -
  8. [9][9] M.-Z. Poh, D. J. McDuff, and R. W. Picard, 'Advancements in noncontact, multiparameter physiological measurements using a webcam,' IEEE transactions on biomedical engineering , vol. 58, no. 1, pp. 7-11, 2011. -
  9. [10][10] J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore, 'Real-time human pose recognition in parts from single depth images,' Communications of the ACM , vol. 56, no. 1, pp. 116-124, 2013. -
  10. [11][11] R. Stiefelhagen, J. Yang, and A. Waibel, 'A model-based gaze tracking system,' International Journal on Artificial Intelligence Tools , vol. 6, no. 02, pp. 193-209, 1997. -
  11. [12][12] N. Charness, E. M. Reingold, M. Pomplun, and D. M. Stampe, 'The perceptual aspect of skilled performance in chess: Evidence from eye movements,' Memory & cognition , vol. 29, no. 8, pp. 1146-1152, 2001. -
  12. [13][13] L. Paletta, A. Dini, C. Murko, S. Yahyanejad, M. Schwarz, G. Lodron, S. Ladst¨ atter, G. Paar, and R. Velik, 'Towards real-time probabilistic evaluation of situation awareness from human gaze in human-robot interaction,' in Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction , ser. HRI '17. New York, NY, USA: ACM, 2017, pp. 247-248. [Online]. Available: http://doi.acm.org/10.1145/3029798.3038322 - · DOI
  13. [14][14] T. Giraud, M. Soury, J. Hua, A. Delaborde, M. Tahon, D. A. G. Jauregui, V. Eyharabide, E. Filaire, C. Le Scanff, L. Devillers et al. , 'Multimodal expressions of stress during a public speaking task: Collection, annotation and global analyses,' in Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on . IEEE, 2013, pp. 417-422. -
  14. [15][15] M. K. Abadi, J. Staiano, A. Cappelletti, M. Zancanaro, and N. Sebe, 'Multimodal engagement classification for affective cinema,' in Affective Computing and 16. Intelligent Interaction (ACII), 2013 Humaine Association Conference on . IEEE, 2013, pp. 411-416. -
  15. [16][16] E. M. Reingold and N. Charness, 'Perception in chess: Evidence from eye movements,' Cognitive processes in eye guidance , pp. 325-354, 2005. -
  16. [17][17] M. Portaz, M. Garcia, A. Barbulescu, A. Begault, L. Boissieux, M.-P. Cani, R. Ronfard, and D. Vaufreydaz, 'Figurines, a multimodal framework for tangible storytelling,' in WOCCI 2017 - 6th Workshop on Child Computer Interaction at ICMI 2017 - 19th ACM International Conference on Multi-modal Interaction , Glasgow, United Kingdom, Nov. 2017, author version. [Online]. Available: https://hal.inria.fr/hal-01595775 -
  17. [18][18] D. Vaufreydaz and A. N` egre, 'MobileRGBD, An Open Benchmark Corpus for mobile RGB-D Related Algorithms,' in 13th International Conference on Control, Automation, Robotics and Vision , Singapour, Singapore, Dec. 2014. [Online]. Available: https://hal.inria.fr/hal-01095667 -
  18. [19][19] K. Holmqvist, M. Nystr¨ om, R. Andersson, R. Dewhurst, H. Jarodzka, and J. Van de Weijer, Eye tracking: A comprehensive guide to methods and measures . OUP Oxford, 2011. -
  19. [20][20] A. Poole and L. J. Ball, 'Eye tracking in hci and usability research,' Encyclopedia of human computer interaction , vol. 1, pp. 211-219, 2006. -
  20. [21][21] C. Ehmke and S. Wilson, 'Identifying web usability problems from eye-tracking data,' in Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI… but not as we know it-Volume 1 . British Computer Society, 2007, pp. 119-128. -
  21. [22][22] M. Den Uyl and H. Van Kuilenburg, 'The facereader: Online facial expression recognition,' in Proceedings of measuring behavior , vol. 30, 2005, pp. 589-590. -
  22. [23][23] O. Langner, R. Dotsch, G. Bijlstra, D. H. Wigboldus, S. T. Hawk, and A. Van Knippenberg, 'Presentation and validation of the radboud faces database,' Cognition and emotion , vol. 24, no. 8, pp. 1377-1388, 2010. -
  23. [24][24] E. Goeleven, R. De Raedt, L. Leyman, and B. Verschuere, 'The karolinska directed emotional faces: a validation study,' Cognition and emotion , vol. 22, no. 6, pp. 1094-1118, 2008. -
  24. [26][26] S. M. Anzalone, S. Boucenna, S. Ivaldi, and M. Chetouani, 'Evaluating the engagement with social robots,' International Journal of Social Robotics , vol. 7, no. 4, pp. 465-478, 2015. -
  25. [27][27] J. A. Harrigan, 'Self-touching as an indicator of underlying affect and language processes,' Social Science & Medicine , vol. 20, no. 11, pp. 1161-1168, 1985. -
  26. [28][28] W. Johal, D. Pellier, C. Adam, H. Fiorino, and S. Pesty, 'A cognitive and affective architecture for social human-robot interaction,' in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts . ACM, 2015, pp. 71-72. -
  27. [29][29] J. Aigrain, M. Spodenkiewicz, S. Dubuisson, M. Detyniecki, D. Cohen, and M. Chetouani, 'Multimodal stress detection from multiple assessments,' IEEE Transactions on Affective Computing , 2016.

Cited by

Sorted by citations

Metadata

Authors
Guntz, T., Balzarini, R., Vaufreydaz, D., & Crowley, J. L.
Affiliations
  • Laboratoire d'Informatique de Grenoble (FR)
  • Université Grenoble Alpes (FR)
  • Institut polytechnique de Grenoble (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1710.04486
arXiv
1710.04486
OpenAlex
W2763258256
Cited by
4
References
0
Citations by year
2022: 2 · 2020: 1 · 2017: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
Agence Nationale de la Recherche (ANR-11-EQPX-0002, ANR-15-CE23-0005)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Keywords
multimodal sensing, gaze, posture, emotion recognition, cognitive state estimation, chess problem solving, chess expertise, physiological signals, screen-based interaction, problem solving

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review