A Video-Based Screening System for Automated Risk Assessment Using Nuanced Facial Features
Abstract
This study investigates the development of an automated interviewing system that uses facial behavior as an indicator of the risk of given illicit behavior. Traditional facial emotion indicators of risk in semistructured dialogue may have limitations in an automated approach. However, an initial analysis of mock crime interviews suggests that the face may exhibit some form of rigidity during highly structured interviews. An interviewing system design using facial rigidity analysis was implemented and experimentally evaluated, the results of which further reveal that the rigidity is fairly generalized across the face. Whereas existing theory traditionally focuses on leakage of facial expressions, this study provides evidence that neutralization of facial expression may be a valuable alternative for automated interviewing systems. The proof-of-concept system in this study may help human risk assessment move beyond traditional boundaries, into fields such as auditing, emergency room management, and security screening.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our results confirm that questioning strategies which maximize cognitive burden can amplify signals of deception.
“Our results confirm findings of previous research in face-to-face and remote person-to-person contexts that questioning strategies which maximize cognitive burden can amplify signals of deception (e.g., Bogaard et al., 2016; DePaulo et al., 2003; Pentland et al., 2017), highlighting the potential of remote automated CMC.”
From Sorting Insiders From Co-Workers: Remote Synchronous Computer-Mediated Triage for Investigating Insider Attacks (2024) · cites this paper as Pentland et al., 2017
Cites
The psychology of fear and stress
(1988) · Choice Reviews Online · doi:10.5860/choice.26-2399 · Cited by 3,134 · see citing works
DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.1.74 · Cited by 2,479 · see citing works
Nonverbal Leakage and Clues to Deception <sup>†</sup>
Ekman, P., & Friesen, W. V. (1969) · Psychiatry · doi:10.1080/00332747.1969.11023575 · Cited by 1,854 · see citing works
Verbal and Nonverbal Communication of Deception
Zuckerman, M., DePaulo, B. M., & Rosenthal, R. (1981) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60369-x · Cited by 1,341 · see citing works
Detecting Lies and Deceit: Pitfalls and Opportunities
Vrij, A. (2008) · Cited by 1,309 · see citing works
Detecting deception from the body or face.
Ekman, P., & Friesen, W. V. (1974) · Journal of Personality and Social Psychology · doi:10.1037/h0036006 · Cited by 739 · see citing works
Porter, S., & ten Brinke, L. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02116.x · Cited by 467 · see citing works
Ben‐Shakhar, G., & Elaad, E. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.1.131 · Cited by 405 · see citing works
A Tremor in the Blood: Uses and Abuses of the Lie Detector
Lykken, D. T. (1982) · Michigan Law Review · doi:10.2307/1288253 · Cited by 350 · see citing works
Psychology and the lie detector industry.
Lykken, D. T. (1974) · American Psychologist · doi:10.1037/h0037441 · Cited by 325 · see citing works
The Polygraph and Lie Detection
National Research Council (2003) · National Academies Press eBooks · doi:10.17226/10420 · Cited by 308 · see citing works
THE TIMING OF FACIAL MOTION IN POSED AND SPONTANEOUS SMILES
Cohn, J. F., & Schmidt, K. L. (2004) · International Journal of Wavelets Multiresolution and Information Processing · doi:10.1142/s021969130400041x · Cited by 301 · see citing works
Facing Freeze: Social Threat Induces Bodily Freeze in Humans
Roelofs, K., Hagenaars, M. A., & Stins, J. (2010) · Psychological Science · doi:10.1177/0956797610384746 · Cited by 288 · see citing works
Embodied Conversational Agent-Based Kiosk for Automated Interviewing
Nunamaker, J. F., Derrick, D. C., Elkins, A. C., Burgoon, J. K., & Patton, M. W. (2011) · Journal of Management Information Systems · doi:10.2753/mis0742-1222280102 · Cited by 280 · see citing works
Self-citationThe Motivational Impairment Effect in the Communication of Deception
DePaulo, B. M., & Kirkendol, S. E. (1989) · Springer eBooks · doi:10.1007/978-94-015-7856-1_3 · Cited by 233 · see citing works
The truth about lies: What works in detecting high‐stakes deception?
Porter, S., & ten Brinke, L. (2009) · Legal and Criminological Psychology · doi:10.1348/135532509x433151 · Cited by 204 · see citing works
An Empirical Test of the Behaviour Analysis Interview.
Vrij, A., Mann, S. A., & Fisher, R. P. (2006) · Law and Human Behavior · doi:10.1007/s10979-006-9014-3 · Cited by 141 · see citing works
Imaging Facial Physiology for the Detection of Deceit
Tsiamyrtzis, P., Dowdall, J. B., Shastri, D. J., Pavlidis, I. T., Frank, M. G., & Ekman, P. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-006-6106-y · Cited by 136 · see citing works
Behavioral Correlates of Deception in a Simulated Police Interview
Vrij, A. (1995) · The Journal of Psychology · doi:10.1080/00223980.1995.9914944 · Cited by 94 · see citing works
Does “lie to me” lie to you? An evaluation of facial clues to high-stakes deception
Su, L., & Levine, M. D. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.01.009 · Cited by 76 · see citing works
Derrick, D. C., Jenkins, J. L., & Nunamaker, J. F. (2011) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00027 · Cited by 71 · see citing works
Self-citationNonverbal cues to deception among intimates, friends, and strangers
Buller, D. B., & Aune, R. K. (1987) · Journal of Nonverbal Behavior · doi:10.1007/bf00987257 · Cited by 68 · see citing works
Twyman, N. W., Lowry, P. B., Burgoon, J. K., & Nunamaker, J. F. (2014) · Journal of Management Information Systems · doi:10.1080/07421222.2014.995535 · Cited by 60 · see citing works
Self-citationControlling nonverbal displays: Facial expressions and tone of voice
Zuckerman, M., Larrance, D. T., Spiegel, N. H., & Klorman, R. (1981) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(81)90037-8 · Cited by 52 · see citing works
The Polygraph and the Detection of Deception
Meijer, E. H., & Verschuère, B. (2010) · Journal of Forensic Psychology Practice · doi:10.1080/15228932.2010.481237 · Cited by 50 · see citing works
Detecting Deceptive Chat-Based Communication Using Typing Behavior and Message Cues
Derrick, D. C., Meservy, T. O., Jenkins, J. L., Burgoon, J. K., & Nunamaker, J. F. (2013) · ACM Transactions on Management Information Systems · doi:10.1145/2499962.2499967 · Cited by 49 · see citing works
Self-citationBorder Security Credibility Assessments via Heterogeneous Sensor Fusion
Derrick, D. C., Elkins, A. C., Burgoon, J. K., Nunamaker, J. F., & Zeng, D. D. (2010) · IEEE Intelligent Systems · doi:10.1109/mis.2010.79 · Cited by 46 · see citing works
Self-citationTrial by polygraph: Scientific and juridical issues in lie detection
Ben‐Shakhar, G., Bar‐Hillel, M., & Lieblich, I. (1986) · Behavioral Sciences & the Law · doi:10.1002/bsl.2370040408 · Cited by 41 · see citing works
A Rigidity Detection System for Automated Credibility Assessment
Twyman, N. W., Elkins, A. C., Burgoon, J. K., & Nunamaker, J. F. (2014) · Journal of Management Information Systems · doi:10.2753/mis0742-1222310108 · Cited by 34 · see citing works
Self-citationRobustness of Multiple Indicators in Automated Screening Systems for Deception Detection
Twyman, N. W., Proudfoot, J. G., Schuetzler, R. M., Elkins, A. C., & Derrick, D. C. (2015) · Journal of Management Information Systems · doi:10.1080/07421222.2015.1138569 · Cited by 34 · see citing works
Self-citationThe classification of facial emotions: a computer-based taxonomic approach
Pilowsky, I., & Katsikitis, M. (1994) · Journal of Affective Disorders · doi:10.1016/0165-0327(94)90152-x · Cited by 29 · see citing works
Too Tuned-Out to Take: The Role of Nonverbal Sensitivity in Help-Seeking
DePaulo, B. M., & Fisher, J. D. (1981) · Personality and Social Psychology Bulletin · doi:10.1177/014616728172003 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Using Design-Science Based Gamification to Improve Organizational Security Training and Compliance
Silić, M., & Lowry, P. B. (2020) · Journal of Management Information Systems · doi:10.1080/07421222.2019.1705512 · Cited by 212 · see citing works
Zhu, H., Samtani, S., Brown, R. A., & Chen, H. (2021) · MIS Quarterly · doi:10.25300/misq/2021/15574 · Cited by 61 · see citing works
Human Identification for Activities of Daily Living: A Deep Transfer Learning Approach
Zhu, H., Samtani, S., Chen, H., & Nunamaker, J. F. (2020) · Journal of Management Information Systems · doi:10.1080/07421222.2020.1759961 · Cited by 58 · see citing works
Self-citationYang, J., Amrollahi, A., & Marrone, M. (2024) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2024.101864 · Cited by 59 · see citing works
Chen, Y., Zahedi, F. M., Abbasi, A., & Dobolyi, D. (2021) · Information & Management · doi:10.1016/j.im.2020.103394 · Cited by 53 · see citing works
Microexpressions Are Not the Best Way to Catch a Liar
Burgoon, J. K. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01672 · Cited by 50 · see citing works
Self-citationNonverbal Behaviors “Speak” Relational Messages of Dominance, Trust, and Composure
Burgoon, J. K., Wang, X., Chen, X., Pentland, S. J., & Dunbar, N. E. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.624177 · Cited by 38 · see citing works
Self-citationSuen, H.-Y., Hung, K.-E., Liu, C., Su, Y.-S., & Fan, H.-C. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3376732 · Cited by 17 · see citing works
Twyman, N. W., Pentland, S. J., & Spitzley, L. A. (2020) · Journal of Management Information Systems · doi:10.1080/07421222.2020.1790201 · Cited by 13 · see citing works
Self-citationThe Research Method we Need or Deserve? A Literature Review of the Design Science Research Landscape
Nagle, T., Doyle, C., Alhassan, I. M., & Sammon, D. (2022) · Communications of the Association for Information Systems · doi:10.17705/1cais.05015 · Cited by 12 · see citing works
Enhancing cybersecurity capability investments: Evidence from an experiment
Pigola, A., da Costa, P. R., Ferasso, M., & da Silva, L. F. C. (2023) · Technology in Society · doi:10.1016/j.techsoc.2023.102449 · Cited by 12 · see citing works
Identity Unbiased Deception Detection by 2D-to-3D Face Reconstruction
Ngo, M., Wang, W., Mandira, B., Karaoğlu, S., Bouma, H., Dibeklioğlu, H., & Gevers, T. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00019 · Cited by 12 · see citing works
Detecting Deceptive Behaviours through Facial Cues from Videos: A Systematic Review
D’Ulizia, A., D’Andrea, A., Grifoni, P., & Ferri, F. (2023) · Applied Sciences · doi:10.3390/app13169188 · Cited by 11 · see citing works
Twyman, N. W., Proudfoot, J. G., Cameron, A., Case, E., Burgoon, J. K., & Twitchell, D. P. (2020) · Journal of Management Information Systems · doi:10.1080/07421222.2020.1759938 · Cited by 11 · see citing works
Self-citationDando, C. J., Taylor, P. J., Menacere, T., Ormerod, T. C., Ball, L. J., & Sandham, A. L. (2024) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208211068292 · Cited by 10 · see citing works
“…that questioning strategies which maximize cognitive burden can amplify signals of deception (e.g., Bogaard et al., 2016; DePaulo et al., 2003; Pentland et al., 2017), highlighting the potential of remote automated CMC. Open questions increased the information harvested, eliciting an…”
BackgroundThe power of satellite imagery in credit scoring: a spatial analysis of rural loans
Leng, M., Li, Z., Dai, W., & Shi, B. (2024) · Annals of Operations Research · doi:10.1007/s10479-024-06299-5 · Cited by 7 · see citing works
Trust and deception with high stakes: Evidence from the friend or foe dataset
Chen, X., Wang, X., Spitzley, L. A., & Nunamaker, J. F. (2023) · Decision Support Systems · doi:10.1016/j.dss.2023.113997 · Cited by 6 · see citing works
Self-citationTotty, S., Li, H., Zhang, C., & Janz, B. D. (2024) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3685235.3685242 · Cited by 5 · see citing works
Beyond the Face: Nonverbal Messages from the Voice, Head, Eyes and Posture
Burgoon, J. K., Dunbar, N. E., Pentland, S. J., Wang, X., Chen, X., & Ge, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-70064-4_1 · Cited by 4 · see citing works
Self-citationBurgoon, J. K. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00774 · Cited by 4 · see citing works
Self-citationDeception Detection in Online Automated Job Interviews
Twyman, N. W., Pentland, S. J., & Spitzley, L. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-91716-0_16 · Cited by 4 · see citing works
Self-citationAttention-Based Facial Behavior Analytics in Social Communication
Wang, L., Bai, C., Bolonkin, M., Burgoon, J. K., Dunbar, N. E., Subrahmanian, V. S., & Metaxas, D. (2021) · Terrorism, security and computation · doi:10.1007/978-3-030-54383-9_7 · Cited by 3 · see citing works
Self-citationBurgoon, J. K., Hamel, L. M., Blair, J. P., & Twyman, N. W. (2020) · Springer eBooks · doi:10.1007/978-3-030-34964-6_4 · Cited by 2 · see citing works
Self-citationTeaching Real-time Object Detection with an Emphasis on Engagement and Inclusiveness
Khachatryan, D., & Parise, S. (2025) · Communications of the Association for Information Systems · doi:10.17705/1cais.05625 · Cited by 2 · see citing works
Petrunin, Y. Y., & Pugacheva, A. (2021) · Digital Technology Security · doi:10.17212/2782-2230-2021-2-97-121 · Cited by 2 · see citing works
The Research Method we Need or Deserve? A Literature Review of the Design Science Research LandscapePreprint
Nagle, T., Doyle, C., Alhassan, I., & Sammon, D. (2020) · doi:10.31219/osf.io/8t7ra · Cited by 2 · see citing works
Pentland, S. J., Wang, X., & Twyman, N. W. (2024) · Information & Management · doi:10.1016/j.im.2024.104077 · Cited by 1 · see citing works
Self-citationMovement of deception in motion capture
Lee, S.-K., & Fujiwara, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-20163-x · Cited by 1 · see citing works
Don’t Lie to Me: Tracking Eye Movement and Mouse Trajectory to Detect Deception in Sharing Economy
Wu, P., Gu, J., & Lu, T. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-92270-6_54 · Cited by 1 · see citing works
AI-powered Recruitment: A Bibliometric Analysis to Identify Emerging Trends and Research Gaps
Pandiya, B. (2025) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83662-982-520251009 · Cited by 1 · see citing works
Deception Detection by 2D-to-3D Face Reconstruction from VideosPreprint
Ngô, M., Mandira, B., Yilmaz, S. F., Heij, W., Karaoğlu, S., Bouma, H., Dibeklioğlu, H., & Gevers, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.10558 · Cited by 1 · see citing works
Developing Smart and Unobtrusive Mobile Home Care: A Deep Learning Approach
Zhu, H. (2019) · UA Campus Repository (The University of Arizona)
The Research Method we Need or Deserve? A Literature Review of the Design Science Research LandscapePreprint
Nagle, T., Doyle, C., Alhassan, I., & Sammon, D. (2020) · doi:10.31219/osf.io/yjbd7
The future of automated capture of social kinesic signals for psychiatric purposes
Burgoon, J. K., Elkins, A. C., Derrick, D. C., Walls, B. E., & Metaxas, D. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1168712
Self-citationExpressive liars, credible impressions: Facial expressions across three deception paradigmsPreprint
Gullotta, M., Paterson, H., & MacCann, C. (2026) · Research Square · doi:10.21203/rs.3.rs-10945861/v1
- Authors
- Pentland, S. J., Twyman, N. W., Burgoon, J. K., Nunamaker, J. F., & Diller, C. B. R.
- Affiliations
- Missouri University of Science and Technology (US)
- University of Nebraska at Omaha (US)
- Venue
- Journal of Management Information Systems
- Publisher
- Informa UK Limited
- Published
- 2017-10-02
- Volume
- 34
- Issue
- 4
- Pages
- 970-993
- ISSN
- 0742-1222 (ISSN-L)
- 1557-928X
- OpenAlex
- W2782393250
- Cited by
- 35
- References
- 32
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 6 · 2023: 4 · 2022: 2 · 2021: 5 · 2020: 8 · 2019: 2 · 2018: 4
- FWCI
- 5.48
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-10-26)
- Open access
- closed
- Locations
- Journal of Management Information Systems (publishedVersion)
- ScholarsArchive (Brigham Young University) (submittedVersion)
- Funders
- Department of Homeland Security’s (DHS) National Center for Border Security and Immigration (BORDERS)
- National Center for Credibility Assessment (NCCA)
- Center for Identification Technology Research (CITeR), a National Science Foundation (NSF) Industry/University Cooperative Research Center (I/UCRC)
- Topics
- Social Sciences › Psychology › Social Psychology › Deception detection and forensic psychology
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- risk assessment, semi-structured interviews, security screening
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
