Face Recognition Performance: Role of Demographic Information
The DOI matches its Crossref record; 37 of the 41 references it cites were found in a registry or an index.
Abstract
This paper studies the influence of demographics on the performance of face recognition algorithms. The recognition accuracies of six different face recognition algorithms (three commercial, two nontrainable, and one trainable) are computed on a large scale gallery that is partitioned so that each partition consists entirely of specific demographic cohorts. Eight total cohorts are isolated based on gender (male and female), race/ethnicity (Black, White, and Hispanic), and age group (18–30, 30–50, and 50–70 years old). Experimental results demonstrate that both commercial and the nontrainable algorithms consistently have lower matching accuracies on the same cohorts (females, Blacks, and age group 18–30) than the remaining cohorts within their demographic. Additional experiments investigate the impact of the demographic distribution in the training set on the performance of a trainable face recognition algorithm. We show that the matching accuracy for race/ethnicity and age cohorts can be improved by training exclusively on that specific cohort. Operationally, this leads to a scenario, called dynamic face matcher selection, where multiple face recognition algorithms (each trained on different demographic cohorts) are available for a biometric system operator to select based on the demographic information extracted from a probe image. This procedure should lead to improved face recognition accuracy in many intelligence and law enforcement face recognition scenarios. Finally, we show that an alternative to dynamic face matcher selection is to train face recognition algorithms on datasets that are evenly distributed across demographics, as this approach offers consistently high accuracy across all cohorts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another issue commonly overlooked in ML applications is sampling enough data points across all gender, race, age, and sexuality.[449]”
From Flexible Electronics and Devices as Human–Machine Interfaces for Medical Robotics (2022) · cites this paper as [449]
In the citing paper
“Recognizing that there is a racial, gender, sexuality, and age component in medical data reaffirms the need to account for such factors in ML applications in medical robotics to prevent the algorithm from finding a local optimum that overfits the majority of the population while underperforming on minority groups.[449,451]”
From Flexible Electronics and Devices as Human–Machine Interfaces for Medical Robotics (2022) · cites this paper as [449]
In the citing paper
“Previous studies show that accuracy of face recognition algorithms is sensitive to the demographic composition of both training and test image databases [28, 29].”
From Error Rates in Users of Automatic Face Recognition Software (2015) · cites this paper as [29]
Citations
Cites
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · cited 1×
“, bagging [28]).”
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398 · cited 1×
“: A seminal method in face recognition is the use of local binary patterns [7] (LBP) to represent the face [27].”
Eigenfaces vs. Fisherfaces: recognition using class specific linear projection
Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598228 · Cited by 11,707
The random subspace method for constructing decision forests
Ho, T. K. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.709601 · Cited by 6,842 · cited 1×
“The randomly sampled subspaces are based on Ho’s original method [32], however the proposed approach is unique in that the sampling is weighted based on the spectral densities.”
Face recognition: A literature survey
Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2003) · ACM Computing Surveys · doi:10.1145/954339.954342 · Cited by 6,163
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikainen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,584 · cited 1×
“: A seminal method in face recognition is the use of local binary patterns [7] (LBP) to represent the face [27].”
The FERET evaluation methodology for face-recognition algorithms
Phillips, P. J., Moon, H., Rizvi, S. A., & Rauss, P. J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.879790 · Cited by 3,692
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309 · cited 1×
“One reason Gabor features are popular for representing both facial and natural images is their similarity with human neurological receptor fields [30], [31].”
Face recognition by elastic bunch graph matching
Wiskott, L., Fellous, J.-M., Kuiger, N., & von der Malsburg, C. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598235 · Cited by 2,883
Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions
Tan, X., & Triggs, B. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2042645 · Cited by 2,842
Liu, C., & Wechsler, H. (2002) · IEEE Transactions on Image Processing · doi:10.1109/tip.2002.999679 · Cited by 1,886 · cited 1×
“Gabor features are one of the facial features [26] to have been used with wide success in representing facial images [8], [20], [29].”
A survey of approaches and challenges in 3D and multi-modal 3D+2D face recognition
Bowyer, K. W., Chang, K., & Flynn, P. (2006) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2005.05.005 · Cited by 979
Moghaddam, B., Jebara, T., & Pentland, A. (2000) · Pattern Recognition · doi:10.1016/s0031-3203(99)00179-x · Cited by 612 · cited 2×
“, local binary patterns [7], Gabor features [8]), feature extraction [9], [10]), and matching [11].”
Meta-analysis of facial identification studies.
Shapiro, P. N., & Penrod, S. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.100.2.139 · Cited by 547
A review on Gabor wavelets for face recognition
Shen, L., & Bai, L. (2006) · Pattern Analysis and Applications · doi:10.1007/s10044-006-0033-y · Cited by 462 · cited 1×
“Gabor features are one of the facial features [26] to have been used with wide success in representing facial images [8], [20], [29].”
FRVT 2006 and ICE 2006 Large-Scale Experimental Results
Phillips, P. J., Scruggs, W. T., O'Toole, A. J., Flynn, P. J., Bowyer, K. W., Schott, C. L., & Sharpe, M. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.59 · Cited by 406 · cited 1×
“The study in [17] was motivated by a rather surprising result in the 2006 NIST Face Recognition Vendor Test (FRVT) [19].”
Face recognition under varying pose
Beymer (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323893 · Cited by 373
A unified framework for subspace face recognition
Wang, X., & Tang, X. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.57 · Cited by 359 · cited 3×
“While 4SF is intended to be representative of learning-based methods in face recognition, it could be the case that other learning-based algorithms (such as [20]–[23]) exhibit different amounts of sensitivity to the demographic distribution of the training data.”
Performance of Biometric Quality Measures
Grother, P., & Tabassi, E. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1019 · Cited by 333
An Investigation of the Contact Hypothesis of the Own-race Bias in Face Recognition
Chiroro, P., & Valentine, T. (1995) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640749508401421 · Cited by 324 · cited 1×
“While there is no generally agreed upon explanation for this phenomenon, many researchers believe the decreased performance on other races is explained by the “contact” hypothesis, which postulates that the lower performance on other races is due to a decreased exposure [14].”
Eigenfeature Regularization and Extraction in Face Recognition
Jiang, X., Mandal, B., & Kot, A. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70708 · Cited by 310 · cited 3×
“While 4SF is intended to be representative of learningbased methods in face recognition, it could be the case that other learning-based algorithms (such as [20], [21], [22], [23]) exhibit different amounts of sensitivity to the demographic distribution of the training data.”
Bothwell, R. K., Brigham, J. C., & Malpass, R. S. (1989) · Personality and Social Psychology Bulletin · doi:10.1177/0146167289151002 · Cited by 282 · cited 1×
“Coined the “other-race effect”, humans have consistently demonstrated a decreased ability to recognize subjects from races different from their own [12], [13].”
A Genomewide Admixture Mapping Panel for Hispanic/Latino Populations
Mao, X., Bigham, A. W., Mei, R., Gutierrez, G., Weiss, K. M., Brutsaert, T. D., Leon-Velarde, F., Moore, L. G., Vargas, E., McKeigue, P. M., et al. (2007) · The American Journal of Human Genetics · doi:10.1086/518564 · Cited by 235
Random Sampling for Subspace Face Recognition
Wang, X., & Tang, X. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-006-8098-z · Cited by 206 · cited 2×
“For example, variants of the linear discriminant analysis (LDA) algorithm [9], [10] use training data to compute between-class and within-class scatter matrices.”
Furl, N. (2002) · Cognitive Science · doi:10.1016/s0364-0213(02)00084-8 · Cited by 185 · cited 1×
“However, studies by Furl et al. [16] and O'Toole et al. [17] investigated the impact of the Whites and East Asians.”
An other-race effect for face recognition algorithms
Phillips, P. J., Jiang, F., Narvekar, A., Ayyad, J., & O'Toole, A. J. (2011) · ACM Transactions on Applied Perception · doi:10.1145/1870076.1870082 · Cited by 172 · cited 1×
“However, studies by Furl et al. [16] and O'Toole et al. [17] investigated the impact of the Whites and East Asians.”
Linear Subspace Learning-Based Dimensionality Reduction
Jiang, X. (2011) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.939041 · Cited by 163
Cross-Race Facial Recognition: Failure of the Contact Hypothesis
Ng, W.-J., & Lindsay, R. C. L. (1994) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022194252004 · Cited by 159
Using Biologically Inspired Features for Face Processing
Meyers, E. M., & Wolf, L. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0058-8 · Cited by 143 · cited 1×
“One reason Gabor features are popular for representing both facial and natural images is their similarity with human neurological receptor fields [30], [31].”
An introduction to the good, the bad, & the ugly face recognition challenge problem
Phillips, P. J., Beveridge, J. R., Draper, B. A., Givens, G., O'Toole, A. J., Bolme, D. S., Dunlop, J., Lui, Y. M., Sahibzada, H., & Weimer, S. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771424 · Cited by 135 · cited 1×
“S OURCES of errors in automated face recognition algorithms are generally attributed to the well studied variations in pose, illumination, and expression [1], collectively”
Pose invariant face recognition
Huang, F. J., Zhou, Z.-H., Zhang, H.-J., & Chen, T. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840642 · Cited by 134
Face matching and retrieval in forensics applications
Jain, A. K., Klare, B., & Park, U. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.4 · Cited by 105 · cited 1×
“, resolution, compression, blur), time lapse (facial aging), and occlusion also contribute to face recognition errors [2].”
Self-citationOn a taxonomy of facial features
Klare, B., & Jain, A. K. (2010) · 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS) · doi:10.1109/btas.2010.5634533 · Cited by 70 · cited 2×
“Both of these methods operate by representing the face with Level 2 facial features (LBP and Gabor), where Level 2 facial features are features that encode the structure and shape of the face, and are critical to face recognition algorithms [26].”
Self-citationHuman and automatic face recognition: a comparison across image formats
Burton, A. M., Miller, P. C., Bruce, V., Hancock, P., & Henderson, Z. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00186-9 · Cited by 61
Face recognition across time lapse: On learning feature subspaces
Klare, B., & Jain, A. K. (2011) · 2011 International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb.2011.6117547 · Cited by 46 · cited 1×
“Similar training biases were investigated by Klare and Jain [18], showing that aging-invariant face recognition algorithms suffer from decreased performance in nonaging scenarios.”
Self-citationDemographic effects on estimates of automatic face recognition performance
O'Toole, A. J., Phillips, P. J., An, X., & Dunlop, J. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2011.12.007 · Cited by 43 · cited 1×
“For each demographic that was controlled (e.g. gender), the other demographics were uncontrolled so that they have roughly the same distribution (following the findings in [25]).”
Face recognition algorithms and the "other-race" effect
O’Toole, A. J., Phillips, P. J., Narvekar, A., Jiang, F., & Ayyad, J. (2010) · Journal of Vision · doi:10.1167/8.6.256 · Cited by 6 · cited 4×
“Furl et als [16] and O’Toole et als [17] conducted studies to investigate the impact of cross training and matching on White and Asian races.”
Influential
Research works printed but not linked: 6
- 10.1207/s15516709cog2606_4 · doi:10.1207/s15516709cog2606_4 · Paper page
- grother, 2010, MBE 2010: Report on the evaluation of 2D still-image face recognition algorithms, National Institute of Standards and Technology, 7709, 1
- phillips, 2003, Face recognition vendor test 2002: Evaluation report, National Institute of Standards and Technology, 6965, 1
- 2011, Handbook of Face Recognition
- klare, 2011, Spectrally sampled structural subspace features (4SF)
- mccabe, 2000, Data format for the interchange of fingerprint, facial, scar mark & tattoo (SMT) information, American National Standard ANSI/NIST-ITL 1-2000
Cited by
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
“ially problematic when models are used in applications that have serious impacts on people’s lives, such as in health care [14, 42, 44], employment [1, 13, 29], education [23, 45] and law enforcement [2, 7, 20, 34]. Researchers have discovered systematic biases in commercial machine learning models used for face detec…”
BackgroundDeep face recognition: A survey
Wang, M., & Deng, W. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.10.081 · Cited by 1,466
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
“, 2020), to computer vision (Klare et al., 2012), to hiring (Garcia, 2016).”
Background50 years of biometric research: Accomplishments, challenges, and opportunities
Jain, A. K., Nandakumar, K., & Ross, A. (2016) · Pattern Recognition Letters · doi:10.1016/j.patrec.2015.12.013 · Cited by 708
Self-citation
Metadata
- Authors
- Klare, B. F., Burge, M. J., Klontz, J. C., Vorder Bruegge, R. W., & Jain, A. K.
- Affiliations
- Noblis (US)
- Korea University (KR)
- ORCID
- Jain, Anil K.
- Venue
- IEEE Transactions on Information Forensics and Security
- Published
- 2012-12
- Volume
- 7
- Issue
- 6
- Pages
- 1789-1801
- ISSN
- 1556-6013 (ISSN-L)
- 1556-6021
- OpenAlex
- W1978642336
- Cited by
- 516
- References
- 41
- Citations by year
- 2026: 18 · 2025: 40 · 2024: 70 · 2023: 55 · 2022: 57 · 2021: 68 · 2020: 60 · 2019: 44 · 2018: 25 · 2017: 12 · 2016: 17 · 2015: 22 · 2014: 16 · 2013: 9 · 2012: 2
- FWCI
- 16.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Information Forensics and Security (publishedVersion)
- Funders
- National Institute of Standards and Technology
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- face recognition, demographic bias, gender, race and ethnicity, age, biometric systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.