Towards Measuring Fairness in AI: The Casual Conversations Dataset
Open accessThe DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index.
Abstract
This paper introduces a novel dataset to help researchers evaluate their computer vision and audio models for accuracy across a diverse set of age, genders, apparent skin tones and ambient lighting conditions. Our dataset is composed of 3,011 subjects and contains over 45,000 videos, with an average of 15 videos per person. The videos were recorded in multiple U.S. states with a diverse set of adults in various age, gender and apparent skin tone groups. A key feature is that each subject agreed to participate for their likenesses to be used. Additionally, our age and gender annotations are provided by the subjects themselves. A group of trained annotators labeled the subjects’ apparent skin tone using the Fitzpatrick skin type scale. Moreover, annotations for videos recorded in low ambient lighting are also provided. As an application to measure robustness of predictions across certain attributes, we provide a comprehensive study on the top five winners of the DeepFake Detection Challenge (DFDC). Experimental evaluation shows that the winning models are less performant on some specific groups of people, such as subjects with darker skin tones and thus may not generalize to all people. In addition, we also evaluate the state-of-the-art apparent age and gender classification methods. Our experiments provides a thorough analysis on these models in terms of fair treatment of people from various backgrounds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Similar observations are found in [19].”
From Improving Fairness in Deepfake Detection (2024) · doi:10.1109/wacv57701.2024.00459 · cites this paper as [19]
Citations
Cites
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
(2018) · Cited by 3,510
Dlib-ml: A Machine Learning Toolkit
King, D. E. (2009) · doi:10.5555/1577069.1755843 · Cited by 2,777
Age and gender classification using convolutional neural networks
Levi, G., & Hassncer, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301352 · Cited by 1,151
Age Progression/Regression by Conditional Adversarial Autoencoder
Zhang, Z., Song, Y., & Qi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.463 · Cited by 1,145
Age and Gender Estimation of Unfiltered Faces
Eidinger, E., Enbar, R., & Hassner, T. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2359646 · Cited by 809
MORPH: A Longitudinal Image Database of Normal Adult Age-Progression
Ricanek, K., & Tesafaye, T. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.78 · Cited by 778
Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A
Klare, B. F., Klein, B., Taborsky, E., Blanton, A., Cheney, J., Allen, K., Grother, P., Mah, A., Burge, M., & Jain, A. K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298803 · Cited by 701
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Research works printed but not linked: 9
- 10.1145/1577069.1755843 · doi:10.1145/1577069.1755843
- Seferbekov, 2021, DeepFake Detection Challenge (DFDC) Solution
- Zhao, 2021, DeepFake Detection Challenge (DFDC) Solution
- Davletshin, 2021, DeepFake Detection Challenge (DFDC) Solution
- Shao, 2021, DeepFake Detection Challenge (DFDC) Solution
- Howard, 2021, DeepFake Detection Challenge (DFDC) Solution
- Fitzpatrick, 1975, Soleil et peau’ [Sun and skin], J. Med. Esthétique, 2, 33
- Barocas, 2019, Fairness and Machine Learning
- 10.1145/1577069.1755843 · doi:10.1145/1577069.1755843
Cited by
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“Using this classifier, we run inference on 2955 images from the Casual Conversations dataset (Hazirbas et al., 2021) and keep all labels in the top-5 that are assigned a probability of 0.1 or more.”
BackgroundMethodInfluentialWe Need to Consider Disagreement in Evaluation
Basile, V., Fell, M., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M., & Uma, A. (2021) · Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future · doi:10.18653/v1/2021.bppf-1.3 · Cited by 73
A survey on bias in visual datasets
Fabbrizzi, S., Papadopoulos, S., Ntoutsi, E., & Kompatsiaris, I. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103552 · Cited by 137
Preserving Fairness Generalization in Deepfake Detection
Lin, L., He, X., Ju, Y., Wang, X., Ding, F., & Hu, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01591 · Cited by 97
Auto-essentialization: Gender in automated facial analysis as extended colonial project
Scheuerman, M. K., Pape, M., & Hanna, A. (2021) · Big Data & Society · doi:10.1177/20539517211053712 · Cited by 75
Metadata
- Authors
- Hazirbas, C., Bitton, J., Dolhansky, B., Pan, J., Gordo, A., & Ferrer, C. C.
- Affiliations
- Meta AI, AI Integrity, Seattle, WA, USA
- Meta (United States) (US)
- Venue
- IEEE Transactions on Biometrics, Behavior, and Identity Science
- Published
- 2022-07
- Volume
- 4
- Issue
- 3
- Pages
- 324-332
- ISSN
- 2637-6407 (ISSN-L)
- OpenAlex
- W3140832216
- Cited by
- 79
- References
- 35
- Citations by year
- 2026: 8 · 2025: 23 · 2024: 14 · 2023: 15 · 2022: 12
- FWCI
- 4.32
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Biometrics Behavior and Identity Science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- algorithmic fairness, age estimation, gender classification, ambient lighting, deepfake detection, DFDC, algorithmic bias, Fitzpatrick skin type
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards Measuring Fairness in AI: The Casual Conversations Dataset. https://app.cochat.ai/papers/10.1109/tbiom.2021.3132237
- The paper
- Hazirbas, C., Bitton, J., Dolhansky, B., Pan, J., Gordo, A., & Ferrer, C. C. (2022). Towards Measuring Fairness in AI: The Casual Conversations Dataset. IEEE Transactions on Biometrics, Behavior, and Identity Science. https://doi.org/10.1109/tbiom.2021.3132237
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.