Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
Open accessAbstract
Machine learning datasets have elicited concerns about privacy, bias, and unethical applications, leading to the retraction of prominent datasets such as DukeMTMC, MS-Celeb-1M, and Tiny Images. In response, the machine learning community has called for higher ethical standards in dataset creation. To help inform these efforts, we studied three influential but ethically problematic face and person recognition datasets -- Labeled Faces in the Wild (LFW), MS-Celeb-1M, and DukeMTM -- by analyzing nearly 1000 papers that cite them. We found that the creation of derivative datasets and models, broader technological and social change, the lack of clarity of licenses, and dataset management practices can introduce a wide range of ethical concerns. We conclude by suggesting a distributed approach to harm mitigation that considers the entire life cycle of a dataset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There has been an increased focus on analyzing other dimensions of data quality and stewardship, with several initiatives aiming to document existing and newly developed resources.
“There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [102]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · see citing works
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781 · see citing works
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · see citing works
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479 · see citing works
From few to many: illumination cone models for face recognition under variable lighting and pose
Georghiades, A. S., Belhumeur, P. N., & Kriegman, D. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927464 · Cited by 4,917 · see citing works
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,445 · see citing works
(2018) · Cited by 3,510 · see citing works
Performance Measures and a Data Set for Multi-target, Multi-camera Tracking
Ristani, E., Solera, F., Zou, R., Cucchiara, R., & Tomasi, C. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-48881-3_2 · Cited by 3,104 · see citing works
VGGFace2: A Dataset for Recognising Faces across Pose and Age
Cao, Q., Shen, L., Xie, W., Parkhi, O. M., & Zisserman, A. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00020 · Cited by 3,032 · see citing works
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · see citing works
Self-citationModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in Vitro
Zheng, Z., Zheng, L., & Yang, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.405 · Cited by 2,142 · see citing works
MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
Guo, Y., Zhang, L., Hu, Y., He, X., & Gao, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_6 · Cited by 2,122 · see citing works
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,043 · see citing works
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921 · see citing works
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581 · see citing works
Facial Landmark Detection by Deep Multi-task Learning
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_7 · Cited by 1,410 · see citing works
Deep Convolutional Network Cascade for Facial Point Detection
Sun, Y. J., Wang, X., & Tang, X. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.446 · Cited by 1,405 · see citing works
Attribute and simile classifiers for face verification
Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459250 · Cited by 1,383 · see citing works
A Light CNN for Deep Face Representation With Noisy Labels
Wu, X., He, R., Sun, Z., & Tan, T. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2833032 · Cited by 1,213 · see citing works
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 3× · see citing works
“Researchers have worked to make sense of ethical considerations involved in dataset creation [43, 69, 32], have proposed ways to identify and mitigate biases in datasets [11, 82], have developed means to protect the privacy of individuals in datasets [70, 91], and have improved methods to document datasets [35, 47, 12…”
BackgroundIs that you? Metric learning approaches for face identification
Guillaumin, M., Verbeek, J. J., & Schmid, C. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459197 · Cited by 764 · see citing works
Pose-Guided Feature Alignment for Occluded Person Re-Identification
Miao, J., Wu, Y., Liu, P., Ding, Y., & Yang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00063 · Cited by 682 · see citing works
Effective face frontalization in unconstrained images
Hassner, T., Harel, S., Paz, E., & Enbar, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299058 · Cited by 613 · see citing works
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · cited 2× · see citing works
“(See Section 3.1 of [69] for a survey of work discussing representational concerns.)”
BackgroundInfluentialEnd-to-End People Detection in Crowded Scenes
Stewart, R., Andriluka, M., & Ng, A. Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.255 · Cited by 539 · see citing works
Exploit the Unknown Gradually: One-Shot Video-Based Person Re-identification by Stepwise Learning
Wu, Y., Lin, Y., Dong, X., Yan, Y., Ouyang, W., & Yang, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00543 · Cited by 401 · see citing works
Gross, R., Matthews, I., Cohn, J., Kanade, T., & Baker, S. (2008) · 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition · doi:10.1109/afgr.2008.4813399 · Cited by 373 · see citing works
Real-time facial feature detection using conditional regression forests
Dantone, M., Gall, J., Fanelli, G., & Gool, L. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247976 · Cited by 370 · see citing works
Racial faces in the wild: Reducing racial bias by information maximization adaptation network
Wang, M., Deng, W., Hu, J., Tao, X., & Huang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00078 · Cited by 353 · see citing works
Unsupervised Joint Alignment of Complex Images
Huang, G. B., Jain, V., & Miller, E. G. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408858 · Cited by 340 · see citing works
Wolf, L., Hassner, T., & Taigman, Y. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.230 · Cited by 332 · see citing works
Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference
Sanderson, C., & Lovell, B. C. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01793-3_21 · Cited by 303 · see citing works
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295 · cited 1× · see citing works
“These concerns include the privacy of individuals included [45, 70], representational harms introduced by annotations [25, 44], effects of biases on downstream use [20, 21, 18], and use for ethically dubious purposes [45, 75, 66].”
BackgroundHeterogeneous Face Attribute Estimation: A Deep Multi-Task Learning Approach
Han, H., Jain, A. K., Fang, W. S., Shan, S., & Chen, X. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2738004 · Cited by 280 · see citing works
Learning to Align from Scratch
Huang, G. B., Mattar, M., Lee, H., & Miller, E. G. (2012) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 270 · see citing works
Lessons from archives: strategies for collecting sociocultural data in machine learning
Jo, E. S., & Gebru, T. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372829 · Cited by 240 · see citing works
Marginal Loss for Deep Face Recognition
Deng, J., Zhou, Y., & Zafeiriou, S. P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.251 · Cited by 220 · see citing works
Sox, H. C., & Rennie, D. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-8-200604180-00123 · Cited by 200 · see citing works
Unsupervised Tracklet Person Re-Identification
Li, M., Zhu, X., & Gong, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2903058 · Cited by 186 · see citing works
Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
Steed, R., & Caliskan, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445932 · Cited by 167 · see citing works
Face Alignment Via Component-Based Discriminative Search
Liang, L., Xiao, R., Wen, F., & Sun, J. Z. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88688-4_6 · Cited by 160 · see citing works
Lightweight Face Recognition Challenge
Deng, J., Guo, J., Zhang, D., Deng, Y., Lu, X., & Shi, S. X. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00322 · Cited by 155 · see citing works
What’s next for Registered Reports?
Chambers, C. (2019) · Nature · doi:10.1038/d41586-019-02674-6 · Cited by 154 · see citing works
A benchmark study of large-scale unconstrained face recognition
Liao, S., Lei, Z., Yi, D., & Li, S. Z. (2014) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/btas.2014.6996301 · Cited by 128 · see citing works
Racial categories in machine learning
Benthall, S., & Haynes, B. D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287575 · Cited by 110 · see citing works
DukeMTMC4ReID: A Large-Scale Multi-camera Person Re-identification Dataset
Gou, M., Karanam, S., Liu, W., Camps, O., & Radke, R. J. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.185 · Cited by 108 · see citing works
A Study of Face Obfuscation in ImageNetPreprint
Yang, K., Yau, J., Fei-Fei, L., Deng, J., & Russakovsky, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06191 · Cited by 103 · see citing works
Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine LearningPreprint
Jo, E. S., & Gebru, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10389 · Cited by 102 · see citing works
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
Denton, R., Hanna, A., Amironesei, R., Smart, A., Nicole, H., & Scheuerman, M. K. (2020) · arXiv · doi:10.48550/arxiv.2007.07399 · Cited by 100 · see citing works
Large image datasets: A pyrrhic win for computer vision?
Prabhu, V. U., & Birhane, A. (2020) · arXiv · doi:10.48550/arxiv.2006.16923 · Cited by 91 · see citing works
Schneider, J., Ye, D., Hill, A. M., & Whitehorn, A. S. (2020) · Scientometrics · doi:10.1007/s11192-020-03631-1 · Cited by 88 · cited 1× · see citing works
“Several studies have shown that articles continue to be cited after retraction (e.g., [22, 9, 74]).”
BackgroundBornemann-Cimenti, H., Szilagyi, I. S., & Sandner-Kiesling, A. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-015-9680-y · Cited by 87 · see citing works
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
Wang, A., Narayanan, A., & Russakovsky, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_43 · Cited by 72 · see citing works
Self-citationDoes retraction after misconduct have an impact on citations? A pre–post study
Candal-Pedreira, C., Ruano-Ravina, A., Fernández, E., Ramos, J., Campos-Varela, I., & Pérez-Ríos, M. (2020) · BMJ Global Health · doi:10.1136/bmjgh-2020-003719 · Cited by 67 · see citing works
Algorithmic Injustices: Towards a Relational EthicsPreprint
Birhane, A., & Cummins, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07376 · Cited by 66 · see citing works
FAIR Data Reuse – the Path through Data Citation
Groth, P., Cousijn, H., Clark, T., & Goble, C. (2019) · Data Intelligence · doi:10.1162/dint_a_00030 · Cited by 64 · see citing works
Evaluating Open-Universe Face Identification on the Web
Becker, B. C., & Ortiz, E. G. (2013) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2013.133 · Cited by 52 · see citing works
Effective 3D based frontalization for unconstrained face recognition
Ferrari, C., Lisanti, G., Berretti, S., & Bimbo, A. D. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899774 · Cited by 50 · see citing works
Unconstrained face recognition: Establishing baseline human performance via crowdsourcing
Best-Rowden, L., Bisht, S., Klontz, J. C., & Jain, A. K. (2014) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/btas.2014.6996296 · Cited by 49 · see citing works
Khan, Z., & Fu, Y. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445920 · Cited by 46 · see citing works
Deep Features for Recognizing Disguised Faces in the Wild
Bansal, A., Ranjan, R., Castillo, C. D., & Chellappa, R. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00009 · Cited by 44 · see citing works
About Face: A Survey of Facial Recognition EvaluationPreprint
Raji, I. D., & Fried, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.00813 · Cited by 40 · see citing works
Xiao, H., Lin, W., Sheng, B., Lü, K., Yan, J., Wang, J., Ding, E., Zhang, Y., & Xiong, H. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240539 · Cited by 40 · see citing works
A Survey of Deep Facial Attribute Analysis
Zheng, X., Guo, Y.-Q., Huang, H.-B., Li, Y., & He, R. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-020-01308-z · Cited by 30 · see citing works
Privacy, Ethics, and Data Access: A Case Study of the Fragile Families Challenge
Lundberg, I., Narayanan, A., Levy, K. E. C., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118813023 · Cited by 29 · see citing works
Self-citationTowards Standardization of Data Licenses: The Montreal Data License
Benjamin, M., Gagnon, P., Rostamzadeh, N., Pal, C., Bengio, Y., & Shee, A. (2019) · arXiv · doi:10.48550/arxiv.1903.12262 · Cited by 26 · see citing works
Cohen, J., & Lo, H. Z. (2014) · Extreme Science and Engineering Discovery Environment · doi:10.1145/2616498.2616528 · Cited by 18 · see citing works
An Ethical Highlighter for People-Centric Dataset CreationPreprint
Hanley, M., Khandelwal, A., Averbuch‐Elor, H., Snavely, N., & Nissenbaum, H. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13583 · Cited by 9 · see citing works
Like a Researcher Stating Broader Impact For the Very First TimePreprint
Abuhamad, G., & Rheault, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13032 · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- [15][15] Tamara Berg, Alexander Berg, Jaety Edwards, and David Forsyth. Who's In the Picture. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 17. MIT Press, 2005. URL https://proceedings.neurips.cc/paper/2004/file/03fa2f7502f5f6b9169e67d17cbf51bbPaper.pdf . -
- [18][18] Su Lin Blodgett and Brendan T. O'Connor. Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English. ArXiv , abs/1707.00061, 2017. - · arXiv
- [34][34] Allen Institute for AI. Semantic Scholar API. https://api.semanticscholar.org . -
- [38][38] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker. Multi-PIE. 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition , pages 1-8, 2008. -
- [47][47] Sarah Holland, A. Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards. ArXiv , abs/1805.03677, 2018. - · arXiv
- [58][58] Yutian Lin, L. Zheng, Zhedong Zheng, Yu Wu, and Y. Yang. Improving Person Re-identification by Attribute and Identity Learning. ArXiv , abs/1703.07220, 2019. - · arXiv
- [92][92] Zhong yuan Wang, Guangcheng Wang, Baojin Huang, Zhangyang Xiong, Q. Hong, Hao Wu, Peng Yi, Kui Jiang, Nanxi Wang, Yingjiao Pei, Heling Chen, Yu Miao, Z. Huang, and Jinbi Liang. Masked Face Recognition Dataset and Application. ArXiv , abs/2003.09093, 2020. - · arXiv
- [96][96] Yuxiang Zhou and S. Zafeiriou. Deformable Models of Ears in-the-Wild for Alignment and Recognition. 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) , pages 626-633, 2017.
Cited by
Sorted by citations
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · see citing works
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160 · see citing works
Algorithmic fairness datasets: the story so far
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00854-z · Cited by 128 · see citing works
Policy advice and best practices on bias and fairness in AI
Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 128 · see citing works
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
Wang, A., Liu, A., Zhang, R., Kleiman, A., Kim, L., Zhao, D., Shirai, I., Narayanan, A., & Russakovsky, O. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01625-5 · Cited by 85 · see citing works
Self-citationData Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 45 · see citing works
Real Risks of Fake Data: Synthetic Data, Diversity-Washing and Consent Circumvention
Whitney, C. D., & Norman, J. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659002 · Cited by 47 · see citing works
Identifying and Categorizing Bias in AI/ML for Earth Sciences
McGovern, A., Bostrom, A., McGraw, M. C., Chase, R. J., Gagne, D. J., Ebert‐Uphoff, I., Musgrave, K. D., & Schumacher, A. (2024) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-23-0196.1 · Cited by 24 · see citing works
SoK: Machine Learning GovernancePreprint
Chandrasekaran, V., Jia, H., Thudi, A., Travers, A., Yaghini, M., & Papernot, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10870 · Cited by 22 · see citing works
Publicly Available Health Research Datasets: Opportunities and Responsibilities
BaHammam, A. S., & Chee, M. W. L. (2022) · Nature and Science of Sleep · doi:10.2147/nss.s390292 · Cited by 12 · see citing works
Uncoupling inequality: Reflections on the ethics of benchmarks for digital media
Washington, A. L., Rhue, L. A., Nakamura, L., & Stevens, R. C. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.352 · Cited by 11 · see citing works
Considerations for Ethical Speech Recognition Datasets
Papakyriakopoulos, O., & Xiang, A. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3575793 · Cited by 10 · see citing works
Feature and Label Embedding Spaces Matter in Addressing Image Classifier Bias
Thong, W. E., & Snoek, C. G. M. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.46 · Cited by 9 · see citing works
SynthBio: A Case Study in Human-AI Collaborative Curation of Text DatasetsPreprint
Yuan, A., Ippolito, D., Nikolaev, V., Callison-Burch, C., Coenen, A., & Gehrmann, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.06467 · Cited by 9 · see citing works
Synthetic Dataset Generation for Fairer Unfairness Research
Jiang, L., Belitz, C., & Bosch, N. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636868 · Cited by 7 · see citing works
Croskey, P., Offert, F., Jacobs, J., & Thaler, K. M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732058 · Cited by 6 · see citing works
RAI4IoE: Responsible AI for Enabling the Internet of Energy
Xue, M., Nepal, ., Liu, L., Sethuvenkatraman, S., Yuan, X., Rudolph, C., Sun, R., & Eisenhauer, G. (2023) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa58951.2023.00012 · Cited by 5 · see citing works
Bommasani, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.522 · Cited by 4 · see citing works
GPT Deciphering Fedspeak: Quantifying Dissent Among Hawks and Doves
Peskoff, D., Visokay, A., Schulhoff, S., Wachspress, B., Blinder, A. S., & Stewart, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.434 · Cited by 3 · see citing works
Advances in MRI-Based Radiomics for Prostate Cancer
Chaddad, A., Katib, Y., & Tanougast, C. (2023) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi53787.2023.10230328 · Cited by 3 · see citing works
LicenseGPT: A Fine-tuned Foundation Model for Publicly Available Dataset License Compliance
Tan, J., Rajbahadur, G. K., Li, Z., Song, X., Lin, J., Li, D., Zheng, Z., & Hassan, A. E. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728530 · Cited by 2 · see citing works
Analyzing the Impact of Shape & Context on the Face Recognition Performance of Deep Networks
Banerjee, S., Scheirer, W. J., Bowyer, K. W., & Flynn, P. J. (2023) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg57933.2023.10042805 · Cited by 2 · see citing works
Essential work, invisible workers: The role of digital curation in COVID ‐19 Open Science
Pasquetto, I. V., Abdu, A. A., & Chtena, N. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24965 · Cited by 2 · see citing works
The Problem of Zombie Datasets: A Framework For Deprecating Datasets.
Corry, F., Sridharan, H., Luccioni, A. S., Ananny, M., Schultz, J., & Crawford, K. (2021) · Cited by 2 · see citing works
SMAuC - The Scientific Multi-Authorship Corpus
Bevendorff, J., Sauer, P., Gienapp, L., Kircheis, W., Körner, E., Stein, B., & Potthast, M. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00013 · Cited by 2 · see citing works
Can I use this publicly available dataset to build commercial AI software? Most likely notPreprint
Rajbahadur, G. K., Tuck, E., Li, Z., Zhang, W., Lin, D., Chen, B., Ming, Z., Jiang, & Germán, D. M. (2021) · arXiv (Cornell University) · Cited by 2 · see citing works
Ethical considerations for multimodal artificial intelligence in healthcare
Kostick, K. M., Wagner, J. K., Cabrera, L. Y., & Mandl, K. D. (2026) · AI and Ethics · doi:10.1007/s43681-026-01259-0 · Cited by 1 · see citing works
Baeza-Yates, R. A., & Murgai, L. (2023) · Springer eBooks · doi:10.1007/978-3-031-45304-5_28 · Cited by 1 · see citing works
Hupont, I., Tolan, S., Güneş, H., & Gómez, E. (2022) · Research Square · doi:10.21203/rs.3.rs-1306704/v1
Resource papers as registered reports: a proposal
van Miltenburg, E. (2023) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2023.4884
FACE RECOGNITION USING DEEP CONVOLUTIONAL NEURAL NETWORKS: A SURVEY
Semwal, K. A. A. S. (2023) · IADIS International Conferences Interfaces and Human Computer Interaction; Game and Entertainment Technologies; and Computer Graphics, Visualization, Computer Vision and Image Processing (IHCI/GET/CGVCVIP) · doi:10.33965/mccsis2023_202304l018
- Authors
- Peng, K., Mathur, A., & Narayanan, A.
- Affiliations
- Princeton University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2108.02922
- OpenAlex
- W3190472000
- Cited by
- 31
- References
- 71
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 5 · 2023: 10 · 2022: 7 · 2021: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1704444, 1763642)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Keywords
- harm mitigation, face recognition datasets, Labeled Faces in the Wild, privacy, bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
