Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Twitter uses machine learning to crop images, where crops are centered around the part predicted to be the most salient. In fall 2020, Twitter users raised concerns that the automated image cropping system on Twitter favored light-skinned over dark-skinned individuals, as well as concerns that the system favored cropping woman's bodies instead of their heads. In order to address these concerns, we conduct an extensive analysis using formalized group fairness metrics. We find systematic disparities in cropping and identify contributing factors, including the fact that the cropping based on the single most salient point can amplify the disparities because of an effect we term argmax bias. However, we demonstrate that formalized fairness metrics and quantitative analysis on their own are insufficient for capturing the risk of representational harm in automatic cropping. We suggest the removal of saliency-based cropping in favor of a solution that better preserves user agency. For developing a new solution that sufficiently address concerns related to representational harm, our critique motivates a combination of quantitative and qualitative methods that include human-centered design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















![Fig. 6 An image with multiple AI researchers (source: [78]) with multiple almost equally likely salient points. Top: the heat map and the crops based on the first and second most salient points. Bottom: saliency scores of all salient points in the images (top left to bottom right, each row is demarcated by vertical slices). Always cropping based on the top salient point amplifies this crop on the social system compared to the next best salient point.](https://figures.cochat.ai/paper-figures/f1508679af2b4e2286eebee49f4c3c91/image-cropping-on-twitter-fairness-metrics-their-limitations-and-the-importance-figure-6.png)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the algorithmic audit has emerged as one of the most popular approaches to algorithmic accountability [10,66].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All individuals mentioned by multiple respondents have published at least one audit: Cathy O'Neil (mentioned by 9 respondents); Dr. Joy Buolamwini, Dr. Timnit Gebru, and Deborah Raji (6 respondents); Christo Wilson (3 respondents); and Dr. Rumman Chowdhury (2 respondents) [10,28,50,66,67].”
From Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem (2022) · cites this paper as [66]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Black Feminist Thought: Knowledge, Consciousness, and the Politics of Empowerment.
Brewer, R. M., & Collins, P. H. (1992) · Contemporary Sociology · doi:10.2307/2074808 · Cited by 19,445
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“In order to reduce the computational cost to be able to use the model in production, Twitter leverages a combination of fisher pruning [72] and knowledge distillation [43].”
BackgroundCrenshaw, K. (2018) · Feminist Legal Theory · doi:10.4324/9780429500480-5 · Cited by 9,510
Algorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709
Learning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016 · cited 1×
“The model is trained on three publicly available external datasets: Borji and Itti [11], Jiang et al. [46], Judd et al. [47].”
MethodCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 1×
“where R = 1 denotes that the most salient point (the focal point for the crop) is on the person, A represents group status [5, 33], ε is a slack value, and P (R = 1|A = a) is the probability that when two persons, one from a and one from b, are given, the auto-cropping model has the most salience on the person from a.”
BackgroundThe Chicago face database: a free stimulus set of faces and norming data
Ma, D. S., Correll, J., & Wittenbrink, B. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0532-5 · Cited by 1,788
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 2×
“Posing concerns related to image cropping purely in terms of demographic parity fails to question the "closure of predetermined,technical problems" [38] and fails to consider sociotechnical considerations, an example of the "framing trap" [66].”
BackgroundGender and Race: (What) Are They? (What) Do We Want Them To Be?
Haslanger, S. (2000) · Noûs · doi:10.1111/0029-4624.00201 · Cited by 1,564 · cited 1×
“Many critical race scholars also define race in terms of marked or perceived difference, rather than traceable ancestry in itself [42, 44, 65].”
BackgroundJiang, M., Huang, S., Duan, J., & Zhao, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298710 · Cited by 729
Value Sensitive Design and Information Systems
Friedman, B., Kahn, P. H., Borning, A., & Huldtgren, A. (2013) · Philosophy of Engineering and Technology · doi:10.1007/978-94-007-7844-3_4 · Cited by 678 · cited 5×
“These have been long standing topics of research within the HCI community [4, 12, 22, 36, 37, 68].”
BackgroundMethodInfluentialTowards a feminist HCI methodology: social science, feminism, and HCI
Bardzell, S., & Bardzell, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979041 · Cited by 509
Sweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Automatic thumbnail cropping and its effectiveness
Suh, B., Ling, H., Bederson, B. B., & Jacobs, D. W. (2003) · Proceedings of the 16th annual ACM symposium on User interface software and technology · doi:10.1145/964696.964707 · Cited by 421 · cited 1×
“In addition, automated image cropping can help individuals identify important content in the image and do it faster compared to no-crop [62].”
BackgroundUX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391
Next steps for value sensitive design
Borning, A., & Muller, M. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208560 · Cited by 365 · cited 3×
“These have been long standing topics of research within the HCI community [4, 12, 22, 36, 37, 68].”
BackgroundHUMAN-CENTERED VS. USER-CENTERED APPROACHES TO INFORMATION SYSTEM DESIGN
Gasson, S. (2003) · The Journal of Information Technology Theory and Application · Cited by 331
Design Justice: towards an intersectional feminist framework for design theory and practice
Costanza-Chock, S. (2018) · Proceedings of DRS · doi:10.21606/drs.2018.679 · Cited by 325
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295 · cited 4×
“The use of standardized racial and gender taxonomy here is not meant to essentialize or naturalize the construction of race and gender, or to imply that race is solely dependent on ancestry, but rather to study the impact of representational harm on historically marginalized populations [41, 61].”
BackgroundInfluentialHuman Dignity and Human Rights: Thoughts on the Principles of Human-Centered Design
Buchanan, R. (2001) · Design Issues · doi:10.1162/074793601750357178 · Cited by 285
DeepGaze II: Reading fixations from deep features trained on object recognition
Kümmerer, M., Wallis, T. S. A., & Bethge, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01563 · Cited by 236 · cited 1×
“Twitter’s saliency model builds on the deep learning architecture DeepGaze II [51], which leverages feature maps pretrained on object recognition.”
BackgroundContextual encoder–decoder network for visual saliency prediction
Kroner, A., Senden, M., Driessens, K., & Goebel, R. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.05.004 · Cited by 220
CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research
Borji, A., & Itti, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.03581 · Cited by 212 · cited 1×
“The model is trained on three publicly available external datasets: Borji and Itti [11], Jiang et al. [46], Judd et al. [47].”
MethodA sex difference in facial contrast and its exaggeration by cosmetics
Russell, R. (2009) · Perception · doi:10.1068/p6331 · Cited by 206 · cited 2×
“Female heads have higher variability than male heads, which may be party attributed to make-up being more commonly used by females than males [64].”
BackgroundScheuerman, M. K., Wade, K., Lustig, C., & Brubaker, J. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392866 · Cited by 193 · cited 3×
“Many critical race scholars also define race in terms of marked or perceived difference, rather than traceable ancestry in itself [42, 44, 65].”
BackgroundFairness, Equality, and Power in Algorithmic Decision-Making
Kasy, M., & Abebe, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445919 · Cited by 144 · cited 1×
“The fairness framing is often too narrowly scoped as it only considers difference of treatment within the algorithm and fails to incorporate an holistic understanding of the socio-technical system in which the technology is deployed [48, 66].”
BackgroundThe Black/White Binary Paradigm of Race: The "Normal Science" of American Racial Thought
Perea, J. F. (1997) · California Law Review · doi:10.2307/3481059 · Cited by 138
Self-citation is the hallmark of productive authors, of any gender
Mishra, S., Fegley, B. D., Diesner, J., & Torvik, V. I. (2018) · PLOS ONE · doi:10.1371/journal.pone.0195773 · Cited by 126 · cited 1×
“This is an important aspect as bias related work can often be better explained away by more systematic factors in presence of higher quality datasets and more robust analysis (see [54]).”
BackgroundSelf-citationFaster gaze prediction with dense networks and Fisher pruning
Theis, L., Korshunova, I., Tejani, A., & Huszár, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05787 · Cited by 117 · cited 2×
“In order to reduce the computational cost to be able to use the model in production, Twitter leverages a combination of fisher pruning [72] and knowledge distillation [43].”
BackgroundAutomatic Image Cropping: A Computational Complexity Study
Chen, J., Bai, G., Liang, S., & Li, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.61 · Cited by 116 · cited 1×
“The idea of automated image cropping [16, 18] has been present in the industry since at least 1997 [9].”
MethodThe Black/White Binary Paradigm of Race: The Normal Science of American Racial Thought
Perea, J. F. (1997) · bepress Legal Repository · doi:10.15779/z38mf05 · Cited by 111
Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445888 · Cited by 111
Racial categories in machine learning
Benthall, S., & Haynes, B. D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287575 · Cited by 110 · cited 1×
“In addition, a more fundamental critique of racial and gender taxonomies is that they pose a risk of reifying racial and gender categories as natural and inherent rather than socially constructed [7, 34, 41].”
BackgroundQuantitative Analysis of Automatic Image Cropping Algorithms: A Dataset and Comparative Study
Chen, Y., Huang, T.-W., Chang, K.-H., Tsai, Y.-C., Chen, H.-T., & Chen, B. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.32 · Cited by 95
Nash, J. C. (2008) · Social Text · doi:10.1215/01642472-2008-010 · Cited by 81
Assessing Demographic Bias in Named Entity RecognitionPreprint
Mishra, S., He, S., & Belli, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.03415 · Cited by 54 · cited 1×
“[55] report that names from certain demographics are less likely to be identified as person names by major text processing systems.”
BackgroundSelf-citationBrowsing large pictures under limited display sizes
Xie, X., Liu, H., Ma, W., & Zhang, H.-J. (2006) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.876294 · Cited by 47 · cited 1×
“Automated image crops can also produce more desirable thumbnails to users compared to shrinking the whole image to fit the viewport [70].”
BackgroundArdizzone, E., Bruno, A., & Mazzola, G. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41181-6_78 · Cited by 41 · cited 1×
“Saliency is the extent that humans tend to gaze on a given area of the image, and it was proposed as a good proxy to identify the most important part of the image to center a crop around as importance is relative and difficult to quantify [3].”
BackgroundExpression of Concern: Facial feature discovery for ethnicity recognition
Wang, C., Zhang, Q., Liu, W., Liu, Y., & Miao, L. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1278 · Cited by 35
Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future ChallengesPreprint
Borji, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03716 · Cited by 24 · cited 1×
“prediction performance, though there are still gaps in prediction compared to human and limitations such as robustness to noise [10].”
BackgroundQuantitative Analysis of Automatic Image Cropping Algorithms: A Dataset and Comparative StudyPreprint
Chen, Y., Huang, T.-W., Chang, K.-H., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.01480 · Cited by 8 · cited 1×
“The idea of automated image cropping [17, 19] has been present in the industry since at least 1997 [9].”
Method
Research works printed but not linked: 22
- Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. John Wiley & Sons.
- J.D. Biersdorfer. 2017. Fitting Your Photos on Instagram. The New York Times (2017).
- James E Bollman, Ramana L Rao, Dennis L Venable, and Reiner Eschbach. 1999. Automatic image cropping. US Patent 5,978,519.
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 77-91.
- Judith Butler, Fabiana AA Jardim, Jacqueline Moraes Teixeira, and Sebastião Rinaldi. 2020. Endangered/endangering: Schematic racism and white paranoia. Educação e Pesquisa 46 (2020).
- Kate Crawford. 2017. The trouble with bias. In Conference on Neural Information Processing Systems, invited speaker.
- Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does object recognition work for everyone?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52-59.
- Malkia Devich-Cyril. 2020. Defund Facial Recognition. The Atlantic (2020).
- Ravit Dotan and Smitha Milli. 2019. Value-laden disciplinary shifts in machine learning. arXiv preprint arXiv:1912.01172 (2019). · arXiv
- Arturo Escobar. 2018. Designs for the Pluriverse. Duke University Press.
- Karen E Fields and Barbara Jeanne Fields. 2014. Racecraft: The soul of inequality in American life. Verso Trade.
- Barbara L Fredrickson and Tomi-Ann Roberts. 1997. Objectification theory: Toward understanding women's lived experiences and mental health risks. Psychology of women quarterly 21, 2 (1997), 173-206.
- Karamjit S Gill. 1990. Summary of human-centered systems research in Europe. University of Brighton, SEAKE Centre.
- Charles Hirschman. 2004. The origins and demise of the concept of race. Population and development review 30, 3 (2004), 385-415.
- Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. 2015. Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision. 262-270. · doi:10.5555/2919332.2919806
- Catharine A MacKinnon. 1987. Feminism unmodified: Discourses on life and law. Harvard university press.
- Paul Mozur. 2019. One month, 500,000 face scans: How China is using AI to profile a minority. The New York Times 4 (2019).
- Laura Mulvey. 1989. Visual pleasure and narrative cinema. In Visual and other pleasures. Springer, 14-26.
- Arvind Narayanan. 2018. Translation tutorial: 21 fairness definitions and their politics. In Proc. Conf. Fairness Account- ability Transp., New York, USA, Vol. 1170.
- Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. interactions 4, 6 (1997), 42-61.
- Nancy A Van House. 2011. Feminist HCI meets Facebook: Performativity and social networking sites. Interacting with computers 23, 5 (2011), 422-429.
- Qian Yang. 2017. The role of design in creating machine-learning-enhanced user experience. In 2017 AAAI spring symposium series.
Cited by
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
“The authors described this emphasis on the female body as perpetuating the ‘male gaze, a term used for the pervasive depiction of women as sexual objects for the pleasure of and from the perspective heterosexual men’ (Yee et al., 2021), emphasis added.”
BackgroundInfluentialCan Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“Representational harms include instances where certain social groups experience both over- and under-exposure [27], leading to unequal visibility [242].”
BackgroundWho Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533213 · Cited by 277
Metadata
- Authors
- Yee, K., Tantipongpipat, U., & Mishra, S.
- Affiliations
- Twitter (United States) (US)
- Twitter Inc., NYC, NY, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-24
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3163680969
- Cited by
- 59
- References
- 26
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 10 · 2023: 24 · 2022: 11 · 2021: 3
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- Twitter, skin tone bias, representational harm, user agency, human-centered design, algorithmic fairness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.