From Gender Biases to Gender-Inclusive Design: An Empirical Investigation
The DOI matches its Crossref record, with a small difference in title or year; 46 of the 46 references it cites were found in a registry or an index.
Abstract
In recent years, research has revealed gender biases in numerous software products. But although some researchers have found ways to improve gender participation in specific software projects, general methods focus mainly on detecting gender biases -- not fixing them. To help fill this gap, we investigated whether the GenderMag bias detection method can lead directly to designs with fewer gender biases. In our 3-step investigation, two HCI researchers analyzed an industrial software product using GenderMag; we derived design changes to the product using the biases they found; and ran an empirical study of participants using the original product versus the new version. The results showed that using the method in this way did improve the software's inclusiveness: women succeeded more often in the new version than in the original; men's success rates improved too; and the gender gap entirely disappeared.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“When tools misalign with users' cognitive styles [6, 19, 142], they impose additional usability barriers, forcing users to expend extra cognitive effort to achieve the same outcomes.”
From What Needs Attention? Prioritizing Drivers of Developers' Trust and Adoption of Generative AI (2025) · cites this paper as [142]
Citations
Cites
INDIVIDUAL RISK ATTITUDES: MEASUREMENT, DETERMINANTS, AND BEHAVIORAL CONSEQUENCES
Dohmen, T., Falk, A., Huffman, D., Sunde, U., Schupp, J., & Wagner, G. G. (2011) · Journal of the European Economic Association · doi:10.1111/j.1542-4774.2011.01015.x · Cited by 3,896
Strong Evidence for Gender Differences in Risk Taking
Charness, G., & Gneezy, U. (2012) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2011.06.007 · Cited by 1,357
Feminist HCI: taking stock and outlining an agenda for design
Bardzell, S. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753521 · Cited by 1,123
An iterative design methodology for user-friendly natural language office information applications
Kelley, J. F. (1984) · ACM Transactions on Information Systems · doi:10.1145/357417.357420 · Cited by 702
Bias in algorithmic filtering and personalization
Bozdag, E. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9321-6 · Cited by 649
Revisiting gender differences: what we know and what lies ahead.
Meyers‐Levy, J., & Loken, B. (2015) · Journal of Consumer Psychology · doi:10.1016/j.jcps.2014.06.003 · Cited by 614
Storytelling alice motivates middle school girls to learn computer programming
Kelleher, C., Pausch, R., & Kiesler, S. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240844 · Cited by 505
Fairness testing: testing software for discrimination
Galhotra, S., Brun, Y., & Meliou, A. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106277 · Cited by 405 · cited 1×
“Also emerging are research methods to help detect such biases (e.g., [25, 56]).”
BackgroundState of the Art on the Cognitive Walkthrough Method, Its Variants and Evolutions
Mahatody, T., Sagar, M., & Kolski, C. (2010) · International Journal of Human–Computer Interaction · doi:10.1080/10447311003781409 · Cited by 335
GenderMag: A Method for Evaluating Software's Gender Inclusiveness
Burnett, M., Stumpf, S., Macbeth, J., Makri, S., Beckwith, L., Kwan, I., Peters, A., & Jernigan, W. (2016) · Interacting with Computers · doi:10.1093/iwc/iwv046 · Cited by 293
Self-citationBias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr
Hannák, A., Wagner, C., Garcia, D., Mislove, A., Strohmaier, M., & Wilson, C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998327 · Cited by 286 · cited 1×
“Some biases are embedded in data and algorithms that power a software product, out of the sight of regular users (e.g., [8, 30]).”
BackgroundUsing technology in higher education: The influence of gender roles on technology self-efficacy
Huffman, A. H., Whetten, J., & Huffman, W. H. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.02.012 · Cited by 275
LilyPad in the wild: how hardware's long tail is supporting new engineering and design communities
Buechley, L., & Hill, B. M. (2010) · Proceedings of the 8th ACM Conference on Designing Interactive Systems · doi:10.1145/1858171.1858206 · Cited by 272
The LilyPad Arduino: Toward Wearable Engineering for Everyone
Buechley, L., & Eisenberg, M. (2008) · IEEE Pervasive Computing · doi:10.1109/mprv.2008.38 · Cited by 271
Kirkpatrick, K. (2016) · Communications of the ACM · doi:10.1145/2983270 · Cited by 141
Beyond access: broadening participation in high school computer science
Margolis, J., Ryoo, J. J., Sandoval, C. D. M., Lee, C., Goode, J., & Chapman, G. (2012) · ACM Inroads · doi:10.1145/2381083.2381102 · Cited by 131
Personifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125
Learning from IKEA hacking: i'm not one to decoupage a tabletop and call it a day.
Rosner, D., & Bean, J. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518768 · Cited by 122
Open source barriers to entry, revisited: a sociotechnical perspective
Mendez, C., Padala, H. S., Steine-Hanson, Z., Hilderbrand, C., Horvath, A., Hill, C., Simpson, L., Patil, N., Sarma, A., & Burnett, M. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180241 · Cited by 118 · cited 3×
“The GenderMag method [14, 15, 31, 47] used in this paper is in this category.”
BackgroundMethodSelf-citationPrinciples of a debugging-first puzzle game for computing education
Lee, M. J., Bahmani, F., Kwan, I., LaFerte, J., Charters, P., Horvath, A., Luor, F., Cao, J., Law, C., Beswetherick, M., et al. (2014) · 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2014.6883023 · Cited by 117
Self-citationAgencies in Technology Design: Feminist Reconfigurations*
Suchman, L. (2020) · Machine Ethics and Robot Ethics · doi:10.4324/9781003074991-32 · Cited by 115 · cited 1×
“This core of individual differences also supports non-binary notions of gender identification as per prominent feminist literature (e.g., [18, 52, 61]).”
BackgroundEffectiveness of end-user debugging software features: are there gender issues?
Beckwith, L., Burnett, M., Wiedenbeck, S., Cook, C., Sorte, S., & Hastings, M. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055094 · Cited by 108
Self-citationWomen go with the (optical) flow
Tan, D. S., Czerwinski, M. P., & Robertson, G. G. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642649 · Cited by 106
Finding Gender-Inclusiveness Software Issues with GenderMag: A Field Investigation
Burnett, M., Peters, A., Hill, C., & Elarief, N. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858274 · Cited by 101
Self-citationGender-Inclusiveness Personas vs. Stereotyping: Can We Have it Both Ways?
Hill, C. G., Haag, M., Oleson, A., Mendez, C., Marsden, N., Sarma, A., & Burnett, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025609 · Cited by 98
Self-citationGender differences and programming environments: across programming populations
Burnett, M., Fleming, S. D., Iqbal, S., Venolia, G., Rajaram, V., Farooq, U., Grigoreanu, V., & Czerwinski, M. (2010) · Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852824 · Cited by 95
Self-citationGender pluralism in problem-solving software
Burnett, M. M., Beckwith, L., Wiedenbeck, S., Fleming, S. D., Cao, J., Park, T. H., Grigoreanu, V., & Rector, K. (2011) · Interacting with Computers · doi:10.1016/j.intcom.2011.06.004 · Cited by 89
Self-citationSpecialization, homophily, and gender in a social curation site: findings from pinterest
Chang, S., Kumar, V., Gilbert, E., & Terveen, L. G. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531660 · Cited by 86
Gender HCI: What About the Software?
Beckwith, L., Burnett, M., Grigoreanu, V., & Wiedenbeck, S. (2006) · Computer · doi:10.1109/mc.2006.382 · Cited by 82
Self-citationWhy-oriented end-user debugging of naive Bayes text classification
Kulesza, T., Stumpf, S., Wong, W.-K., Burnett, M. M., Perona, S., Ko, A. J., & Oberst, I. (2011) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2030365.2030367 · Cited by 71
Self-citationEvaluation of Filesystem Provenance Visualization Tools
Borkin, M. A., Yeh, C. S., Boyd, M., Macko, P., Gajos, K. Z., Seltzer, M., & Pfister, H. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.155 · Cited by 63
End-user debugging strategies: A sensemaking perspective
Grigoreanu, V., Burnett, M., Wiedenbeck, S., Cao, J., Rector, K., & Kwan, I. (2012) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2147783.2147788 · Cited by 60 · cited 2×
“Research spanning numerous software domains has shown that today’s software mostly supports only the styles favored by men—for example in spreadsheets [4, 5, 6, 26, 28, 32, 60], visualization systems [7, 62], online classwork platforms [50], web and home automation [19, 51, 52, 55], intelligent agents and robots [41,…”
BackgroundSelf-citationA Debugging Perspective on End-User Mashup Programming
Cao, J., Rector, K., Park, T. H., Fleming, S. D., Burnett, M., & Wiedenbeck, S. (2010) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2010.29 · Cited by 59
Self-citationTesting vs. code inspection vs. what else?: male and female end users' debugging strategies
Subrahmaniyan, N., Beckwith, L., Grigoreanu, V., Burnett, M., Wiedenbeck, S., Narayanan, V., Bucht, K., Drummond, R., & Fern, X. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357153 · Cited by 57
Self-citationPutting the gender back in digital housekeeping
Rode, J. A., & Poole, E. S. (2018) · Proceedings of the 4th Conference on Gender & IT - GenderIT '18 · doi:10.1145/3196839.3196845 · Cited by 56
Are you sure your software is gender-neutral?
Williams, G. (2014) · interactions · doi:10.1145/2524808 · Cited by 51
Can feature design reduce the gender gap in end-user software development environments?
Grigoreanu, V., Cao, J., Kulesza, T., Bogart, C., Rector, K., Burnett, M., & Wiedenbeck, S. (2008) · 2008 IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2008.4639077 · Cited by 49
Self-citationCognitive walkthrough of a learning management system with gendered personas
Shekhar, A., & Marsden, N. (2018) · Proceedings of the 4th Conference on Gender & IT - GenderIT '18 · doi:10.1145/3196839.3196869 · Cited by 45
Role of gender, self-efficacy, anxiety and testing formats in learning spreadsheets
Singh, A., Bhadauria, V. S., Jain, A., & Gurung, A. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2012.11.009 · Cited by 45
"Girls don't waste time": pre-adolescent attitudes toward ICT
Hou, W., Kaur, M., Komlódi, A., Lutters, W. G., Boot, L., Cotten, S. R., Morrell, C., Ozok, A. A., & Tüfekçi, Z. (2006) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1125451.1125622 · Cited by 39
Arcand, M., & Nantel, J. A. (2012) · Journal of theoretical and applied electronic commerce research · doi:10.4067/s0718-18762012000200010 · Cited by 37
A strategy-centric approach to the design of end-user debugging tools
Grigoreanu, V. I., Burnett, M. M., & Robertson, G. G. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753431 · Cited by 31
Self-citationA principled evaluation for a principled idea garden
Jernigan, W., Horvath, A., Lee, M., Burnett, M., Cuilty, T., Kuttal, S., Peters, A., Kwan, I., Bahmani, F., & Ko, A. (2015) · 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357222 · Cited by 28 · cited 1×
“Its gender inclusiveness comes from innovating certain programming environment characteristics, such as: portraying the computer as fallible, personifying error messages, and presenting explanatory help in forms compatible with both women’s tendencies toward comprehensive information processing and men’s tendencies to…”
BackgroundSelf-citationShowkat, D., & Grimm, C. M. (2018) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/urai.2018.8441766 · Cited by 25
Design Planning in End-User Web Development: Gender, Feature Exploration and Feelings of Success
Rosson, M. B., Sinha, H., & Edor, T. (2010) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2010.28 · Cited by 12
Gender and technology in free play in Swedish early childhood education
Hällström, J., Elvstrand, H., & Hellberg, K. (2014) · International Journal of Technology and Design Education · doi:10.1007/s10798-014-9274-z
Research works printed but not linked: 17
- 10.4135/9781446221129.n6 · DOI · Paper page
- 10.1109/VLHCC.2007.15 · doi:10.1109/vlhcc.2007.15
- Adam Bradley, Cayley MacArthur, Mark Hancock, and Sheelagh Carpendale. 2015. Gendered or neutral? Considering the language of HCI. In Proceedings of the 41st Graphics Interface Conference (GI'15), 163--170.
- Margaret Burnett, Robin Counts, Ronette Lawrence, and Hannah Hanson. 2017. Gender HCI and Microsoft: Highlights from a longitudinal study. IEEE VLHCC, 139--143.
- Margaret Burnett, Simone Stumpf, Laura Beckwith, and Anicia Peters. 2018. The GenderMag Kit: How to Use the GenderMag Method to Find Inclusiveness Issues through a Gender Lens, http://gendermag.org, June 28, 2018.
- Judith Butler. 1999. Gender Trouble: Feminism and the Subversion of Identity. Routledge.
- Sally J. Cunningham, Annika Hinze, and David Nicols. 2016. "Supporting gender-neutral digital library creation A case study using the GenderMag Toolkit," Knowledge, Information, and Data in An Open Access Society, vol. 10075, pp. 45--50, 2016.
- FAT*, 2018. ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*). Retrieved Sept. 11, 2018 from https://fatconference.org/
- IFIP News, 2018. ACM President Cherri Pancake Highlights Importance of Diversity in ICT, July 2018. Retrieved Sept. 11, 2018 from https://www.ifipnews.org/acm-president-cherri-pancakehighlights-importance-diversity-ict/
- 10.5555/2670296.2670301 · doi:10.5555/2670296.2670301
- Caitlin Kelleher. 2009. Barriers to programming engagement. Advances in Gender and Education 1, 1: 5- 10.
- 10.1145/2445196.2445202 · DOI · Paper page
- Jennifer Ann Rode. 2008. An ethnographic examination of the relationship of gender & end-user programming. Ph.D Thesis. University of California Irvine Irvine CA.
- René Riedl Marco Hubert and Peter Kenning. 2010. Are There Neural Gender Differences in Online Trust? An fMRI Study on the Perceived Trustworthiness of eBay Offers MIS Quarterly (34: 2) 397428. · doi:10.5555/2017458.2017469
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Data and Discrimination: Converting Critical Concerns into Productive Inquiry (a preconference at the 64th Annual Meeting of the International Communication Association).
- Verdict. 2018. Ethics, GDPR, and diversity: Verdict talks to the president of the Association for Computing Machinery, July 2018. Retrieved Sept. 11, 2018 from https://www.verdict.co.uk/associationfor-computing-machinery-ethics/
- 10.5555/189200.189214 · doi:10.5555/189200.189214
Cited by
Balayn, A., Lofi, C., & Houben, G.-J. (2021) · The VLDB Journal · doi:10.1007/s00778-021-00671-8 · Cited by 98
Break the Loop: Gender Imbalance in Music Recommenders
Ferraro, A., Serra, X., & Bauer, C. (2021) · Proceedings of the 2021 Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446033 · Cited by 92
“Studies have shown that the design of software [6, 7, 9, 50] or websites [33] may introduce bias for users of different genders.”
BackgroundKuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445659 · Cited by 44
What Needs Attention? Prioritizing Drivers of Developers' Trust and Adoption of Generative AI
Choudhuri, R., Trinkenreich, B., Pandita, R., Kalliamvakou, E., Steinmacher, I., Gerosa, M., Sanchez, C., & Sarma, A. (2025) · arXiv · doi:10.48550/arxiv.2505.17418 · Cited by 14
Kirks-Cler, D. N., Safford, S., & Marathe, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757680 · Cited by 4
“To remedy these harms, researchers have proposed inclusive design techniques to represent gender diverse people in individual systems [47, 48].”
Metadata
- Authors
- Vorvoreanu, M., Zhang, L., Huang, Y.-H., Hilderbrand, C., Steine-Hanson, Z., & Burnett, M.
- Affiliations
- Microsoft Research & Purdue University, Redmond, WA, USA
- Purdue University, West Lafayette, IN, USA
- Oregon State University, Corvallis, OR, USA
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-14
- Cited by
- 113
- References
- 46
- Funders
- NSF (1528061)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.