The pushback effects of race, ethnicity, gender, and age in code review
Open accessAbstract
Research shows that White, male, and younger engineers receive less pushback than those in other demographics.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Code review can be carried out by an independent experienced coder or auditor for bias detection.
“This starts with a code review that can be carried out by an independent experienced coder/auditor.49”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Role congruity theory of prejudice toward female leaders.
Eagly, A. H., & Karau, S. J. (2002) · Psychological Review · doi:10.1037//0033-295x.109.3.573 · Cited by 7,300 · see citing works
Generalized Collinearity Diagnostics
Fox, J., & Monette, G. A. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10475190 · Cited by 1,916 · see citing works
Expectations of brilliance underlie gender distributions across academic disciplines
Leslie, S.-J., Cimpian, A., Meyer, M., & Freeland, E. (2015) · Science · doi:10.1126/science.1261375 · Cited by 1,516 · see citing works
Age Stereotypes in the Workplace: Common Stereotypes, Moderators, and Future Research Directions†
Posthuma, R. A., & Campion, M. A. (2007) · Journal of Management · doi:10.1177/0149206308318617 · Cited by 1,101 · see citing works
Smith, T. (2021) · The Encyclopedia of Research Methods in Criminology and Criminal Justice · doi:10.1002/9781119111931.ch11 · Cited by 687 · see citing works
Expectations, outcomes, and challenges of modern code review
Bacchelli, A., & Bird, C. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606617 · Cited by 650 · see citing works
The Protege's Perspective Regarding Negative Mentoring Experiences: The Development of a Taxonomy
Eby, L. T., McManus, S. E., Simon, S. A., & Russell, J. E. A. (2000) · Journal of Vocational Behavior · doi:10.1006/jvbe.1999.1726 · Cited by 450 · see citing works
Gender differences and bias in open source: pull request acceptance of women versus men
Terrell, J., Kofink, A., Middleton, J., Rainear, C., Murphy-Hill, E., Parnin, C., & Stallings, J. (2017) · PeerJ Computer Science · doi:10.7717/peerj-cs.111 · Cited by 287 · cited 1× · see citing works
“Prior work 16 suggests that the relationship between author and reviewer moderates gender bias effects.”
BackgroundSelf-citationSadowski, C., Söderberg, E., Church, L., Sipko, M., & Bacchelli, A. (2018) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3183519.3183525 · Cited by 280 · see citing works
Wait for It: Determinants of Pull Request Evaluation Latency on GitHub
Yu, Y., Wang, H., Filkov, V., Dévanbu, P., & Vasilescu, B. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.42 · Cited by 197 · see citing works
Why Google stores billions of lines of code in a single repository
Potvin, R., & Levenberg, J. (2016) · Communications of the ACM · doi:10.1145/2854146 · Cited by 158 · see citing works
Occupational Stereotyping of Asian Americans
Leong, F. T. L., & Hayes, T. J. (1990) · The Career Development Quarterly · doi:10.1002/j.2161-0045.1990.tb00835.x · Cited by 74 · see citing works
What distinguishes great software engineers
Li, P. L., Ko, A. J., & Begel, A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09773-y · Cited by 48 · see citing works
Mozilla plan seeks to debug scientific code
Hayden, E. C. (2013) · Nature · doi:10.1038/501472a · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Egelman, C.D., Murphy-Hill, E., Kammer, E., Hodges, M.M., Green, C., Jaspan, C., and Lin, J. Pushback: Characterizing and detecting negative interpersonal interactions in code review. Intern. Conf. on Software Engineering (2020), 174-185.
- Murphy-Hill, E., Dicker, J., Hodges, M., Egelman, C.D., Jaspan, C.N.C., Cheng, L., Kammer, L., Holtz, B., Jorde, M.A., Dolan, A.M.K., and Green, C. Engineering impacts of anonymous author code review: A field experiment. Trans. on Software Engineering. (To appear).
- Nadri, R., Rodriguez-Perez, G., and Nagappan, M. On the relationship between the developer's perceptible race and ethnicity and the evaluation of contributions in OSS. Trans. on Software Engineering. (To appear).
Cited by
Sorted by citations
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
“49 Such a code review can be also performed retrospectively by the scientific community for manuscripts published with open-access code.”
Dagan, E., Sarma, A., Chang, A., D’Angelo, S., Dicker, J., & Murphy-Hill, E. R. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616273 · Cited by 23 · see citing works
Self-citationImpostor Phenomenon in Software Engineers
Guenes, P., Tomaz, R., Kalinowski, M., Baldassarre, M. T., & Storey, M. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1145/3639475.3640114 · Cited by 22 · see citing works
Diversity’s Double-Edged Sword: Analyzing Race’s Effect on Remote Pair Programming Interactions
Mason, S. A., & Kuttal, S. K. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3699601 · Cited by 18 · see citing works
Hyrynsalmi, S. M., Baltes, S., Brown, C., Prikladnicki, R., Rodríguez-Pérez, G., Serebrenik, A., Simmonds, J., Trinkenreich, B., Wang, Y., & Liebel, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3711904 · Cited by 17 · see citing works
A Human-Centered Approach to Developer Productivity
Jaspan, C. N. C., & Green, C. (2022) · IEEE Software · doi:10.1109/ms.2022.3212165 · Cited by 10 · see citing works
Self-citationHicks, C. M., Lee, C. S., & Foster-Marks, K. L. (2024) · doi:10.31234/osf.io/2gej5 · Cited by 6 · see citing works
Understanding and effectively mitigating code review anxiety
Lee, C. S., & Hicks, C. M. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10550-9 · Cited by 5 · see citing works
Liang, A., Murphy-Hill, E. R., Weimer, W., & Huang, Y. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3437355 · Cited by 6 · see citing works
Self-citationDevelopers Foraging Behavior in Code Hosting Sites: A Gender Perspective
Sedhain, A., Leon, S., Raasch, R., & Kuttal, S. K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35129-7_42 · Cited by 4 · see citing works
Diversity in issue assignment: humans vs bots
Murali, A., Sahu, G., Thangarajah, K., Zimmerman, B. D., Rodríguez-Pérez, G., & Nagappan, M. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10424-6 · Cited by 4 · see citing works
Exploring ChatGPT for identifying sexism in the communication of software developers
Sultana, S., & Kali, M. B. (2024) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3652037.3663918 · Cited by 3 · see citing works
Did Gerrit’s Respectful Code Review Reminders Reduce Comment Toxicity?
Murphy-Hill, E. R., Dicker, J., Carlson, D., Harbach, M., Murillo, A., & Zhou, T. (2024) · Apress eBooks · doi:10.1007/978-1-4842-9651-6_18 · Cited by 3 · see citing works
Self-citationComparing Foraging Behavior Across Code Hosting and Q&A Platforms Through a Gender Lens
Leon, S., Tamanna, M., & Kuttal, S. K. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00040 · Cited by 2 · see citing works
From Overload to Insight: Bridging Code Search and Code Review with LLMs
Rao, N., Vasilescu, B., & Holmes, R. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728518 · Cited by 2 · see citing works
Hicks, C. M. (2024) · doi:10.31234/osf.io/qz43x · Cited by 2 · see citing works
Are We on the Same Page? Examining Developer Perception Alignment in Open Source Code Reviews
Alebachew, Y. B., Ko, M., & Brown, C. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756929 · Cited by 1 · see citing works
ToxiShield: Promoting Inclusive Developer Communication through Real-Time Toxicity Filtering
Anindya, M. A. A., Biswas, S., Iqbal, A., Sarker, J., & Bosu, A. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808130 · Cited by 1 · see citing works
Dynamics of Gender Bias within Computer Science
Misa, T. J. (2024) · Information & Culture · doi:10.7560/ic59203 · Cited by 1 · see citing works
Effective Teaching through Code Reviews: Patterns and Anti-patterns
Sarma, A., & Chen, N. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660764 · Cited by 1 · see citing works
TWIST: Programa de Capacitação de Desenvolvedoras de Software
Rodriguez, N. D. L. R., de Lima, M. J., de Moura, A. L., & Szenberg, R. (2024) · Anais do Women in Information Technology (WIT) · doi:10.5753/wit.2024.2647 · Cited by 1 · see citing works
Perceptions of Software Developer Inclusion: A Survey at Google
Hodges, M., & Murphy-Hill, E. R. (2024) · Apress eBooks · doi:10.1007/978-1-4842-9651-6_13 · Cited by 1 · see citing works
Self-citationTowards An Integrated Study of Human Cognition in IT at Deep AI Era
Chen, Z. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.11.223
Experiences Implementing and Deploying Anonymous Code Review
Dicker, J., Murphy-Hill, E. R., & Sarma, A. (2024) · Apress eBooks · doi:10.1007/978-1-4842-9651-6_19
Self-citationExploring disparities in open source contribution experiences through the diversity lens of race
Dandamudi, R., Rodríguez-Pérez, G., Nagappan, M., Alegria, S. N., Ventura, F., & Coronel, F. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113030
Özdemir, Ö., Aydoğan, R., & Sözer, H. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112791
Improving Artificial Intelligence Tools in Education Towards Equity and Legality
Parland, M., & Shcherbakov, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07319-8_9
From Artifacts to Strategies: A Gendered Lens on Web Information Foraging
Kuttal, S. K., Sthapit, S., Leon, S., Phillips, R., & Rahal, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29583-5_4
Who’s Left Out? A Case Study on Promoting Equitable Participation in Remote Collaboration Software
Mason, S. A., Meuser, N., Si, A., Sthapit, S., Leon, S., Teke, M. K., & Kuttal, S. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29593-4_32
- Authors
- Murphy-Hill, E., Jaspan, C., Egelman, C., & Cheng, L.
- Affiliations
- Google (United States) (US)
- Venue
- Communications of the ACM
- Published
- 2022-02-23
- Volume
- 65
- Issue
- 3
- Pages
- 52-57
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W4212872938
- Cited by
- 30
- References
- 14
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 16 · 2023: 3 · 2022: 2
- FWCI
- 5.94
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Safety Research › Career Development and Diversity
- Keywords
- code review, pushbacks, race, ethnicity, gender, age, demographic bias, software engineering, diversity in computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
