Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
Open accessAbstract
In this work, we introduce a novel metric for auditing group fairness in ranked lists. Our approach offers two benefits compared to the state of the art. First, we offer a blueprint for modeling of user attention. Rather than assuming a logarithmic loss in importance as a function of the rank, we can account for varying user behaviors through parametrization. For example, we expect a user to see more items during a viewing of a social media feed than when they inspect the results list of a single web search query. Second, we allow non-binary protected attributes to enable investigating inherently continuous attributes (e.g., political alignment on the liberal to conservative spectrum) as well as to facilitate measurements across aggregated sets of search results, rather than separately for each result list. By combining these two elements into our metric, we are able to better address the human factors inherent in this problem. We measure the whole sociotechnical system, consisting of a ranking algorithm and individuals using it, instead of exclusively focusing on the ranking algorithm. Finally, we use our metric to perform three simulated fairness audits. We show that determining fairness of a ranked output necessitates knowledge (or a model) of the end-users of the particular service. Depending on their attention distribution function, a fixed ranking of results can appear biased both in favor and against a protected group1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Facebook provides a relevance score between 1 and 10, calculated based on expected positive and negative feedback from the target audience.
“Facebook provides advertisers with an overview of how well-matched it believes an ad is with the target audience using a metric called relevance score, which ranges between 1 and 10.Facebook says [68]: Relevance score is calculated based on the positive and negative feedback we expect an ad to receive from its target audience.”
From Discrimination through Optimization (2019) · cites this paper as [68]
Facebook's ad delivery system is designed to show the right content to the right people, and a high relevance score is seen by the system as a positive signal.
“Facebook goes on to say [68]: Put simply, the higher an ad's relevance score, the less it will cost to be delivered.This is because our ad delivery system is designed to show the right content to the right people, and a high relevance score is seen by the system as a positive signal.”
From Discrimination through Optimization (2019) · cites this paper as [68]
Cites
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
The serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717 · cited 1× · see citing works
“A second challenge is taking order effects [40] into account, i.e., the well-established tendency of human beings to place more attention and emphasis on items at the top 1 of a ranked list.”
BackgroundPredicting clicks: estimating the click-through rate for new ads
Richardson, M., Dominowska, E., & Ragno, R. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242643 · Cited by 1,049 · see citing works
An experimental comparison of click position-bias models
Craswell, N., Zoeter, O. R., Taylor, M. J. H., & Ramsey, B. (2008) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1341531.1341545 · Cited by 947 · see citing works
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · see citing works
Eye-tracking analysis of user behavior in WWW search
Granka, L. A., Joachims, T., & Gay, G. K. (2004) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1008992.1009079 · Cited by 701 · see citing works
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694 · see citing works
Discrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676 · see citing works
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · see citing works
Fairness of Exposure in Rankings
Singh, A., & Joachims, T. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220088 · Cited by 607 · see citing works
The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections
Epstein, R., & Robertson, R. E. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1419828112 · Cited by 578 · see citing works
Unbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531 · see citing works
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · see citing works
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · see citing works
Fairness-aware Learning through Regularization Approach
Kamishima, T., Akaho, S., & Sakuma, J. (2011) · 2011 IEEE 11th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2011.83 · Cited by 475 · see citing works
Building Classifiers with Independency Constraints
Calders, T., Kamiran, F., & Pechenizkiy, M. (2009) · 2009 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2009.83 · Cited by 468 · see citing works
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · see citing works
Measuring Fairness in Ranked Outputs
Yang, K., & Stoyanovich, J. (2017) · Proceedings of the 29th International Conference on Scientific and Statistical Database Management · doi:10.1145/3085504.3085526 · Cited by 378 · see citing works
Equity of Attention: Amortizing Individual Fairness in Rankings
Biega, A. J., Gummadi, K. P., & Weikum, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210063 · Cited by 337 · see citing works
A Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312 · see citing works
Discrimination Aware Decision Tree Learning
Kamiran, F., Calders, T., & Pechenizkiy, M. (2010) · 2010 IEEE International Conference on Data Mining · doi:10.1109/icdm.2010.50 · Cited by 303 · see citing works
Decision Theory for Discrimination-Aware Classification
Kamiran, F., Karim, A., & Zhang, X. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.45 · Cited by 300 · see citing works
Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr
Hannák, A., Wagner, C., Garcia, D., Mislove, A., Strohmaier, M., & Wilson, C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998327 · Cited by 286 · cited 2× · see citing works
“ch presented results that were stereotypically gendered [34], while Hannak et al. and Chen et al. showed that search engines on employment websites were not group fair with respect to race and gender [9, 23]. Audit studies have also examined the political partisanship of search results from Twitter and Google [12, 38,…”
BackgroundSelf-citationDisparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · see citing works
k-NN as an implementation of situation testing for discrimination discovery and prevention
Luong, B. T., Ruggieri, S., & Turini, F. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020488 · Cited by 263 · see citing works
An eye tracking study of the effect of target rank on web search
Guan, Z., & Cutrell, E. B. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240691 · Cited by 243 · see citing works
Ranking with Fairness ConstraintsPreprint
Celis, L. E., Straszak, D., & Vishnoi, N. K. (2017) · arXiv (Cornell University) · doi:10.4230/lipics.icalp.2018.28 · Cited by 237 · see citing works
Auditing Partisan Audience Bias within Google Search
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274417 · Cited by 209 · see citing works
Self-citationAre people biased in their use of search engines?
Keane, M. T., O’Brien, M., & Smyth, B. (2008) · Communications of the ACM · doi:10.1145/1314215.1314224 · Cited by 170 · see citing works
Controlling Attribute Effect in Linear Regression
Calders, T., Karim, A., Kamiran, F., Ali, W., & Zhang, X.-L. (2013) · 2013 IEEE 13th International Conference on Data Mining · doi:10.1109/icdm.2013.114 · Cited by 162 · see citing works
Primacy and Recency Effects on Clicking Behavior
Murphy, J., Hofacker, C. F., & Mizerski, R. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2006.00025.x · Cited by 157 · see citing works
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · see citing works
Self-citationQuantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998321 · Cited by 148 · see citing works
Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social MediaPreprint
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01347 · Cited by 134 · see citing works
Handling Conditional Discrimination
liobaite, I., Kamiran, F., & Calders, T. (2011) · 2011 IEEE 11th International Conference on Data Mining · doi:10.1109/icdm.2011.72 · Cited by 125 · see citing works
Suppressing the Search Engine Manipulation Effect (SEME)
Epstein, R., Robertson, R. E., Lazer, D. M. J., & Wilson, C. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134677 · Cited by 90 · see citing works
Self-citationWhat Makes a Link Successful on Wikipedia?
Dimitrov, D., Singer, P., Lemmerich, F., & Strohmaier, M. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052613 · Cited by 39 · see citing works
Fairness in Online Social Network Timelines
Hargreaves, E. M., Agosti, C., Menasché, D., Neglia, G., Reiffers-Masson, A., & Altman, E. (2018) · Performance evaluation (Print) · doi:10.1016/j.peva.2018.09.009 · Cited by 24 · see citing works
A discrete truncated Pareto distribution
Kozubowski, T. J., Panorska, A. K., & Forister, M. L. (2015) · Statistical Methodology · doi:10.1016/j.stamet.2015.04.002 · Cited by 23 · see citing works
FATREC Workshop on Responsible Recommendation
Ekstrand, M. D., & Sharma, A. K. (2017) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3109859.3109960 · Cited by 18 · see citing works
Fairness in Online Social Network Timelines
Hargreaves, E. M., Agosti, C., Menasché, D. S., Neglia, G., Reiffers-Masson, A., & AltmanEitan (2019) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/3308897.3308928 · Cited by 6 · see citing works
Flagged first, then by citations
2 of the 46 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Evaluating Recommender Systems: Survey and Framework
Zangerle, E., & Bauer, C. (2022) · ACM Computing Surveys · doi:10.1145/3556536 · Cited by 288 · see citing works
Discrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268 · see citing works
Self-citationEvaluating Stochastic Rankings with Expected Exposure
Díaz, F., Mitra, B., Ekstrand, M. D., Biega, A. J., & Carterette, B. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411962 · Cited by 175 · see citing works
Building Human Values into Recommender Systems: An Interdisciplinary Synthesis
Stray, J., Halevy, A. Y., Assar, P., Hadfield-Menell, D., Boutilier, C. E., Ashar, A., Bakalar, C., Beattie, L., Ekstrand, M. D., Leibowicz, C., et al. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3632297 · Cited by 145 · see citing works
Exploring author gender in book rating and recommendation
Ekstrand, M. D., Tian, M., Kazi, M. I. R., Mehrpouyan, H., & Kluver, D. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3240373 · Cited by 110 · see citing works
Minkkinen, M., Laine, J., & Mäntymäki, M. (2022) · Digital Society · doi:10.1007/s44206-022-00022-2 · Cited by 111 · see citing works
Interventions for ranking in the presence of implicit bias
Celis, L. E., Mehrotra, A., & Vishnoi, N. K. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372858 · Cited by 77 · see citing works
Measuring Fairness in Ranked Results
Raj, A., & Ekstrand, M. D. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532018 · Cited by 74 · see citing works
Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · arXiv · doi:10.48550/arxiv.1904.02095 · Cited by 72 · see citing works
Self-citationExploring author gender in book rating and recommendation
Ekstrand, M. D., & Kluver, D. (2021) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-020-09284-2 · Cited by 66 · see citing works
Interplay between upsampling and regularization for provider fairness in recommender systems
Boratto, L., Fenu, G., & Marras, M. (2021) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-021-09294-8 · Cited by 61 · see citing works
Evaluation metrics for measuring bias in search engine results
Gezi̇ci, G., Lipani, A., Saygın, Y., & Yılmaz, E. (2021) · Information Retrieval · doi:10.1007/s10791-020-09386-w · Cited by 45 · see citing works
Enabling cross-continent provider fairness in educational recommender systems
Gómez, E., Zhang, C. S., Boratto, L., Salamó, M., & Ramos, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.08.025 · Cited by 44 · see citing works
Provider fairness across continents in collaborative recommender systems
Gómez, E., Boratto, L., & Salamó, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2021.102719 · Cited by 39 · see citing works
Draws, T., Tintarev, N., Gadiraju, U., Bozzon, A., & Timmermans, B. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462851 · Cited by 38 · see citing works
Fairness in Recommender Systems
Ekstrand, M. D., Das, A., Burke, R., & Díaz, F. (2012) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_18 · Cited by 33 · see citing works
Characterizing Intersectional Group Fairness with Worst-Case ComparisonsPreprint
Ghosh, A., Genuit, L., & Reagan, M. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.01673 · Cited by 27 · see citing works
Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
Draws, T., Tintarev, N., Gadiraju, U., Bozzon, A., & Timmermans, B. (2020) · SIGKDD Explorations · doi:10.1145/3468507.3468515 · Cited by 36 · see citing works
The multisided complexity of fairness in recommender systems
Sonboli, N., Burke, R., Ekstrand, M. D., & Mehrotra, R. (2022) · AI Magazine · doi:10.1002/aaai.12054 · Cited by 24 · see citing works
Smith, J. J., Beattie, L., & Cramer, H. S. M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583204 · Cited by 22 · see citing works
Understanding and Mitigating the Effect of Outliers in Fair Ranking
Sarvi, F., Heuss, M., Aliannejadi, M., Schelter, S., & de Rijke, M. (2021) · Web Search and Data Mining · doi:10.1145/3488560.3498441 · Cited by 28 · see citing works
<scp>FAIR</scp> : Fairness‐aware information retrieval evaluation
Gao, R., Ge, Y., & Shah, C. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24648 · Cited by 19 · see citing works
Distributionally-Informed Recommender System Evaluation
Ekstrand, M. D., Carterette, B., & Díaz, F. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3613455 · Cited by 19 · see citing works
Invisible Relevance Bias: Text-Image Retrieval Models Prefer AI-Generated Images
Xu, S., Hou, D., Pang, L., Deng, J., Xu, J., Shen, H., & Cheng, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657750 · Cited by 24 · see citing works
Search results diversification for effective fair ranking in academic search
McDonald, G., Macdonald, C., & Ounis, I. (2021) · Information Retrieval · doi:10.1007/s10791-021-09399-z · Cited by 17 · see citing works
Online Post-Processing in Rankings for Fair Utility Maximization
Gupta, A., Johnson, E., Payan, J., Roy, A. K., Kobren, A., Panda, S., Tristan, J.-B., & Wick, M. (2021) · Proceedings of the 14th ACM International Conference on Web Search and Data Mining · doi:10.1145/3437963.3441724 · Cited by 16 · see citing works
Fairness-Aware Exposure Allocation via Adaptive Reranking
Jaenich, T., McDonald, G., & Ounis, I. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657794 · Cited by 14 · see citing works
Ferraro, A., Ekstrand, M. D., & Bauer, C. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688163 · Cited by 14 · see citing works
Disparate Impact in Item Recommendation: A Case of Geographic Imbalance
Gómez, E., Boratto, L., & Salamó, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_13 · Cited by 14 · see citing works
Matchmaking Under Fairness Constraints: A Speed Dating Case Study
Paraschakis, D., & Nilsson, B. J. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-52485-2_5 · Cited by 13 · see citing works
Yi, F., Singh, A., & Tao, Z. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000101 · Cited by 12 · see citing works
Emergence of network effects and predictability in the judicial system
Mones, E., Sapieżyński, P., Thordal, S., Olsen, H. P., & Lehmann, S. (2021) · Scientific Reports · doi:10.1038/s41598-021-82430-x · Cited by 12 · see citing works
Comparing Fair Ranking MetricsPreprint
Raj, A., & Ekstrand, M. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01311 · Cited by 12 · see citing works
A Versatile Framework for Evaluating Ranked Lists in Terms of Group Fairness and Relevance
Sakai, T., Kim, J. Y., & Kang, I.-H. (2023) · ACM Transactions on Information Systems · doi:10.1145/3589763 · Cited by 11 · see citing works
Robust reputation independence in ranking systems for multiple sensitive attributes
Ramos, G., Boratto, L., & Marras, M. (2022) · Machine Learning · doi:10.1007/s10994-022-06173-0 · Cited by 10 · see citing works
A GA-based algorithm meets the fair ranking problem
Tahery, S., Aftabi, S. Z., & Farzi, S. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102711 · Cited by 10 · see citing works
Subverting Fair Image Search with Generative Adversarial Perturbations
Ghosh, A., Jagielski, M., & Wilson, C. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533128 · Cited by 10 · see citing works
Self-citationTowards QoS-Aware RecommendationsPreprint
Sermpezis, P., Kastanakis, S., Pinheiro, J. I., de Assis, F. M. F., Nogueira, M., Menasché, D. S., & Spyropoulos, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06392 · Cited by 10 · see citing works
Unmasking Gender Bias in Recommendation Systems and Enhancing Category-Aware Fairness
Kheya, T. A., Bouadjenek, M. R., & Aryal, S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714528 · Cited by 10 · see citing works
The Role of Relevance in Fair Ranking
Balagopalan, A., Jacobs, A. Z., & Biega, A. J. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591933 · Cited by 9 · see citing works
Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking
Vardasbi, A., Sarvi, F., & de Rijke, M. (2022) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3532045 · Cited by 12 · see citing works
Measuring Item Fairness in Next Basket Recommendation: A Reproducibility Study
Liu, Y., Li, M., Ariannezhad, M., Mansoury, M., Aliannejadi, M., & de Rijke, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_18 · Cited by 8 · see citing works
Zhao, Y., Wang, Y., Zhang, Y., Wiśniewski, P., Aggarwal, C. C., & Derr, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30263 · Cited by 8 · see citing works
On the trade-off between ranking effectiveness and fairness
Melucci, M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122709 · Cited by 7 · see citing works
Fairness and Diversity for Rankings in Two-Sided Markets.Preprint
Wang, L., & Joachims, T. (2020) · arXiv (Cornell University) · Cited by 7 · see citing works
Rodriguez, J., & Piccoli, G. (2024) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2024.101456 · Cited by 6 · see citing works
Measuring Bias in a Ranked List Using Term-Based Representations
Abolghasemi, A., Azzopardi, L. A., Askari, A., de Rijke, M., & Verberne, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_1 · Cited by 6 · see citing works
Properties of Group Fairness Measures for Rankings
Schumacher, T., Lutz, M., Sikdar, S., & Strohmaier, M. (2022) · ACM Transactions on Social Computing · doi:10.1145/3674883 · Cited by 9 · see citing works
Enhancing recommender systems with provider fairness through preference distribution-awareness
Gómez, E., Contreras, D., Boratto, L., & Salamó, M. (2025) · Int. J. Inf. Manag. Data Insights · doi:10.1016/j.jjimei.2024.100311 · Cited by 8 · see citing works
Fairness of Machine Learning in Search Engines
Fang, Y., Liu, H., Tao, Z., & Yurochkin, M. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557501 · Cited by 6 · see citing works
Bower, A., Eftekhari, H., Yurochkin, M., & Sun, Y. (2021) · International Conference on Learning Representations · Cited by 6 · see citing works
Responsible Opinion Formation on Debated Topics in Web Search
Rieger, A., Draws, T., Mattis, N. M., Maxwell, D., Elsweiler, D., Gadiraju, U., McKay, D., Bozzon, A., & Pera, M. S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_32 · Cited by 5 · see citing works
ColBERT-FairPRF: Towards Fair Pseudo-Relevance Feedback in Dense Retrieval
Jaenich, T., McDonald, G., & Ounis, I. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_36 · Cited by 5 · see citing works
Cau, F. M., & Tintarev, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_30 · Cited by 5 · see citing works
Fairness and Explainability for Enabling Trust in AI Systems
Sacharidis, D. (2024) · Human-computer interaction series · doi:10.1007/978-3-031-55109-3_3 · Cited by 5 · see citing works
Learn to be Fair without Labels: A Distribution-based Learning Framework for Fair Ranking
Chen, F. L., & Fang, H. (2023) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3578337.3605132 · Cited by 4 · see citing works
Evaluating Content Exposure Bias in Social Networks
Bartley, N., Burghardt, K., & Lerman, K. (2023) · Proceedings of the International Conference on Advances in Social Networks Analysis and Mining · doi:10.1145/3625007.3627724 · Cited by 4 · see citing works
Intersectional Affirmative Action Policies for Top-k Candidates SelectionPreprint
Barnabò, G., Castillo, C., Mathioudakis, M., & Celis, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14775 · Cited by 4 · see citing works
What Is Fair? Exploring the Artists’ Perspective on the Fairness of Music Streaming Platforms
Ferraro, A., Serra, X., & Bauer, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85616-8_33 · Cited by 3 · see citing works
Explaining Recommendation Fairness from a User/Item Perspective
Li, J., Ren, Y., Sanderson, M., & Deng, K. (2024) · ACM Transactions on Information Systems · doi:10.1145/3698877 · Cited by 4 · see citing works
Preference eigensystems for fair ranking
Melucci, M. (2025) · Expert systems with applications · doi:10.1016/j.eswa.2024.126324 · Cited by 2 · see citing works
Fair Exposure Allocation Using Generative Query Expansion
Jaenich, T., McDonald, G., & Ounis, I. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88717-8_20 · Cited by 3 · see citing works
Fairness in Ranking under Disparate Uncertainty
Rastogi, R., & Joachims, T. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694703 · Cited by 3 · see citing works
Fairness-Aware Question Answering for Intelligent Assistants
Cherumanal, S. P. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531682 · Cited by 3 · see citing works
Privacy-Preserving Fair Item Ranking
Sun, J. A., Pentyala, S., Cock, M. D., & Farnadi, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_13 · Cited by 3 · see citing works
A Conceptual Framework for Evaluating Fairness in SearchPreprint
Das, A., & Lease, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09328 · Cited by 3 · see citing works
Fairness Through Regularization for Learning to RankPreprint
Konstantinov, N., & Lampert, C. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05996 · Cited by 8 · see citing works
Individually Fair RankingPreprint
Bower, A., Eftekhari, H., Yurochkin, M., & Sun, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11023 · Cited by 3 · see citing works
Avula, S., Lee, C., Zhang, R., & Murdock, V. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730230 · Cited by 2 · see citing works
Three methods for fair ranking of multiple protected items
Melucci, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-12735-8 · Cited by 2 · see citing works
Impacts of Personalization on Social Network Exposure
Bartley, N., Burghardt, K., & Lerman, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78538-2_3 · Cited by 2 · see citing works
Improving Algorithmic Visibility Through Competitive Repertoires When Entering Platforms
Rodriguez, J., & Piccoli, G. (2025) · Information Systems Management · doi:10.1080/10580530.2025.2477458 · Cited by 1 · see citing works
Group Fair Rated Preference Aggregation: Ties Are (Mostly) All You Need
Cachel, K., & Rundensteiner, E. A. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732042 · Cited by 1 · see citing works
Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement
Mousavian, M., Abbasiantaeb, Z., Aliannejadi, M., & Crestani, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744620 · Cited by 1 · see citing works
A Reproducibility Study of Product-side Fairness in Bundle Recommendation
Nguyen, H.-S., Liu, Y., Mansoury, M., Aliannejadi, M., Hanjalić, A., & de Rijke, M. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3748169 · Cited by 1 · see citing works
Towards Optimizing Ranking in Grid-Layout for Provider-Side Fairness
Raj, A., & Ekstrand, M. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_7 · Cited by 1 · see citing works
Toward Automatic Group Membership Annotation for Group Fairness Evaluation
Chen, F. L., Yang, D., & Fang, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_20 · Cited by 2 · see citing works
FairRankTune: A Python Toolkit for Fair Ranking Tasks
Cachel, K., & Rundensteiner, E. A. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679238 · Cited by 1 · see citing works
Dealing with Intransitivity, Non-Convexity, and Algorithmic Bias in Preference Learning
Bower, A. (2020) · Deep Blue (University of Michigan) · doi:10.7302/43
Sarvi, F., Aliannejadi, M., Schelter, S., & de Rijke, M. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578278
Measuring the Echo-Chamber Phenomenon Through Exposure Bias
Bartley, N., Burghardt, K., & Lerman, K. (2024) · Lecture notes in social networks · doi:10.1007/978-3-031-75204-9_13
LambdaFair for Fair and Effective Ranking
Marcuzzi, F., Lucchese, C., & Orlando, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88717-8_15
FAIR-QR: Enhancing Fairness-Aware Information Retrieval Through Query Refinement
Chen, F. L., & Fang, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88717-8_1
CAFE: Context-Aware Applicability-Weighted Fairness Evaluation
Chen, F. L., & Fang, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97141-9_18
Diversification and Fairness in Search: Two Sides of the Same Coin?
Cherumanal, S. P., Scholer, F., & Spina, D. (2025) · ACM Transactions on Information Systems · doi:10.1145/3774320
Measuring Group Fairness in Web Search: AWRF or GFR?
Tao, S., & Sakai, T. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769504
Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-reasoning Rerankers
Samuel, S., Durme, B. V., & Yang, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21324-2_25
Optimizing stock market investments using the fuzzy equitable fair domination integrity graph model
Prabha, E., Balasubramaniam, K., Sankar, R., & Gunasekar, T. (2026) · Boletim da Sociedade Paranaense de Matemática · doi:10.5269/bspm.82044
Fair Information Retrieval Methods
Mishra, H., & Soundarajan, S. (2026) · SpringerBriefs in intelligent systems · doi:10.1007/978-3-032-24145-0_5
Mishra, H., & Soundarajan, S. (2026) · SpringerBriefs in intelligent systems · doi:10.1007/978-3-032-24145-0_3
You, J., Lee, K., & Kwon, H.-Y. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809651
Assessing the Interplay of Counteracting Biases in GNN-Based Recommender Systems
Chizari, N., Tajfar, K., & Moreno-García, M. N. (2025) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat67162.2025.00048
- Authors
- Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C.
- Affiliations
- Northeastern Univ.
- Northeastern University (US)
- Venue
- Companion Proceedings of The 2019 World Wide Web Conference
- Publisher
- ACM
- Published
- 2019-05-13
- Pages
- 553-562
- OpenAlex
- W2912638141
- Cited by
- 92
- References
- 46
- Citations by year
- 2026: 5 · 2025: 16 · 2024: 19 · 2023: 11 · 2022: 11 · 2021: 18 · 2020: 7 · 2019: 3 · 2018: 1 · 2012: 1
- FWCI
- 10.19
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International World Wide Web Conference (WWW) (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- group fairness, ranked lists, user attention, fairness auditing, fairness metrics, search result ranking, socio-technical systems, political alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
