FAIR : Fairness‐aware information retrieval evaluation
Open accessAbstract
Abstract With the emerging needs of creating fairness‐aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness‐aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR‐based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Fairness-aware performance metrics provide a holistic view of fairness model performance.
“Provides a holistic view of fairness model performance”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [260]
The choice of fairness metric may not fully capture desired notions of fairness.
“Choice of fairness metric desired notions of fairness may not fully capture”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [260]
Fairness-aware performance metrics simultaneously evaluate accuracy and fairness.
“Simultaneously evaluates accuracy and fairness”
From AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications (2023) · cites this paper as [260]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Agrawal, R., Gollapudi, S., Halverson, A., & Ieong, S. (2009) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1498759.1498766 · Cited by 1,031 · see citing works
Novelty and diversity in information retrieval evaluation
Clarke, C. L. A., Kolla, M., Cormack, G. V., Olga Vechtomova, A., Ashkan, A., Büttcher, S., & MacKinnon, I. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390446 · Cited by 968 · see citing works
Expected reciprocal rank for graded relevance
Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1645953.1646033 · Cited by 840 · see citing works
Rank-biased precision for measurement of retrieval effectiveness
Moffat, A., & Zobel, J. (2008) · ACM Transactions on Information Systems · doi:10.1145/1416950.1416952 · Cited by 620 · see citing works
Fairness of Exposure in Rankings
Singh, A., & Joachims, T. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220088 · Cited by 607 · see citing works
The movielens datasets: History and context
Harper, F. M. (2024) · TIB Data Manager · doi:10.57702/2vwfx36r · Cited by 548 · see citing works
Unbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531 · see citing works
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · see citing works
Baeza-Yates, R. (2018) · Communications of the ACM · doi:10.1145/3209581 · Cited by 432 · cited 1× · see citing works
“Furthermore, because users frequently only click on the top results, a ranking or recommendation algorithm that takes user feedback into account, will continue putting those same items at the top, creating a positive feedback loop and a vicious cycle of unfairness (Baeza-Yates, 2018; Joachims et al., 2017).”
BackgroundFairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330691 · Cited by 382 · see citing works
Measuring Fairness in Ranked Outputs
Yang, K., & Stoyanovich, J. (2017) · Proceedings of the 29th International Conference on Scientific and Statistical Database Management · doi:10.1145/3085504.3085526 · Cited by 378 · see citing works
Fairness in Recommendation Ranking through Pairwise Comparisons
Beutel, A., Chen, J., Doshi, T. S., Qian, H., Wei, L., Wu, Y., Heldt, L., Zhao, Z., Hong, L., Ed, H., et al. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330745 · Cited by 374 · see citing works
Mehrotra, R., McInerney, J., Bouchard, H., Lalmas, M., & Díaz, F. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3272027 · Cited by 280 · see citing works
Controlling Fairness and Bias in Dynamic Learning-to-Rank
Morik, M., Singh, A., Hong, J., & Joachims, T. (2020) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401100 · Cited by 260 · see citing works
Ranking with Fairness ConstraintsPreprint
Celis, L. E., Straszak, D., & Vishnoi, N. K. (2017) · arXiv (Cornell University) · doi:10.4230/lipics.icalp.2018.28 · Cited by 237 · see citing works
Towards Long-term Fairness in Recommendation
Ge, Y., Liu, S., Gao, R., Xian, Y., Li, Y., Zhao, X., Pei, C., Sun, F., Ge, J., Ou, W., et al. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441824 · Cited by 201 · see citing works
Self-citationWhy is my classifier discriminatory?
Chen, I., Johansson, F. D., & Sontag, D. (2018) · arXiv · doi:10.48550/arxiv.1805.12002 · Cited by 185 · cited 1× · see citing works
“…as a measure for bias in many previous works, the d KL + 1 may be replaced by other metrics such as the distance between two fairness representation vectors (Chen et al., 2018; Equation 2), the squared error between the expected exposure and the desired exposure (Diaz et al., 2020; Equation 1).”
BackgroundEvaluating Stochastic Rankings with Expected Exposure
Díaz, F., Mitra, B., Ekstrand, M. D., Biega, A. J., & Carterette, B. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411962 · Cited by 175 · see citing works
Beyond Parity: Fairness Objectives for Collaborative Filtering
Yao, S., & Huang, B. (2017) · arXiv · doi:10.48550/arxiv.1705.08804 · Cited by 169 · see citing works
Designing Fair Ranking Schemes
Asudeh, A., Jagadish, H. V., Stoyanovich, J., & Das, G. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3300079 · Cited by 159 · see citing works
How well does result relevance predict session satisfaction?
Huffman, S. B., & Hochster, M. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277839 · Cited by 136 · see citing works
An Effectiveness Measure for Ambiguous and Underspecified Queries
Clarke, C. L. A., Kolla, M., & Vechtomova, O. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04417-5_17 · Cited by 120 · see citing works
Toward creating a fairer ranking in search engine results
Gao, R., & Shah, C. S. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102138 · Cited by 107 · see citing works
Self-citationFairness Metrics: A Comparative Analysis
Garg, P., Villasenor, J. D., & Foggo, V. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378025 · Cited by 96 · see citing works
Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317595 · Cited by 92 · see citing works
Addressing Marketing Bias in Product Recommendations
Wan, M., Ni, J., Misra, R., & McAuley, J. (2020) · Proceedings of the 13th International Conference on Web Search and Data Mining · doi:10.1145/3336191.3371855 · Cited by 81 · see citing works
Controlling Polarization in Personalization
Celis, L. E., Kapoor, S., Salehi, F., & Vishnoi, N. K. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287601 · Cited by 65 · see citing works
Fair and Diverse DPP-based Data SummarizationPreprint
Celis, L. E., Keswani, V., Straszak, D., Deshpande, A., Kathuria, T., & Vishnoi, N. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04023 · Cited by 59 · see citing works
FARE: Diagnostics for Fair Ranking using Pairwise Error Metrics
Kuhlman, C., VanValkenburg, M., & Rundensteiner, E. A. (2019) · The Web Conference · doi:10.1145/3308558.3313443 · Cited by 57 · see citing works
8th international colloquium on automata, languages and programming
(1980) · Information Processing Letters · doi:10.1016/0020-0190(80)90016-2 · Cited by 45 · see citing works
Maximizing Marginal Fairness for Dynamic Learning to Rank
Yang, T., & Ai, Q. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449901 · Cited by 36 · see citing works
Balancing Between Accuracy and Fairness for Interactive Recommendation with Reinforcement Learning
Liu, W., Liu, F., Tang, R., Liao, B., Chen, G., & Heng, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-47426-3_13 · Cited by 31 · see citing works
Gao, R., & Shah, C. S. (2019) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3341981.3344215 · Cited by 28 · see citing works
Self-citationDear search engine: what's your opinion about...?
Demartini, G., & Siersdorfer, S. (2010) · doi:10.1145/1863879.1863883 · Cited by 27 · see citing works
Evaluating Fairness Metrics in the Presence of Dataset BiasPreprint
Hinnefeld, J. H., Cooman, P., Mammo, N., & Deese, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09245 · Cited by 24 · see citing works
Addressing Bias and Fairness in Search Systems
Gao, R., & Shah, C. S. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462807 · Cited by 22 · see citing works
Self-citationNew Fairness Metrics for Recommendation that Embrace DifferencesPreprint
Yao, S., & Huang, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09838 · Cited by 20 · see citing works
Comparing Fair Ranking MetricsPreprint
Raj, A., & Ekstrand, M. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01311 · Cited by 12 · see citing works
Quantifying the Impact of User Attention on Fair Group Representation in Ranked ListsPreprint
Sapieżyński, P., Zeng, W., Robertson, R. E., Mislove, A. E., & Wilson, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10437 · Cited by 10 · see citing works
Balancing diversity to counter-measure geographical centralization in microblogging platforms
Graells-Garrido, E., & Lalmas, M. (2014) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/2631775.2631823 · Cited by 9 · see citing works
Flagged first, then by citations
1 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017). FA*IR: A fair topk ranking algorithm. In Proceedings of the 2017 ACM on conference on information and knowledge management (pp. 1569 -1578). ACM.
- How to cite this article: Gao, R., Ge, Y., & Shah, C. (2022). FAIR: Fairness-aware information retrieval evaluation. Journal of the Association for Information Science and Technology , 1 -13. https:// doi.org/10.1002/asi.24648
Cited by
Sorted by citations
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155 · see citing works
Explainable Fairness in Recommendation
Ge, Y., Tan, J., Zhu, Y., Xia, Y., Luo, J., Liu, S., Fu, Z., Geng, S., Li, Z., & Zhang, Y. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531973 · Cited by 55 · see citing works
Self-citationA Systematic Review of Fairness, Accountability, Transparency, and Ethics in Information Retrieval
Bernard, N., & Balog, K. (2023) · ACM Computing Surveys · doi:10.1145/3637211 · Cited by 43 · see citing works
No Prejudice! Fair Federated Graph Neural Networks for Personalized Recommendation
Agrawal, N., Sirohi, A. K., Kumar, S., & Jayadeva (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.28950 · Cited by 34 · see citing works
Li, R., Deng, W., Cheng, Y., Yuan, Z., Zhang, J., & Yuan, F. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761429 · Cited by 51 · see citing works
Yi, F., Singh, A., & Tao, Z. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000101 · Cited by 12 · see citing works
How Well do Offline Metrics Predict Online Performance of Product Ranking Models?
Wang, X., Gao, R., Jain, A., Edge, G., & Ahuja, S. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591865 · Cited by 9 · see citing works
Self-citationNot Just Algorithms: Strategically Addressing Consumer Impacts in Information Retrieval
Ekstrand, M. D., Beattie, L., Pera, M. S., & Cramer, H. S. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_25 · Cited by 8 · see citing works
Vertical Allocation-based Fair Exposure Amortizing in Ranking
Yang, T., Xu, Z., & Ai, Q. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625313 · Cited by 13 · see citing works
Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance
Rampisela, T. V., Ruotsalo, T., Maistro, M., & Lioma, C. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657832 · Cited by 6 · see citing works
FARA: Future-aware Ranking Algorithm for Fairness Optimization
Yang, T., Xu, Z., Wang, Z., & Ai, Q. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614877 · Cited by 4 · see citing works
Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier
Rampisela, T. V., Ruotsalo, T., Maistro, M., & Lioma, C. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714589 · Cited by 2 · see citing works
Rampisela, T. V., Maistro, M., Ruotsalo, T., Scholer, F., & Lioma, C. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3765624 · Cited by 2 · see citing works
Learning-Based Models for Building User Profiles for Personalized Information Access
Hidri, M. S. (2024) · Interdisciplinary Journal of Information Knowledge and Management · doi:10.28945/5275 · Cited by 1 · see citing works
Martens, B. V. D. V. (2023) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-32530-4_8
Cui, W., Ting-feng, L., Congli, C., & Na, D. E. (2023) · International Conference on Communication and Electronics Systems (ICCES) · doi:10.1109/icces57224.2023.10192606
Gazik, M. A. R., & Roayaei, M. (2024) · Research Square · doi:10.21203/rs.3.rs-3933427/v1
Sampling Strategy on Ranked Outputs Data Towards Fairness-Aware Evaluation with Sparse Datasets
Dores, A., & Ravana, S. D. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-4305-1_24
Domain-specific information retrieval
Verberne, S., Piroi, F., & Hanbury, A. (2026) · Foundations and Trends® in Information Retrieval · doi:10.1108/ftinr-09-2025-0088
- Authors
- Gao, R., Ge, Y., & Shah, C.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- University of Washington (US)
- Corresponding author
- Gao, Ruoyuan
- Venue
- Journal of the Association for Information Science and Technology
- Publisher
- Wiley
- Published
- 2022-03-22
- Volume
- 73
- Issue
- 10
- Pages
- 1461-1473
- ISSN
- 2330-1635 (ISSN-L)
- 2330-1643
- OpenAlex
- W4220802549
- Cited by
- 19
- References
- 46
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 6 · 2023: 7 · 2022: 1
- FWCI
- 4.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the Association for Information Science and Technology (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- fairness metrics, user utility, ranking algorithms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
