CoChat
Sign Up
PreprintCited by 1Ethics and Social Impacts of AIGame Theory and Voting SystemsAI and HR Technologies

The Initial Screening Order Problem

Open access
Alvarez, Jose M., Mastropietro, Antonio, Ruggieri, Salvatore·2023·arXiv
DOI 10.48550/arxiv.2307.15398arXiv 2307.15398
Open this paper in CoChat — free accountOpen PDF

Abstract

We investigate the role of the initial screening order (ISO) in candidate screening. The ISO refers to the order in which the screener searches the candidate pool when selecting $k$ candidates. Today, it is common for the ISO to be the product of an information access system, such as an online platform or a database query. The ISO has been largely overlooked in the literature, despite its impact on the optimality and fairness of the selected $k$ candidates, especially under a human screener. We define two problem formulations describing the search behavior of the screener given an ISO: the best-$k$, where it selects the top $k$ candidates; and the good-$k$, where it selects the first good-enough $k$ candidates. To study the impact of the ISO, we introduce a human-like screener and compare it to its algorithmic counterpart, where the human-like screener is conceived to be inconsistent over time. Our analysis, in particular, shows that the ISO, under a human-like screener solving for the good-$k$ problem, hinders individual fairness despite meeting group fairness, and hampers the optimality of the selected $k$ candidates. This is due to position bias, where a candidate's evaluation is affected by its position within the ISO. We report extensive simulated experiments exploring the parameters of the best-$k$ and good-$k$ problems for both screeners. Our simulation framework is flexible enough to account for multiple candidate screening tasks, being an alternative to running real-world procedures.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.15398

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The prevalent human vs. algorithm evaluation framework is of limited utility; more research on recruiter-machine interaction is required.

    “The prevalent human vs. algorithm evaluation framework is of limited utility; to overcome it, more research on recruitermachine interaction is required [240], including candidate screening models [6] leading to more granular measures.”

    From Fairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey (2025) · cites this paper as [6]

    The Introduction (section 1, page unavailable) notes calls to consider the human user's behavior, and the Conclusion (section 5, page unavailable) proposes further work on human decision-making in candidate screening. These support more research on recruiter–machine interaction, but the paper does not say that the prevalent human-versus-algorithm evaluation framework is of limited utility. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 56

  1. Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang. 2013. Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects. Inf. Syst. Res. 24, 4 (2013), 956-975.
  2. José M. Álvarez, Alejandra Bringas Colmenarejo, Alaa Elobaid, Simone Fabbrizzi, Miriam Fahimi, Antonio Ferrara, Siamak Ghodsi, Carlos Mougan, Ioanna Papa- georgiou, Paula Reyero Lobo, Mayra Russo, Kristen M. Scott, Laura State, Xuan Zhao, and Salvatore Ruggieri. 2024. Policy advice and best practices on bias and fairness in AI. Ethics Inf. Technol. 26, 2 (2024), 31.
  3. Susan Athey and Glenn Ellison. 2011. Position Auctions with Consumer Search. The Quarterly Journal of Economics 126, 3 (2011), 1213-1270.
  4. Ricardo Baeza-Yates. 2018. Bias on the web. Commun. ACM 61, 6 (2018), 54-61.
  5. Ricardo A. Baeza-Yates and Berthier A. Ribeiro-Neto. 1999. Modern Information Retrieval. ACM Press / Addison-Wesley.
  6. Eszter Bokányi and Anikó Hannák. 2020. Understanding inequalities in ride- hailing services through simulations. Scientific reports 10, 1 (2020), 1-11.
  7. Z. I. Botev. 2017. The normal law under linear restrictions: simulation and estimation via minimax tilting. Journal of the Royal Statistical Society Series B 79, 1 (2017), 125-148.
  8. Dimitrios Bountouridis, Jaron Harambam, Mykola Makhortykh, Mónica Marrero, Nava Tintarev, and Claudia Hauff. 2019. SIREN: A Simulation Framework for Un- derstanding the Effects of Recommender Systems in Online News Environments. In FAT. ACM, 150-159.
  9. Alejandra Bringas Colmenarejo, Luca Nannini, Alisa Rieger, Kristen M Scott, Xuan Zhao, Gourab K Patro, Gjergji Kasneci, and Katharina Kinder-Kurlanda. 2022. Fairness in agreement with European values: An interdisciplinary perspec- tive on ai regulation. In AIES. ACM, 107-118.
  10. Thomas Kleine Buening, Meirav Segal, Debabrota Basu, Anne-Marie George, and Christos Dimitrakakis. 2022. On Meritocracy in Optimal Set Selection. In EAAMO. ACM, 20:1-20:14.
  11. Ana Karina Caraban, Evangelos Karapanos, Daniel Gonçalves, and Pedro F. Campos. 2019. 23 Ways to Nudge: A Review of Technology-Mediated Nudging in Human-Computer Interaction. In CHI. ACM, 503.
  12. Gonçalo Carriço. 2018. The EU and artificial intelligence: A human-centred perspective. European View 17, 1 (2018), 29-36.
  13. Praveen Chandar, Brian St. Thomas, Lucas Maystre, Vijay Pappu, Roberto Sanchis-Ojeda, Tiffany Wu, Ben Carterette, Mounia Lalmas, and Tony Jebara. 2022. Using Survival Models to Estimate User Engagement in Online Experiments. In WWW. ACM, 3186-3195.
  14. Nick Craswell, Onno Zoeter, Michael J. Taylor, and Bill Ramsey. 2008. An experimental comparison of click position-bias models. In WSDM. ACM, 87-94.
  15. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. 2012. Fairness through awareness. In ITCS. ACM, 214-226.
  16. Jessica Maria Echterhoff, Matin Yarmand, and Julian J. McAuley. 2022. AI- Moderated Decision-Making: Capturing and Balancing Anchoring Bias in Se- quential Decision Tasks. In CHI. ACM, 161:1-161:9.
  17. Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022. Fairness in Information Access Systems. Found. Trends Inf. Retr. 16, 1-2 (2022), 1-177.
  18. Paul Embrechts, Filip Lindskog, and Alexander Mcneil. 2003. Modelling Depen- dence with Copulas and Applications to Risk Management. In Handbook of Heavy Tailed Distributions in Finance, Svetlozar T. Rachev (Ed.). Vol. 1. North-Holland, Amsterdam, 329-384.
  19. Alessandro Fabris, Nina Baranowska, Matthew J. Dennis, David Graus, Philipp Hacker, Jorge Saldivar, Frederik Zuiderveen Borgesius, and Asia J. Biega. 2024. Fairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey. ACM Trans. Intell. Syst. Technol. (Sept. 2024).
  20. Ronald Fagin, Amnon Lotem, and Moni Naor. 2003. Optimal aggregation algo- rithms for middleware. J. Comput. Syst. Sci. 66, 4 (2003), 614-656.
  21. Thomas S Ferguson. 1989. Who solved the secretary problem? Statist. Sci. 4, 3 (1989), 282-289.
  22. Artem Grotov, Aleksandr Chuklin, Ilya Markov, Luka Stout, Finde Xumara, and Maarten de Rijke. 2015. A Comparative Study of Click Models for Web Search. In CLEF (Lecture Notes in Computer Science, Vol. 9283). Springer, 78-90.
  23. Yael S Hadass. 2004. The effect of internet recruiting on the matching of workers and employers. SSRN 497262 (2004).
  24. Stefania Ionescu, Anikó Hannák, and Kenneth Joseph. 2021. An Agent-based Model to Evaluate Interventions on Online Dating Platforms to Decrease Racial Homogamy. In FAccT. ACM, 412-423.
  25. Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard Friedrich. 2010. Recommender Systems -An Introduction. Cambridge University Press.
  26. Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2017. Accurately Interpreting Clickthrough Data as Implicit Feedback. SIGIR Forum 51, 1 (2017), 4-11.
  27. Donald Martin Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S. Isaac. 2020. Extending the Machine Learning Abstraction Bound- ary: A Complex Systems Approach to Incorporate Societal Context. CoRR abs/2006.09663 (2020). · arXiv
  28. Emre Kazim, Adriano Soares Koshiyama, Airlie Hilliard, and Roseline Polle. 2021. Systematizing Audit in Algorithmic Recruitment. Journal of Intelligence 9, 3 (2021).
  29. Marios Kokkodis. 2018. Dynamic Recommendations for Sequential Hiring Deci- sions in Online Labor Markets. In KDD. ACM, 453-461.
  30. Marios Kokkodis, Panagiotis Papadimitriou, and Panagiotis G. Ipeirotis. 2015. Hiring Behavior Models for Online Labor Markets. In WSDM. ACM, 223-232.
  31. Alina Köchling and Marius Claus Wehner. 2020. Discriminated by an algorithm: a systematic review of discrimination and fairness by algorithmic decision-making in the context of HR recruitment and HR development. Business Research 13, 3 (2020).
  32. Anay Mehrotra and L. Elisa Celis. 2021. Mitigating Bias in Set Selection with Noisy Protected Attributes. In FAccT. ACM, 237-248.
  33. Eduardo Mosqueira-Rey, Elena Hernández-Pereira, David Alonso-Ríos, José Bobes-Bascarán, and Ángel Fernández-Leal. 2023. Human-in-the-loop machine learning: A state of the art. Artif. Intell. Rev. 56, 4 (2023), 3005-3054.
  34. Zohreh Ovaisi, Parsa Saadatpanah, Shahin Sefati, Mesrob Ohannessian, and Elena Zheleva. 2024. Fairness of Interaction in Ranking under Position, Selection, and Trust Bias. ACM Trans. Recomm. Syst. (2024). https://doi.org/10.1145/3652864 Just Accepted. · DOI
  35. Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura A. Granka. 2007. In Google We Trust: Users' Decisions on Rank, Position, and Relevance. J. Comput. Mediat. Commun. 12, 3 (2007), 801-823.
  36. Elena Pisanelli. 2022. Your resume is your gatekeeper: Automated resume screen- ing as a strategy to reduce gender gaps in hiring. Economics Letters 221 (2022), 110892.
  37. Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. 2022. Fairness in rankings and recommendations: an overview. VLDB J. 31, 3 (2022), 431-458.
  38. Chuan Qin, Le Zhang, Rui Zha, Dazhong Shen, Qi Zhang, Ying Sun, Chen Zhu, Hengshu Zhu, and Hui Xiong. 2023. A Comprehensive Survey of Artificial Intelligence Techniques for Talent Analytics. CoRR abs/2307.03195 (2023). · arXiv
  39. Manish Raghavan, Solon Barocas, Jon M. Kleinberg, and Karen Levy. 2020. Miti- gating bias in algorithmic hiring: evaluating claims and practices. In FAT*. ACM, 469-481.
  40. Alene K. Rhea, Kelsey Markey, Lauren D'Arinzo, Hilke Schellmann, Mona Sloane, Paul Squires, Falaah Arif Khan, and Julia Stoyanovich. 2022. An external stability audit framework to test the validity of personality prediction in AI hiring. Data Min. Knowl. Discov. 36, 6 (2022), 2153-2193.
  41. Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: estimating the click-through rate for new ads. In WWW. ACM, 521-530.
  42. Salvatore Ruggieri, José M. Álvarez, Andrea Pugnana, Laura State, and Franco Turini. 2023. Can We Trust Fair-AI?. In AAAI. AAAI Press, 15421-15430.
  43. Thomas C Schelling. 1971. Dynamic models of segregation. Journal of Mathe- matical Sociology 1, 2 (1971), 143-186.
  44. Ashudeep Singh and Thorsten Joachims. 2018. Fairness of Exposure in Rankings. In KDD. ACM, 2219-2228.
  45. Mona Sloane, Emanuel Moss, and Rumman Chowdhury. 2022. A Silicon Valley love triangle: Hiring algorithms, pseudo-science, and the quest for auditability. Patterns 3, 2 (2022), 100425.
  46. Keith E. Sonderling, Bradford J. Kelley, and Lance Casimir. 2022. The Promise and The Peril: Artificial Intelligence and Employment Discrimination. U. Miami Law Review 77, 1 (2022), 3.
  47. Poorna Talkad Sukumar, Ronald A. Metoyer, and Shuai He. 2018. Making a Pecan Pie: Understanding and Supporting The Holistic Review Process in Admissions. Proc. ACM Hum. Comput. Interact. 2, CSCW (2018), 169:1-169:22.
  48. Amos Tversky and Daniel Kahneman. 1974. Judgment Under Uncertainty: Heuris- tics and Biases. Science 185, 4157 (1974), 1124-1131.
  49. Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position Bias Estimation for Unbiased Learning to Rank in Personal Search. In WSDM. ACM, 610-618.
  50. Zhihan (Helen) Wang, Jiaxin Pei, and Jun Li. 2023. 30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity. Available at SSRN: https://ssrn.com/abstract=4603146.
  51. Paris Will, Dario Krpan, and Grace Lordan. 2023. People versus machines: introducing the HIRE framework. Artif. Intell. Rev. 56, 2 (2023), 1071-1100.
  52. Christo Wilson, Avijit Ghosh, Shan Jiang, Alan Mislove, Lewis Baker, Janelle Szary, Kelly Trindel, and Frida Polli. 2021. Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. In FAccT. ACM, 666-677.
  53. Ke Yang and Julia Stoyanovich. 2017. Measuring Fairness in Ranked Outputs. In SSDBM. ACM, 22:1-22:6.
  54. Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Mega- hed, and Ricardo Baeza-Yates. 2017. FA*IR: A Fair Top-k Ranking Algorithm. In CIKM. ACM, 1569-1578.
  55. Meike Zehlike, Ke Yang, and Julia Stoyanovich. 2023. Fairness in Ranking, Part I: Score-Based Ranking. ACM Comput. Surv. 55, 6 (2023), 118:1-118:36.
  56. Meike Zehlike, Ke Yang, and Julia Stoyanovich. 2023. Fairness in Ranking, Part II: Learning-to-Rank and Recommender Systems. ACM Comput. Surv. 55, 6 (2023), 117:1-117:41.

Cited by

Sorted by citations

Authors
Alvarez, J. M., Mastropietro, A., & Ruggieri, S.
Affiliations
University of Pisa (IT)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
165-174
DOI
10.48550/arxiv.2307.15398
arXiv
2307.15398
OpenAlex
W4407953602
Cited by
1
References
0
Citations by year
2024: 1
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • European Commission (101070212, HORIZON2020, 860630)
  • Università di Pisa
  • HORIZON EUROPE Framework Programme
  • HORIZON EUROPE Marie Sklodowska-Curie Actions (860630)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
Keywords
applicant screening, position bias, individual fairness, group fairness, top-k selection, search order

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review