CoChat
Sign Up
Cited by 65AI and HR TechnologiesEthics and Social Impacts of AINames, Identity, and Discrimination Research

Hiring as Exploration

Open access
Li, Danielle, Raymond, Lindsey, Bergman, Peter·2020·National Bureau of Economic Research
DOI 10.3386/w27736
Open this paper in CoChat — free account

Abstract

This paper views hiring as a contextual bandit problem: to find the best workers over time, firms must balance "exploitation" (selecting from groups with proven track records) with "exploration" (selecting from under-represented groups to learn about quality).Yet modern hiring algorithms, based on "supervised learning" approaches, are designed solely for exploitation.Instead, we build a resume screening algorithm that values exploration by evaluating candidates according to their statistical upside potential.Using data from professional services recruiting within a Fortune 500 firm, we show that this approach improves the quality (as measured by eventual hiring rates) of candidates selected for an interview, while also increasing demographic diversity, relative to the firm's existing practices.The same is not true for traditional supervised learning based algorithms, which improve hiring rates but select far fewer Black and Hispanic applicants.In an extension, we show that exploration-based algorithms are also able to learn more effectively about simulated changes in applicant hiring potential over time.Together, our results highlight the importance of incorporating exploration in developing decision-making algorithms that are potentially both more efficient and equitable.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper frames hiring as a contextual bandit and builds a UCB exploration-based resume screener, comparing it with static and updating supervised learning using applications to a Fortune 500 professional-services firm. It finds the UCB algorithm improves predicted hiring yield and increases Black/Hispanic representation relative to human recruiters, while supervised learning improves yield but sharply reduces minority representation.

Key findings

  • UCB raises the Black/Hispanic share of selected applicants from 10% under human recruiters to 23%; static SL reduces it to approximately 2%, and updating SL to approximately 5%. For women, human recruiters select 35%, static SL 41%, updating SL 50%, and UCB 39%.
  • Full-sample decomposition estimates average hiring rates of 10% for human-selected applicants, 15% for static SL, 25% for UCB, and 30% for updating SL. In the interviewed-only sample, human scores are weakly if not negatively related to hiring outcomes (coefficient −0.0562), while static SL, updating SL, and UCB scores are positively related (0.232*, 0.215*, and 0.233*, respectively).
  • Among marginally interviewed applicants, following UCB recommendations increases both hiring yield and the share of Black, Hispanic, and female interviewees; following SL recommendations increases hiring yield but decreases minority representation.
  • Simulations show UCB learns faster than updating SL when minority hiring potential rises; blinding the UCB model to race and gender reduces the Black/Hispanic share from 23% to 14% (still above the 10% human baseline) and increases Asian representation to 61% versus 26% White.

Methods

  • Data: 88,666 job applications from January 2016 to April 2019 to high-skill professional-services roles at a Fortune 500 firm; training set of 48,719 applicants before 2018 (2,617 interviewed) and test set of 39,947 applicants afterward (2,275 interviewed), with 106 input features. Applicant quality is defined as likelihood of being hired conditional on being interviewed.
  • Algorithms: static supervised learning (ensemble LASSO and random forest), updating supervised learning (same model with rolling outcome updates), and a UCB contextual bandit GLM with an exploration bonus. Evaluation uses an interviewed-only human-SL comparison, a full-sample DiNardo decomposition, and a marginal-complier analysis using random assignment to screeners as an instrument.
  • Extensions: simulations in which one racial group’s hiring potential changes over time, and a version of the UCB model blinded to race, ethnicity, and gender.

Limitations

  • The paper states selective-labels problems: hiring outcomes are observed only for interviewed applicants, so full-sample and marginal analyses rely on assumptions such as no selection on unobservables or coarse covariate cells; the short test period (2018–2019Q1) and inability to interview all ML-selected candidates limit learning; quality is measured by hiring likelihood rather than on-the-job performance, with performance data available for only a small subset (e.g., 180 workers with mid-year ratings and 233 hires for promotion analysis).
  • The paper states that algorithms use race, ethnicity, and gender as inputs, raising legal concerns about disparate treatment; results come from one Fortune 500 high-skill professional-services firm and may not generalize across sectors or firms.
  • The text makes evident that UCB underperforms updating SL in the actual test sample despite theoretical predictions, which the authors attribute to the short test period and limited updating; simulation results rely on simulated changes in applicant quality, and performance correlations are noisy and subject to sample selection.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3386/w27736

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 24 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Interviews are costly because they divert high-skill current employees from other productive tasks.

    “These interviews are moreover costly because they divert high-skill current employees from other productive tasks (Kuhn and Yu, 2019).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kuhn and Yu, 2019 source

      The Introduction reports productivity costs when incumbent employees devote time and attention to filling a vacancy. The “More on Hiring at Firm A” section says a regional manager spends about 12 hours per hire on a process that includes a short interview. The paper does not identify interviewers as high-skill employees or isolate interviews, rather than recruiting more broadly, as the source of the productivity loss. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Findingown data

    The UCB model continues to select more minority applicants relative to both the human and SL models, even as exploration bonuses fall.

    “Instead, we show that the UCB model continues to select more minority applicants relative to both the human and SL models, even as exploration bonuses fall.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The hiring process can be modeled as a contextual bandit problem.
    • The authors state this as an assumption: The relationship between context X and rewards follows a generalized linear form.
  • Findingnone given

    Traditional supervised learning based algorithms improve hiring rates.

    “The same is not true for traditional supervised learning based algorithms, which improve hiring rates but select far fewer Black and Hispanic applicants.”
    Quote in the paperSubstantial claim
  • Findingown data

    Exploration-based algorithms learn more effectively about simulated changes in applicant hiring potential over time.

    “In an extension, we show that exploration-based algorithms are also able to learn more effectively about simulated changes in applicant hiring potential over time.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The hiring process can be modeled as a contextual bandit problem.
    • The authors state this as an assumption: The relationship between context X and rewards follows a generalized linear form.
  • Findingown data

    ML models trained to maximize hiring likelihood appear more positively correlated with on-the-job performance than a model trained to mimic the choices of human recruiters.

    “We show, however, that our ML models (trained to maximize hiring likelihood) appear more positively correlated with on the job performance than a model trained to mimic the choices of human recruiters.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 27
  • May combine two claims. 59
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The firm has a fairly rigorous data collection process.

    “For example, our firm has a fairly rigorous data collection process: firms that do not may make different adoption decisions and have different potential returns(Athey and Stern, 1998).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Athey and Stern, 1998 source

      The Introduction (p. 4) describes firms making adoption decisions based on differing costs and benefits, and Section 5 (p. 30) discusses unobserved returns to practices. But the paper does not link those differences to whether a firm has a rigorous data-collection process. The citing authors’ description of their own firm is not a claim about this source. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    Following SL recommendations on the margin would generate similar increases in hiring yield but decrease minority representation.

    “In contrast, following SL recommendations would generate similar increases in hiring yield but decrease minority representation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The firm receives approximately 200 applications for each worker it hires.

    “our firm receives approximately 200 applications for each worker it hires.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Women make up 32% of applicants and 34% of hires.

    “Women, meanwhile, make up 32% of applicants and 34% of hires.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The static SL model has an AUC of .67 on out-of-sample test data, meaning it ranks an interviewed applicant who is hired higher than an interviewed but not hired applicant 67 percent of the time.

    “Our model has an AUC of .67,meaning that it will rank an interviewed applicant who is hired higher than an interviewed but not hired applicant 67 percent of the time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The static SL model also outperforms human decision-making, with a 15% predicted hiring yield.

    “Our static SL model also outperforms human decision-making, with a 15% predicted hiring yield.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is no selection on unobservables in the decomposition approach.

  • May combine two claims.

    Findingown data

    The updating SL model slightly outperforms UCB in the actual test sample, but the reverse is true in a simulated environment in which quality measures associated with minority applicants are changing.

    “Whereas our updating SL model slightly outperforms UCB in our actual test sample, the reverse is true in a simulated environment in which quality measures associated with minority applicants are changing.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A supervised learning approach would improve hiring rates, but at the cost of virtually eliminating Black and Hispanic representation.

    “In our setting, a supervised learning approach-which is commonly used by commercial vendors of ML-based HR tools-would improve hiring rates, but at the cost of virtually eliminating Black and Hispanic representation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A model with demographic variables removed can achieve similar improvements in hiring yield, but with more modest increases in share of under-represented minorities who are selected.

    “We show that this model can achieve similar improvements in hiring yield, but with more modest increases in share of under-represented minorities who are selected.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The hiring process can be modeled as a contextual bandit problem.
    • The authors state this as an assumption: The relationship between context X and rewards follows a generalized linear form.

  • May combine two claims.

    Findingown data

    Following UCB recommendations on the margin would increase both hiring yield and the share of Black, Hispanic, and female interviewees.

    “We find that following UCB recommendations on the margin would increase both hiring yield and the share of Black, Hispanic, and female interviewees.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    All algorithms would increase the share of selected applicants who are women, from 35% under human recruiting, to 41%, 50%, and 39%, under static SL, updating SL, and UCB, respectively.

    “In the case of gender, all algorithms would increase the share of selected applicants who are women, from 35% under human recruiting, to 41%, 50%, and 39%, under static SL, updating SL, and UCB, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Asian applicants make up the majority of the applicant sample and would receive low exploration bonuses on the basis of race alone, but are also more likely to have non-traditional work histories or have gone to smaller international colleges, making them appear more distinctive to the UCB model.

    “Asian applicants, for example, make up the majority of our applicant sample and so would receive low exploration bonuses on the basis of race alone; however, they are also more likely to have non-traditional work histories or have gone to smaller international colleges, factors that make them appear more distinctive to the UCB model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Black and Hispanic candidates comprise 13% of all applications but under 5% of hires.

    “Black and Hispanic candidates comprise 13% of all applications, but under 5% of hires.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Within the interviewed sample, scores and hiring outcomes are positively correlated for all ML models, while human scores and hiring outcomes are weakly if not negatively related.

    “While scores and hiring outcomes are positively correlated for all ML models, human scores and hiring outcomes are weakly if not negatively related.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Most applicants in the data are male (68%), Asian (58%), or White (29%).

    “As described in Table 1, most applicants in our data are male (68%), Asian (58%), or White (29%).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The UCB model's training data can only be updated with outcomes of applicants selected by the model and also interviewed in practice.

    “As with the updating SL model, we update the UCB model's training data with the outcomes of applicants it has selected-D U CB t+1 = D U CB t ∪ (I U CB t ∩ I t )-subject to the limitation that we can only add applicants who are selected by the model and also interviewed in practice.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Incorporating exploration in our setting would lead our firm to interview twice as many under-represented minorities while more than doubling its predicted hiring yield.

    “Even though our UCB algorithm places no value on diversity in and of itself, incorporating exploration in our setting would lead our firm to interview twice as many under-represented minorities while more than doubling its predicted hiring yield.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The hiring process can be modeled as a contextual bandit problem.
    • The authors state this as an assumption: The relationship between context X and rewards follows a generalized linear form.

  • May combine two claims.

    Findingown data

    The static and updating SL models would both dramatically decrease Black and Hispanic representation, to approximately 2% and 5%, respectively.

    “The static and updating SL models, however, would both dramatically decrease Black and Hispanic representation, to approximately 2% and 5%, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The UCB model generates an increase in both hiring rates and diversity relative to the firm's existing hiring practices.

    “In contrast to our SL models-which substantially reduce the share of Black and Hispanic applicants who are interviewed-our UCB model generates an increase in both hiring rates and diversity, relative to the firm's existing hiring practices.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Hiring likelihood (receiving and accepting an offer) is a sufficient measure of applicant quality for the main analysis.
    • Rests on an assumption: The firm's application and hiring records are accurate and complete.

  • May combine two claims.

    Findingreasoning

    When the hiring potential of Hispanic applicants is increased throughout 2018, the UCB model quickly picks up on this change while the updating SL model is slower.

    “This same pattern can also be seen in Panel B of Figure 5, which plots the percentage of Hispanic applicants who are selected in the case where we increase the hiring potential of Hispanic applicants throughout 2018.The UCB model quickly picks up on this change while the updating SL model is slower.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.
    • Rests on an assumption: The simulated counterfactual changes in hiring potential for demographic groups are a valid representation of how hiring rates could change in 2018.

  • May combine two claims.

    Findingown data

    In our data, supervised learning algorithms substantially increase applicants' predicted hiring potential and decrease their demographic diversity relative to the firm's actual practices.

    “In our data, supervised learning algorithms substantially increase applicants' predicted hiring potential decrease their demographic diversity relative to the firm's actual practices.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Hiring likelihood (receiving and accepting an offer) is a sufficient measure of applicant quality for the main analysis.
    • Rests on an assumption: The firm's application and hiring records are accurate and complete.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    65% of interviewed applicants are men and 35% are women; this is largely similar to the gender composition of the overall applicant pool.

    “Panel A shows that 65% of interviewed applicants are men and 35% are women; this is largely similar to the gender composition of the overall applicant pool.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    Despite demographic differences, the updating SL and UCB models agree on a greater share of candidates relative to the human model, and there do not appear to be significant differences in overall hiring likelihoods when they disagree; if anything, the UCB model performs slightly better.

    “Finally, Panel C shows that, despite their demographic differences, the updating SL and UCB models agree on a greater share of candidates relative to the human model, and there do not appear to be significant differences in overall hiring likelihoods when they disagree: if anything, the UCB model performs slightly better.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Marginal high UCB score applicants are more likely to be Black, Hispanic, and female.

    “In Panels B through D, we show that marginal high score applicants are more likely to be Black, Hispanic, and female.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Blinding the UCB model to race, gender, and ethnicity reduces the share of selected Black or Hispanic applicants from 23% to 14%, still greater than human hiring (10%).

    “Blinding reduces the share of selected applicants who are Black or Hispanic, from 23% to 14%, although there is still greater representation relative to human hiring (10%).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records are accurate and complete.

  • May combine two claims.

    Findingown data

    Figure 3 shows that the updating SL model performs better than UCB in improving the hiring likelihoods of selected applicants, contrary to the theoretical prediction that exploration models should outperform greedy models in the long run.

    “First, Figure 3 shows that the updating SL model performs better than UCB in terms of improving the hiring likelihoods of selected applicants, in contrast to the theoretical prediction that models which incorporate exploration should outperform greedy models in the long run (Dimakopoulou et al., 2018b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Dimakopoulou et al., 2018b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Human SL score is negatively correlated with top rating and promotion among hired applicants.

    “Human SL Score -0.288** -0.286** -0.0707 -0.0781”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    The human SL model score is negatively signed and sometimes statistically significantly correlated with on-the-job performance ratings and promotion outcomes.

    “In all cases, we observe a negatively signed and sometimes statistically significant relationship: if anything, human recruiters are less likely to interview candidates who turn out to do well on the job.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: On-the-job performance ratings and promotion outcomes are valid measures of the quality firms seek to maximize.
    • Rests on an assumption: The firm's application and hiring records are accurate and complete.

  • May combine two claims.

    Findingown data

    Human recruiters select a similar proportion of Asian and Hispanic applicants (57% and 4%, respectively), but relatively more White and fewer Black applicants (34% and 5%, respectively).

    “Panel A of Figure 1 shows that, from this pool, human recruiters select a similar proportion of Asian and Hispanic applicants (57% and 4%, respectively), but relatively more White and fewer Black applicants (34% and 5%, respectively).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both dynamic models (updating SL and UCB) do better than the static SL model, suggesting substantial returns to increasing training data size.

    “Our results also suggest that there can be substantial returns to increasing the size of the training data we use: both dynamic models (updating SL and UCB) do better than the static SL model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the training data, only 296 of nearly 49,000 applicants are hired and have job performance ratings, making it difficult to accurately build a model predicting on-the-job performance.

    “However, of the nearly 49,000 applicants in our training data, only 296 are hired and have data on job performance ratings, making it difficult to accurately build such a model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The change in Black or Hispanic representation is accompanied by an increase in the proportion of interviewed candidates who are White from 34% to 51% and a decrease in the share who are Asian from 57% to 47%.

    “This change is accompanied by an increase in the proportion of interviewed candidates who are White (from 34% to 51%) and a decrease in the share who are Asian (57% to 47%).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    In simulations of declines for Black and Hispanic applicants, the UCB model is slower to respond but the proportion of selected Black or Hispanic applicants falls to zero within a couple months.

    “When we do this for Black and Hispanic applicants, the UCB model is slower to respond because it continues providing higher exploration bonuses to these groups.However, the proportion of selected Black or Hispanic applicants falls to zero within a couple months, as it learns about their simulated decline in hiring prospects.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The evaluation cohort (applicants from January 2019 to April 2019) is representative of applicants whose scores would be assessed over time.
    • Rests on an assumption: The firm's application and hiring records are accurate and complete.

  • May combine two claims.

    Findingown data

    The variation in screener pass rates is predictive of whether a given applicant is interviewed, but is not related to any of the applicant's covariates.

    “Appendix Table A.3 shows that this variation is predictive of whether a given applicant is interviewed, but is not related to any of the applicant's covariates.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Under the updating SL model, Black and Hispanic representation falls from 10% to under 5%, White representation increases from 34% to 40%, and Asian representation stays largely constant.

    “The updating SL model (Panel C) follows a similar pattern: Black and Hispanic representation falls from 10% to under 5%, White representation increases more modestly from 34% to 40%, and Asian representation stays largely constant.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    Despite their demographic differences, hiring outcomes for applicants selected by the SL and UCB models are comparable to each other, and much better than those selected by human recruiters.

    “We will discuss this extensively in the next section and provide evidence that, despite their demographic differences, hiring outcomes for applicants selected by our SL and UCB models are comparable to each other, and much better than those selected by human recruiters.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    ML models select applicants with higher average predicted hiring rates: 15% for static SL, 25% for UCB, and 30% for updating SL.

    “Our calculations indicate that, in all cases, ML models select applicants with higher average predicted hiring rates: 15% for static SL, 25% for UCB, and 30% for updating SL.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Under the UCB model, the Black share of selected applicants increases from 5% to 14%, and the Hispanic share from 4% to 10%.

    “In contrast, Panel D shows that the UCB model increases the Black share of selected applicants from 5% to 14%, and the Hispanic share from 4% to 10%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    Relative to humans, both SL models sharply reduce the share of Black and Hispanic candidates selected, while the UCB model sharply increases it.

    “Panels B-D describe our main result with respect to racial diversity: relative to humans, both SL models sharply reduce the share of Black and Hispanic candidates who are selected, while the UCB model sharply increases it.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At baseline, 54% of applicants in the test sample are Asian, 25% are White, 8% are Black, and 4% are Hispanic.

    “At baseline, 54% of applicants in our test sample are Asian, 25% are White, 8% are Black, and 4% are Hispanic.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Under the UCB model, the White share stays constant, while the Asian share falls from 57% to 41%.

    “The White share stays constant, while the Asian share falls from 57% to 41%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    All of the ML models are aligned in selecting more women than human recruiters, increasing their representation to 41% (static SL), 50% (updating SL), or 39% (UCB).

    “Unlike the case of race, all of our ML models are aligned in selecting more women than human recruiters, increasing their representation to 41% (static SL), 50% (updating SL), or 39% (UCB).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Findingown data

    In the simulation increasing Black applicants' hiring potential, the UCB model rapidly increases the share of Black candidates it selects and learns enough to always select them after about 800 candidates.

    “In strong contrast with the static SL model the UCB model rapidly increases the share of Black candidates it selects.In this simulation, the UCB model learns enough about the success of Black applicants to always select them after only a couple months (e.g. after seeing about 800 candidates).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The evaluation cohort (applicants from January 2019 to April 2019) is representative of applicants whose scores would be assessed over time.
    • Rests on an assumption: The firm's application and hiring records are accurate and complete.

  • May combine two claims.

    Findingown data

    Among interviewed applicants, human scores are uninformative about hiring likelihood; if anything the relationship is slightly negative.

    “We find that, among those who are interviewed, applicants' human scores are uninformative about their hiring likelihood; if anything this relationship is slightly negative.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    If both algorithms eventually observe enough examples to arrive at similar quality estimates, then both UCB and SL models will make the same interview decisions in the long run, because the UCB exploration bonus eventually goes to zero as training data increases.

    “If this were the case, then both UCB and SL models will make the same interview decisions in the long run; this is because the contribution of the UCB exploration bonus (as can be seen in Equation ( 6)) eventually goes to zero as the size of its training data increases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    There is scope to improve interview practices by either extending interview opportunities to a more appropriate set of candidates or reducing the number of interviews needed to achieve current hiring outcomes.

    “This suggests that there is scope improve interview practices by either extending interview opportunities to a more appropriate set of candidates, or reducing the number of interviews needed to achieve current hiring outcomes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    All of the paper's models are designed to maximize applicant quality, as defined by hiring rates, and have no additional preferences related to diversity.

    “All of our models are designed to maximize applicant quality, as defined by hiring rates, and have no additional preferences related to diversity.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The optimal solution to bandit problems incorporates some exploration.

    “Whereas the optimal solution to bandit problems is widely known to incorporate some exploration, supervised learning based algorithms engage in only in exploitation because they are designed to solve static prediction problems.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A growing literature has shown that supervised learning algorithms can more effectively identify high quality job candidates than human recruiters.

    “indeed, a growing literature has shown that supervised learning algorithms can more effectively identify high quality job candidates than human recruiters.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Theory predicts that exploration-focused models may select applicants with lower expected hiring potential in the short run relative to SL models, but should eventually minimize regret via more efficient learning.

    “theory predicts that while models that focus on exploration may end up selecting applicants with lower expected hiring potential in the short run (relative to SL models), they should eventually minimize regret via more efficient learning (Li et al., 2017; Dimakopoulou et al., 2018b).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Li et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Dimakopoulou et al., 2018b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    The human SL model may not perfectly predict actual human interview decisions and may be worse.

    “Additionally, our human SL model may not perfectly predict actual human interview decisions and may, in fact, be worse.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Algorithms have been shown to outperform human decision-makers across an expanding range of settings.

    “Algorithms have been shown to outperform human decision-makers across an expanding range of settings, from medical diagnosis to image recognition to game play.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The UCB and SL models share the same predicted quality estimate at time t=0 of the testing sample, based on the baseline model trained on the 2016-2017 sample.

    “At time t = 0 of the testing sample, our UCB and SL models share the same predicted quality estimate, which is based on the baseline model trained on the 2016-2017 sample.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    UCB-based algorithms have been shown to be asymptotically efficient in reducing expected regret.

    “algorithms based on this UCB approach have been shown to be asymptotically efficient in terms of reducing expected regret (Lai and Robbins, 1985; Li et al., 2017; Abbasi-Yadkori et al., 2019).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Lai and Robbins, 1985 source
    • cites Li et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Abbasi-Yadkori et al., 2019

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The paper's analyses of diversity and hiring likelihood are based on out-of-sample model performance in the test data.

    “Our models are built on the training data and our analyses of diversity and hiring likelihood are based on out-of-sample model performance in the test data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Standard and contextual bandit UCB algorithms asymptotically minimize expected regret.

    “Standard and contextual bandit UCB algorithms have been shown to be optimal in the sense that they asymptotically minimize expected regret (Lai and Robbins, 1985; Abbasi-Yadkori et al., 2019; Li et al., 2017)”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Lai and Robbins, 1985 source
    • cites Abbasi-Yadkori et al., 2019
    • cites Li et al., 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Algorithms are better at selecting candidates who are more likely to receive and accept an offer; using these algorithms, the firm could hire the same number of people while conducting fewer interviews.

    “These results suggest that algorithms are better at selecting candidates who are more likely to receive and accept an offer; using these algorithms, the firm could hire the same number of people while conducting fewer interviews.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is no selection on unobservables in the decomposition approach.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Black, Hispanic, and female applicants are less likely to be employed in high-income jobs.

    “it is also the case that Black, Hispanic, and female applicants are less likely to be employed in high-income jobs, meaning that they will also appear less often in the historical datasets used to train hiring algorithms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    All quality measures considered (hiring rates, performance ratings, and promotion rates) are based on manager discretion and potentially subject to evaluation and mentoring biases.

    “Finally, we note that all of the quality measures we consider-hiring rates, performance ratings, and promotion rates-are based on the discretion of managers and therefore potentially subject to various types of evaluation and mentoring biases (Rivera and Tilcsik, 2019; Quadlin, 2018; Castilla, 2011).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Rivera and Tilcsik, 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Quadlin, 2018 source
    • cites Castilla, 2011 source

  • Classified as probably not a substantial claim.

    Premisereasoning

    ML tools that incorporate exploration are designed to solve dynamic prediction problems involving learning from sequential actions.

    “By contrast, ML tools that incorporate exploration are designed to solve dynamic prediction problems that involve learning from sequential actions: in the case of hiring, these algorithms value exploration because this allows for more optimal choices in the future.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Raghavan et al. (2019) reports that no firms currently provide information on the validation of their models.

    “We would ideally like to compare our AUC to those of commercial providers, butRaghavan et al. (2019) reports that no firms currently provide information on the validation of their models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Raghavan et al. (2019) source

  • May combine two claims.

    Interpretationreasoning

    As UCB beliefs update, it selects a greater share of White and Asian applicants because the exploration bonuses it grants to other groups are no longer large enough to matter.

    “In this way, exploration bonuses initially come at the expense of maximizing hiring yield; however, as UCB beliefs update, it selects a greater share of White and Asian applicants-the exploration bonuses it grants to other groups are no longer large enough to matter.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Interpretationreasoning

    Following UCB recommendations can increase both hiring yield and diversity relative to the firm's present policies.

    “Again, these results suggest that following UCB recommendations can increase both hiring yield and diversity relative to the firm's present policies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Under all ML models, representation for any group can go to zero if their realized outcomes fall sufficiently, unlike a quota-based system.

    “We note that these selection patterns differ from a”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The interview policy defined by Ĩ would increase quality and diversity on the margin, relative to the firm's current practices.

    “As such, the interview policy defined by Ĩ would increase quality and diversity on the margin, relative to the firm's current practices.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    ML tools can increase the hiring yield of applicants but may have very different implications for demographic representation.

    “Our results show that ML tools can be used to increase the hiring yield of applicants, but may have very different implications for demographic representation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Hiring likelihood (receiving and accepting an offer) is a sufficient measure of applicant quality for the main analysis.

  • May combine two claims.

    Study design and conductown data

    The analysis is limited by the short time span of the test period and by not running an experiment in which ML-selected candidates who are not otherwise interviewed can actually be interviewed.

    “Our analysis however, is limited by the short time span of our test period and the fact that we are not running an experiment in which we can actually interview the candidates that an ML selects (who are not otherwise interviewed).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The UCB algorithm learns almost as quickly as the updating SL, but selects fewer Asian and White applicants because it is still allocating higher exploration bonuses to other groups.

    “The reasons for this is apparent when comparing selection decisions under the full UCB model and UCB beliefs only: the UCB algorithm learns almost as quickly as the updating SL, but selects fewer Asian and White applicants because it is still allocating higher exploration bonuses to other groups.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • May combine two claims.

    Interpretationown data

    The human SL model used in Section 4.2 is trained only on features the paper observes and may miss unobservables that humans may use to correctly predict hiring likelihood, biasing results on the interviewed-only sample.

    “For example, the human SL model we use in Section 4.2 is trained only on features we observe and may therefore miss unobservables that humans may use to correctly predict hiring likelihood, biasing our results on the interviewed-only sample.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Li et al. (2017) show that the GLM-UCB algorithm has a regret bound of order Õ(d √ T ), where d is the number of covariates and T is the number of rounds.

    “In particularLi et al. (2017) show that the GLM-UCB algorithm has a regret bound of order Õ(d √ T ), where d is the number of covariates and T is the number of rounds.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al. (2017)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The best choice of algorithm for a given situation depends on the number of possible actions and contexts, as well as on assumptions regarding the parametric form relating context to reward.

    “In particular, the best choice of algorithm for a given situation will depend on the number of possible actions and contexts, as well as on assumptions regarding the parametric form relating context to reward.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The patterns found may not fully generalize across sectors or across firms that vary in their ability or propensity to adopt ML tools.

    “We focus on the use of ML to hire for high skill professional services firms; the patterns we find may not fully generalize across sectors or across firms that vary in their ability or propensity to adopt ML tools.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The slightly weaker performance of the UCB model may be explained by exploration trading off higher performance in earlier periods for increased learning in later periods.

    “The slightly weaker performance of the UCB model may be explained by the fact that an emphasis on exploration means that the UCB algorithm trades off higher performance in earlier periods for increased learning in later periods.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The instrument (jack-knife mean-interview rate for the recruiter assigned to an applicant) is positively associated with being interviewed and not significantly associated with baseline applicant characteristics.

    “JK interview rate 0.0784*** 0.000767 -0.000234 0.00812 -0.00939 -0.000348 0.00987 -0.00888”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    All of the ML models used are designed solely to maximize applicant hiring potential; they do not have a preference for demographic diversity built into their design.

    “First, we reiterate that all of the ML models we use are designed solely to maximize applicant hiring potential-they do not have a preference for demographic diversity built into their design.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Tests in the AUC range of 0.75-0.85 have intermediate to good discriminative properties depending on the specific context and shape of the curve.

    “Although a”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fischer et al., 2013 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The AUC is identical to the result of a Wilcoxon or Mann-Whitney U test.

    “The AUC is identical to the result of a Wilcoxon or Mann-Whitney U test(Hanley and McNeil, 1982).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hanley and McNeil, 1982 source

  • Probably not a substantial claim: may not pass S4 (based).

    Premiseown data

    Recruiters make decisions on the basis of CV variables that the paper, for the most part, also observes.

    “Although this is a strong assumption, we believe it is plausible in our setting because recruiters make decisions on the basis of CV variables that we, for the most part, also observe.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: There is no selection on unobservables in the sample, so hiring likelihood for ML-selected applicants who were not interviewed can be inferred from observed hiring outcomes of applicants with similar covariates who were interviewed.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Li et al. (2017) generalizes the classic general LinUCB algorithm for nonlinear relationship between context and reward.

    “Li et al. (2017) generalizes the classic general LinUCB algorithm for nonlinear relationship between context and reward.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al. (2017)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisenone given

    Recruiters do not meet, speak with, or otherwise interact with the candidates.

    “Importantly, they do not meet, speak with, or otherwise interact with the candidates.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is no selection on unobservables in the sample, so hiring likelihood for ML-selected applicants who were not interviewed can be inferred from observed hiring outcomes of applicants with similar covariates who were interviewed.

  • May combine two claims.

    Interpretationown data

    Following a contextual bandit algorithm that prioritizes exploration rather than supervised learning can improve the average hiring potential of selected candidates while increasing representation of Black and Hispanic applicants.

    “In particular, we show that-by following a contextual bandit algorithm that prioritizes exploration rather than traditional supervised learning algorithms that focus on exploitation-firms can improve the average hiring potential of the candidates they select to be interviewed, while at the same time increasing the representation of Black and Hispanic applicants.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Hiring likelihood (receiving and accepting an offer) is a sufficient measure of applicant quality for the main analysis.
81 of 81

  • The authors state this as an assumption.

    The hiring process can be modeled as a contextual bandit problem.

    “We model the firm's interview decision as a contextual bandit problem.”

  • The authors state this as an assumption.

    The relationship between context X and rewards follows a generalized linear form.

    “To make our model tractable, we follow Li et al. (2010, 2017) and assume that relationship between context X and rewards (e..g hiring potential) follows a generalized linear form.”

  • The authors state this as an assumption.

    There is no selection on unobservables in the decomposition approach.

    “This approach assumes that there is no selection on unobservables.”

  • The authors state this as an assumption.

    There is no selection on unobservables in the sample, so hiring likelihood for ML-selected applicants who were not interviewed can be inferred from observed hiring outcomes of applicants with similar covariates who were interviewed.

    “Doing so requires us to assume that there is no selection on unobservables in our sample so that we can infer hiring likelihoods for ML-selected applicants who were not actually interviewed in practice using observed hiring outcomes from applicants with similar covariates who were interviewed:”
    “Assuming no selection on unobservables, we can proxy for the term”

  • The authors state this as an assumption.

    Common support holds: for every ML-selected applicant with covariates X, there is an interviewed applicant with the same covariates.

    “Practically, this approach requires common support: for every ML-selected applicant with covariates X, we must be able to find an applicant with the same covariates who is interviewed.”

  • The authors state this as an assumption.

    E[H|X] can be expressed as a generalized linear function µ(Xθ*).

    “we assume that E[H|X] can be expressed as a generalized linear function µ(X θ * ).”

  • The authors state this as an assumption.

    µ is a logistic function.

    “we assume that µ is a logistic function”

  • The paper relies on this without support.

    On-the-job performance ratings and promotion outcomes are valid measures of the quality firms seek to maximize.

    “Appendix Table A.4 presents these results using two measures of performance: on the job performance ratings from an applicant's first mid-year review, and an indicator for whether an applicant has been promoted.”

  • The paper relies on this without support.

    Hiring likelihood (receiving and accepting an offer) is a sufficient measure of applicant quality for the main analysis.

    “One concern with our analysis so far is that our measure of quality-likelihood of receiving and accepting an offer-may not be the ultimate measure of quality that firms are seeking to maximize.”

  • The paper relies on this without support.

    The evaluation cohort (applicants from January 2019 to April 2019) is representative of applicants whose scores would be assessed over time.

    “Specifically, we take the actual set candidates who applied between January 2019 and April 2019 (hereafter, the”

  • The paper relies on this without support.

    The firm's application and hiring records are accurate and complete.

    “All data come from the firm's application and hiring records.”

  • The paper relies on this without support.

    The firm's application and hiring records accurately capture applicants' demographic characteristics, education histories, work experience, and hiring outcomes.

    “All data come from the firm's application and hiring records.”

  • The paper relies on this without support.

    The simulated counterfactual changes in hiring potential for demographic groups are a valid representation of how hiring rates could change in 2018.

    “Panels C and D of Figure 5 plot outcomes for Asian and White applicants under the counterfactual that these groups of increasing hiring likelihood in 2018.”
13 of 13

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Updating SL's White share is reported as both 40% and 34.6%

    Major

    The text says updating SL raises White representation to 40%, but the reported composition for that model gives 34.6%.

  • Static SL's predicted outcome changes from hiring to interviewing

    Minor

    The static SL model is introduced as predicting hiring conditional on an interview, but its construction then says the combined model predicts the probability of receiving an interview.

  • Test-sample racial shares differ from Table 1

    Minor

    The text gives the test sample as 54% Asian, 25% White, and 8% Black, whereas Table 1 reports 59%, 28%, and 9%, respectively.

  • Updating SL's female share is reported as both 50% and 46.5%

    Minor

    The stated 50% female share under updating SL does not match the 46.5% shown in the gender-composition figure.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. "Semiparametric instrumental variable estimation of treatment response models," Journal of econometrics, 2003, 113 (2), 231-263.
  2. Christopher Avery, and Peter Zemsky, "Mentoring and Diversity," American Economic Review, September 2000, 90 (4), 765-786.

Cited by

Sorted by citations

Related

Authors
Li, D., Raymond, L., & Bergman, P.
Affiliations
National Bureau of Economic Research (US)
ORCID
Li, Danielle
Venue
National Bureau of Economic Research
Publisher
National Bureau of Economic Research
Published
2020-08
DOI
10.3386/w27736
OpenAlex
W3082194350
Cited by
64
References
69
Citations by year
2026: 5 · 2025: 14 · 2024: 13 · 2023: 11 · 2022: 7 · 2021: 7 · 2020: 4 · 2019: 2
Type
report
Language
en
Open access
Locations
National Bureau of Economic Research (publishedVersion)
Topics
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
Keywords
hiring algorithms, exploration-exploitation balance, contextual bandits, resume screening, demographic diversity, hiring discrimination, algorithmic fairness, supervised learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-30.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-30.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review