CoChat
Sign Up
Cited by 887Names, Identity, and Discrimination ResearchExperimental Behavioral Economics StudiesSocial and Intergroup Psychology

Meta-analysis of field experiments shows no change in racial discrimination in hiring over time

Open access
Quillian, Lincoln, Pager, Devah, Hexel, Ole, Midtbøen, Arnfinn H.·2017·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1706255114PMC PMC5642692
Open this paper in CoChat — free accountOpen PDF

Abstract

Significance: Many scholars have argued that discrimination in American society has decreased over time, while others point to persisting race and ethnic gaps and subtle forms of prejudice. The question has remained unsettled due to the indirect methods often used to assess levels of discrimination. We assess trends in hiring discrimination against African Americans and Latinos over time by analyzing callback rates from all available field experiments of hiring, capitalizing on the direct measure of discrimination and strong causal validity of these studies. We find no change in the levels of discrimination against African Americans since 1989, although we do find some indication of declining discrimination against Latinos. The results document a striking persistence of racial discrimination in US labor markets.

AI snapshot

Written from the paper’s full text.

TL;DR

A meta-analysis of all available US field experiments of hiring discrimination found no change over time in discrimination against African Americans since 1989, though there was modest evidence of declining discrimination against Latinos. Whites received substantially more callbacks than equally qualified African American and Latino applicants.

Key findings

  • Across studies since 1989, white applicants received on average 36% more callbacks than equally qualified African Americans (95% CI: 25–47% more) and 24% more callbacks than Latinos (95% CI: 15–33% more).
  • For African Americans, the paper found no change in hiring discrimination over the past 25 years; the 95% CI of the annual slope from 1989–2015 was −0.007 to 0.015, ruling out all but a slow decline, with the point estimate indicating no decline.
  • For Latinos, the trend suggested a possible decline, from whites receiving 30% more callbacks than Latinos in 1990 to 15% more in 2010, but this was only marginally significant (P = 0.099) and based on only 9 Latino field experiments.
  • Accounting for applicant education, gender, study method, occupational groups, local labor market conditions, and other controls did little to alter the African American result; including unpublished studies also did little to affect estimates.

Methods

  • The authors conducted a meta-analysis of every available US field experiment of hiring discrimination, including resume audits and in-person audits, with fieldwork through December 2015.
  • The full search located 28 studies representing 55,842 applications for 26,326 positions; the core analysis since 1989 used 24 studies with 30 estimates, representing 54,318 applications for 25,517 positions.
  • They used random-effects meta-regression, regressing the log discrimination ratio—the ratio of white callback rates to African American or Latino callback rates—on year of fieldwork, with controls for study and applicant characteristics.

Limitations

  • The paper states that results pertain only to discrimination at the point of hire, not later employment outcomes such as wage setting or termination, and do not address possible declines during the 1960s or early 1970s.
  • The paper states that the small number of Latino field experiments (n = 9) limits controls and precision, and that the overall sample of African American studies since 1990 (n = 21) limits the ability to detect statistical significance for covariates.
  • The paper notes publication bias as a potential concern but reports that including 11 unpublished field experiments did little to affect estimates; it also states that studies focused on racial discrimination showed no more discrimination than studies where race was secondary.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.1706255114

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 6 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The 95% confidence interval of the annual slope for discrimination against African Americans from 1989 to 2015 is -0.007 to 0.015.

    “The 95% confidence interval of the slope 1989-2015 is -0.007 to 0.015.”
    Quote in the paperSubstantial claim
  • Findingown data

    The point estimate suggests a decline in discrimination against Latinos from whites receiving 30% more callbacks than Latinos in 1990 to 15% more callbacks in 2010.

    “The point estimate suggests a decline from whites receiving 30% more callbacks than Latinos in 1990 to 15% more callbacks in 2010 (1.30 vs. 1.15).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.
    • Rests on an assumption: The year in which data were gathered is a valid and comparable indicator of the time trend in hiring discrimination.
  • Findingown data

    The basic result of no decline in discrimination against African Americans over time holds using alternative measures (difference in proportions and odds ratio).

    “Our basic result-no decline in discrimination against African Americans over time-holds using both of the alternative measures, whereas evidence of a decline in discrimination for Latinos appears somewhat stronger with the difference in proportions or the odds ratio.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The true effect of race on callback rates varies across studies and situations, such as by year, occupation, and applicant skill level.
    • The authors state this as an assumption: The true effects of race on callback rate in each situation estimated by each study vary between studies with a normal distribution around a mean effect.
    • The authors state this as an assumption: Study-level heterogeneity follows a normal distribution around the linear predictor.
  • Findingown data

    There is modest evidence of a decline in hiring discrimination against Latinos over time.

    “We observe no change in the level of hiring discrimination against African Americans over the past 25 years, although we find modest evidence of a decline in discrimination against Latinos.”
    • Also stated in: § Results · hedged
      “Here, we see the line slopes downward, indicating a possible decline in discrimination, although this trend is outside of conventional levels of significance (P = 0.099).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.
    • Rests on an assumption: The year in which data were gathered is a valid and comparable indicator of the time trend in hiring discrimination.
  • Findingown data

    Data are sparse before 1989 (only four studies before 1989).

    “We focus on this period because the data are sparse before this period (only four studies before 1989) and because our reading of the early studies indicates key methodological differences among these early studies that may affect their results.”
    Quote in the paperSubstantial claim
  • May combine two claims. 13
  • Classified as probably not a substantial claim. 12

  • May combine two claims.

    Findingown data

    Overall, our search located 34 studies that were US-based field experiments of hiring, included contrasts between white and nonwhite applicant profiles that were on-average equivalent in their labor-market relevant characteristics.

    “Overall, our search located 34 studies that were US-based field experiments of hiring, included contrasts between white and nonwhite applicant profiles that were on-average equivalent in their labor-market relevant characteristics (e.g., education, experience level in the labor market).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In models adding controls for applicant attributes, region and area unemployment rates, and occupational categories, coefficients for the time trend are close to zero, ranging from an estimated increase of 0.1% per year (0.001) to an increase of 1.3% per year (0.013), suggesting little change in discrimination against African Americans over time.

    “In each case, we see coefficients for the time trend that are close to zero-ranging from an estimated increase of 0.1% per year (0.001) to an increase of 1.3% per year (0.013) -suggesting little change in the level of discrimination facing African Americans over time.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.
    • Rests on an assumption: The year in which data were gathered is a valid and comparable indicator of the time trend in hiring discrimination.

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Studies have fieldwork periods ranging from 1972 to 2015 for African Americans and 1989 to 2015 for Latinos.

    “Studies have fieldwork periods range from 1972 to 2015 for African Americans and 1989 to 2015 for Latinos.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Our remaining 28 studies yielded 24 estimates of discrimination against African Americans and 9 against Latinos relative to whites.

    “Our remaining 28 studies yielded 24 estimates of discrimination against African Americans and 9 against Latinos relative to whites.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With job offer as the outcome instead of callback, the trend line for African Americans slants more downward but remains close to zero (-0.008) and statistically nonsignificant.

    “With this modification, the trend line for African Americans slants more downward, but is still close to zero (-0.008) and statistically nonsignificant.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.
    • Rests on an assumption: The year in which data were gathered is a valid and comparable indicator of the time trend in hiring discrimination.

  • May combine two claims.

    Findingown data

    Eliminating applicant profiles with a fictitious criminal background or disability results in a trend line slanting slightly more upward, providing less evidence for a downward trend.

    “The modified results show the trend line slanting slightly more upward, providing less evidence to support a downward trend than the results including a more heterogeneous set of applicant characteristics.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.
    • Rests on an assumption: The year in which data were gathered is a valid and comparable indicator of the time trend in hiring discrimination.

  • May combine two claims.

    Findingown data

    Studies in which racial discrimination was the focus of the analysis show no more discrimination than studies in which other characteristics were the main focus, reducing concerns over publication bias.

    “studies in which racial discrimination was the focus of the analysis (and for which there may be more pressure to demonstrate a positive effect) show no more discrimination than studies in which other characteristics were the main focus”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    In later models, we include a more extensive set of predictors to control for other factors that might confound the time trend.

    “In later models, we include a more extensive set of predictors to control for other factors that might confound the time trend.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The formula for unpaired groups will slightly overestimate the SE of the effect for studies that are paired but treated as unpaired, underweighting these studies and slightly inflating the overall cross-study SE.

    “This formula will slightly overestimate the SE of the effect for studies that are paired but we treat as unpaired due to lack of information about the outcomes at the pair level, underweighting these studies a bit in computing the overall effect, and slightly inflating the overall cross-study SE.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    From an accountability standpoint, discrimination is less easily detected, and therefore less costly to employers, at the point of hire.

    “Likewise, from an accountability standpoint, discrimination is less easily detected, and therefore less costly to employers, at the point of hire (26).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisereasoning

    The field experimental method has high causal (internal) validity because it benefits from aspects of experimental design.

    “The field experimental method is a design with high causal (internal) validity because it benefits from aspects of experimental design.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The evidence rules out all but a slow decline in discrimination against African Americans, with the most likely estimate indicating no decline at all.

    “We conclude that this evidence rules out all but a slow decline in discriminationwith the most likely estimate being the point estimate, which indicates no decline in discrimination at all.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each study was coded independently by two raters, with disagreement resolved by the first author.

    “Each study was coded independently by two raters, with disagreement resolved by the first author.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Social psychological theories predict hiring to be most vulnerable to the influence of racial bias, given that objective information is limited or unreliable.

    “Social psychological theories would predict hiring to be most vulnerable to the influence of racial bias, given that objective information is limited or unreliable (23)(24)(25).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites (23)(24)(25)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Random effects are recommended whenever there is reason to believe that the effect in question is likely to vary as a function of design features of the study, rather than representing a single underlying effect that is constant over the whole population.

    “Random effects are recommended whenever there is reason to believe that the effect in question is likely to vary as a function of design features of the study, rather than representing a single underlying effect that is constant over the whole population.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The results do not address whether hiring discrimination may have substantially dropped in the 1960s or early 1970s.

    “We note that our results do not address the possibility that hiring discrimination may have substantially dropped in the 1960s or early 1970s, during the civil rights era when many forms of direct discrimination were outlawed, as some evidence suggests (1).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The paper's estimates suggest that levels of discrimination remain largely unchanged, at least at the point of hire, contrary to claims of declining discrimination in American society.

    “Contrary to claims of declining discrimination in American society, our estimates suggest that levels of discrimination remain largely unchanged, at least at the point of hire.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.
    • Rests on an assumption: The year in which data were gathered is a valid and comparable indicator of the time trend in hiring discrimination.

  • May combine two claims.

    Study design and conductown data

    The meta-analysis includes every available field experiment of hiring discrimination against African Americans or Latinos (n = 28), representing 55,842 applications submitted for 26,326 positions.

    “We perform a meta-analysis of every available field experiment of hiring discrimination against African Americans or Latinos (n = 28).”
    • Also stated in: § Abstract · plain
      “Together, these studies represent 55,842 applications submitted for 26,326 positions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Contrasts of hiring outcomes between race or ethnic groups were made for groups that were on average equivalent in their labor market relevant characteristics.

    “We also required that contrasts of hiring outcomes between race or ethnic groups were made for groups that were on average equivalent in their labor market relevant characteristics, since otherwise discrimination estimates are confounded with the difference in nonracial characteristics.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Otherwise discrimination estimates are confounded with the difference in nonracial characteristics.

    “We also required that contrasts of hiring outcomes between race or ethnic groups were made for groups that were on average equivalent in their labor market relevant characteristics, since otherwise discrimination estimates are confounded with the difference in nonracial characteristics.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The initial coding rubric was reviewed and updated by all authors for completeness and subsequently refined as coding progressed.

    “The initial rubric was reviewed and updated by all authors of this study for completeness.It was subsequently refined as coding progressed.”
    Quote in the paperNot a substantial claim
21 of 21

  • The paper relies on this without support.

    The racial identifiers used in the original field experiments (e.g., racially identifiable names or tester race) validly signal race to employers and do not differ systematically on other employment-relevant characteristics.

    “In the typical resume audit, clues indicating race (such as a racially identifiable name) are randomly assigned to otherwise similar resumes, allowing for treatment and control groups to be equated through randomization.”

  • The paper relies on this without support.

    The year in which data were gathered is a valid and comparable indicator of the time trend in hiring discrimination.

    “We analyze the relationship of discrimination ratios to years in which the data were gathered to provide an estimate of the trend in discrimination.”

  • The authors state this as an assumption.

    The true effect of race on callback rates varies across studies and situations, such as by year, occupation, and applicant skill level.

    “This is surely the case in our analysis, as we expect that the level of racial discrimination may depend on the year of the study, the situation the study considers (e.g., the occupational categories), the skill level of the applicants, and so on.”

  • The authors state this as an assumption.

    The true effects of race on callback rate in each situation estimated by each study vary between studies with a normal distribution around a mean effect.

    “More formally, random-effects meta-analysis allows the true effects of race on the callback rate in each situation estimated by each study, θ i , to vary between studies by assuming that they have a normal distribution around a mean effect, θ.”

  • The authors state this as an assumption.

    Study-level heterogeneity follows a normal distribution around the linear predictor.

    “The model assumes the study-level heterogeneity follows a normal distribution around the linear predictor:”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Quillian et al.'s résumé audit studies found that race-associated names produce callback differentials equivalent to approximately d = 0.10-0.30.

    “In Quillian et al. [74]'s résumé audit studies, race-associated names produce callback differentials equivalent to approximately d = 0 . 10˘0 . 30”

    From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [74]

    The Coding and Selection of Analysis section says résumé audits typically signal race through race-typed names, and the Results report callback differences by race. But the paper reports those differences as relative callback percentages—36% for African Americans and 24% for Latinos compared with white applicants—not as approximately d = 0.10–0.30. The claimed standardized-effect range is not supported by our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • There has been no change in discrimination against African Americans in the United States over the past 25 years.

    “Quillian et al. (2017) find that there has been no change over the past 25 years in discrimination against African Americans in the United States.”

    From Comparative Perspectives on Racial Discrimination in Hiring: The Rise of Field Experiments (2021) · cites this paper as Quillian et al. (2017)

    The Results report no change over the past 25 years in hiring discrimination against African Americans in the United States. The Discussion explicitly limits the findings to discrimination at the point of hire; the cited paper does not establish that discrimination more broadly was unchanged. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • The findings align with existing work on the persistence of gender segregation and racial inequality in the U.S. labor market.

    “Both of these findings align with existing work on the persistence of gender segregation (Zhu and Grusky 2022) and racial inequality (Quillian et al. 2017) in the U.S. labor market.”

    From Getting a Foot in the Door: A Meta-Analysis of U.S. Audit Studies of Gender Bias in Hiring (2025) · cites this paper as Quillian et al. 2017

  • Our research is based on a synthesis of only correspondence experiments, whereas some earlier meta-studies also included in-person audits.

    “First of all, our research is based on a synthesis of only correspondence experiments, whereas some earlier meta-studies that focused on specific discrimination grounds have also included in-person audits to paint a broader picture of hiring discrimination (e.g.Quillian et al., 2017).”

    From The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments (2023) · cites this paper as Quillian et al., 2017

  • Tests for publication bias reported in the supplemental material of Quillian et al. (2017, 2019) find either no evidence of publication bias in field experiments on hiring discrimination or that its effects on estimates are small.

    “Tests for publication bias discussed in the supplemental material ofQuillian et al. (2017Quillian et al. ( , 2019)), however, find either no evidence of publication bias or that its effects on estimates is small.”

    From Comparative Perspectives on Racial Discrimination in Hiring: The Rise of Field Experiments (2021) · cites this paper as Quillian et al. (2017

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Overall and post-1989 African-American estimate counts conflict

    Minor

    The paper reports 21 African-American estimates after excluding the four pre-1989 studies, which it describes as African-American resume audits. That implies 25 African-American estimates overall, not 24.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. D Wessel Fear of bias suits may be affecting hiring decisions. Wall Street Journal section A p 2. (September 11 2003).

Cited by

Sorted by citations

Related

Authors
Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H.
Affiliations
  • Northwestern University (US)
  • Harvard University (US)
  • Institute for Social Research (NO)
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2017-09-12
Volume
114
Issue
41
Pages
10870-10875
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1706255114
PMC
PMC5642692
OpenAlex
W2756054281
Cited by
887
References
36
Citations by year
2026: 88 · 2025: 121 · 2024: 111 · 2023: 138 · 2022: 130 · 2021: 104 · 2020: 76 · 2019: 79 · 2018: 29 · 2017: 4 · 2015: 1 · 2014: 1
FWCI
13.97
Type
journal article
Language
en
Open access
Licence
www.pnas.org/site/misc/userlicense.xhtml (vor, 210 d)
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • BIBSYS Brage (BIBSYS (Norway)) (acceptedVersion)
  • HAL (submittedVersion, cc-by)
  • NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
Funders
  • Russell Sage Foundation (RSF) (88-15-06)
  • Northwestern Institute for Policy Research (N/A)
Topics
  • Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
  • Social Sciences › Safety Research › Experimental Behavioral Economics Studies
  • Social Sciences › Sociology and Political Science › Social and Intergroup Psychology
Keywords
racial discrimination in hiring, callbacks, African Americans, Hispanic Americans, labor market discrimination, field experiments, applicant gender
MeSH terms
Humans, Personnel Selection, Racism, Social Discrimination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-04.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-04.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review