A large-scale analysis of racial disparities in police stops across the United States

Classified as probably not a substantial claim.

The threshold test has been proposed as a more robust means for detecting discrimination than outcome tests.

“the threshold test has been proposed as a more robust means for detecting discrimination 7,22 .”

• cites [7] Simoiu, C., Corbett-Davies, S. & Goel, S. The problem of infra-marginality in outcome tests for discrimination. Ann. Appl. Stat. 11, 1193–1216 (2017). sourceSupported
• cites [22] Pierson, E., Corbett-Davies, S. & Goel, S. In International Conference on Artificial Intelligence and Statistics (eds Storkey, A. & Fernando Perez-Cruz, F.) 96–105 (PMLR, 2018).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Thresholds cannot be identified from observed data alone, so inferences depend on the Bayesian model's functional form and prior distributions.

“thresholds cannot be identified by the observed data alone 7 , and so inferences are dependent on the specific functional form of the underlying Bayesian model, including the prior distributions.”

• cites [7] Simoiu, C., Corbett-Davies, S. & Goel, S. The problem of infra-marginality in outcome tests for discrimination. Ann. Appl. Stat. 11, 1193–1216 (2017). sourcePartly supported

  Section 2.3, “Inferring search thresholds” (page unavailable), specifies a Bayesian model with priors, supporting the claim that inference uses modeling assumptions. But a passage reproduced under “Conclusion” says the model would be identified by priors rather than data *without* certain constraints; it does not say thresholds cannot be identified from observed data under the specified model. The paper also says its results are robust to the exact choice of priors, qualifying the claimed dependence on them. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The Bayesian threshold model's functional form and prior distributions are adequate for identifying search thresholds.

May combine two claims.

In the six municipal police departments analysed, search rates were 9.5% for black drivers, 7.2% for Hispanic drivers, and 3.9% for white drivers.

“in the six municipal police departments we consider, the search rates were 9.5% (95% CI 9.4-9.6%)for black drivers, 7.2% (95% CI 7.0-7.3%)for Hispanic drivers and 3.9% (95% CI 3.8-3.9%)for white drivers.”

May combine two claims.

In state patrol stops, the white-Hispanic hit-rate gap was 7.6% and the white-black gap was 2.6%, both statistically significant.

“the gap in hit rates between white and Hispanic drivers was 7.6% (95% CI 6.7-8.6%,P < 0.001), and the gap in hit rates between white and black drivers was 2.6% (95% CI 1.9-3.4%,P < 0.001).”

May combine two claims.

The estimated city-level search-threshold gaps between white and non-white drivers were large and statistically significant, with 95% credible intervals of (-6.4, -4.4%) for Hispanic-white and (-6.1, -4.0%) for black-white.

“The estimated gaps in search thresholds between white and non-white drivers are large and statistically significant: the 95% credible interval for the Hispanic-white difference is (-6.4,-4.4%) and the corresponding interval for the black-white difference is (-6.1, -4.0%).”

May combine two claims.

Aggregated across state patrol stops, contraband was found in 32.0% of searches of white drivers, 24.3% of searches of Hispanic drivers, and 29.4% of searches of black drivers.

“Aggregating across state patrol stops, contraband was found in 32.0% (95% CI 31.6-32.4%) of searches of white drivers compared to 24.3% (95% CI 23.5-25.2%) of searches of Hispanic drivers and 29.4% (95% CI 28.7-30.0%) of searches of black drivers.”

May combine two claims.

Aggregated across municipal police departments, contraband was found in 18.2% of searches of white drivers, 11.0% of searches of Hispanic drivers, and 13.9% of searches of black drivers.

“aggregating across municipal police departments, contraband was found in 18.2% (95% CI 17.8-18.7%) of searches of white drivers compared to 11.0% (95% CI 10.6-11.5%) of searches of Hispanic drivers and 13.9% (95% CI 13.7-14.2%) of searches of black drivers.”

May combine two claims.

In those municipal departments, the black-white search-rate gap was 5.6% and the Hispanic-white gap was 3.3%, both statistically significant.

“The gap in municipal search rates between black and white drivers was 5.6% (95% CI 5.5-5.7%,P <0.001) and the gap in search rates between Hispanic and white drivers was 3.3% (95% CI 3.1-3.5%,P <0.001).”

May combine two claims.

Across states, inferred search thresholds were 20.9% for white drivers, 16.0% for black drivers, and 13.9% for Hispanic drivers.

“Similarly across states, the inferred threshold for white drivers is 20.9% compared to 16.0% for black drivers and 13.9% for Hispanic drivers.”

May combine two claims.

Black drivers had higher annual per-capita stop rates than white drivers in both state patrol and municipal police stops.

“among state patrol stops, the annual per-capita stop rate for black drivers was 0.10 compared to 0.07 for white drivers; and among municipal police stops, the annual per-capita stop rate for black drivers was 0.20 compared to 0.14 for white drivers.”

May combine two claims.

In those state patrol agencies, the black-white search-rate gap was 2.4% and the Hispanic-white gap was 2.2%, both statistically significant.

“the gap in search rates between black and white drivers was 2.4% (95% CI 2.3-2.5%,P < 0.001) and the gap in search rates between Hispanic and white drivers was 2.2% (95% CI 2.1-2.2%,P < 0.001).”

May combine two claims.

Hispanic drivers had lower annual per-capita stop rates than white drivers in both state patrol and municipal police stops.

“For Hispanic drivers, however, we found that stop rates were lower than for white drivers: 0.05 for stops conducted by state patrol (compared to 0.07 for white drivers) and 0.09 for those conducted by municipal police departments (compared to 0.14 for white drivers).”

May combine two claims.

In the eight state patrol agencies analysed, search rates were 4.3% for black drivers, 4.1% for Hispanic drivers, and 1.9% for white drivers.

“In the eight state patrol agencies we consider, search rates were 4.3% (95% CI 4.2-4.4%)for black drivers, 4.1% (95% CI 4.0-4.1%)for Hispanic drivers and 1.9% (95% CI 1.9-1.9%)for white drivers.”

May combine two claims.

In municipal police departments, the white-Hispanic hit-rate gap was 7.2% and the white-black gap was 4.3%, both statistically significant.

“the gap in hit rates between white and Hispanic drivers was 7.2% (95% CI 6.6-7.8%,P < 0.001) and the gap in hit rates between white and black drivers was 4.3% (95% CI 3.8-4.8%,P < 0.001).”

May combine two claims. The quote only partly states this claim.

The primary dataset used in the analysis contains about 95 million stops from 21 state patrol agencies and 35 municipal police departments, limited to 2011-2018 and to drivers classified as white, black, or Hispanic.

“Our primary dataset thus consists of approximately 95 million stops from 21 state patrol agencies and 35 municipal police departments”

• Also stated in: § Data collection and standardization. · plain

  “Our primary dataset includes 94,778,505 stops from 21 state patrol agencies and 35 municipal police departments.”

May combine two claims.

Marijuana legalization reduced the absolute gap in search rates between race groups, but the relative gap persisted, with black and Hispanic drivers still more likely to be searched than white drivers after legalization.

“the policy change reduced the absolute gap in search rates between race groups; however, the relative gap persisted, with black and Hispanic drivers still more likely to be searched than white drivers post-legalization.”

May combine two claims.

The estimated state-level search-threshold gaps were large and statistically significant, with 95% credible intervals of (-8.4, -5.6%) for Hispanic-white and (-6.5%, -3.1%) for black-white.

“These differences are again large and statistically significant: the 95% credible interval for the Hispanic-white gap is (-8.4,-5.6%); and the analogous interval for the black-white gap is (-6.5%, -3.1%).”

May combine two claims.

The proportion of stops resulting in a search fell substantially in Colorado and Washington after marijuana legalization, while search rates did not show sharp drops at the end of 2012 in 12 states where marijuana was not legalized.

“the proportion of stops resulting in a search fell substantially in both Colorado and Washington, while in 12 states where marijuana was not legalized, search rates did not show sharp drops at the end of 2012.”

May combine two claims.

Despite marijuana legalization decreasing search rates for white, black, and Hispanic drivers, the relative disparity between white and non-white drivers remains.

“Despite marijuana legalization decreasing search rates for white, black and Hispanic drivers, Fig. 4 shows that the relative disparity between white and non-white drivers remains.”

May combine two claims.

White drivers faced consistently higher search thresholds than black and Hispanic drivers, both before and after marijuana legalization.

“Supplementary Fig. 3 shows that white drivers faced consistently higher search thresholds than black and Hispanic drivers, both before and after marijuana legalization.”

• AssumptionThe authors state this as an assumption: The latent signal distributions for each race and county are homoskedastic discriminant distributions.
• AssumptionThe authors state this as an assumption: The parameters φ and δ can be decomposed into additive race and location terms.
• AssumptionThe authors state this as an assumption: For each stop, a signal is drawn from the race-and-location-specific distribution; a search occurs if the signal exceeds the race-and-location-specific threshold; if searched, contraband is found with probability equal to the signal.

May combine two claims.

In the veil-of-darkness test, black drivers comprised a smaller share of stopped drivers after dark than before dark across three time windows in Texas, suggestive of racial profiling.

“For all three depicted time windows, black drivers comprise a smaller share of stopped drivers after dark, when a veil of darkness masks their race, suggestive of racial profiling.”

May combine two claims.

The proportion of drivers with typically Hispanic names labelled Hispanic in raw data was 70% in Colorado and 79% in Arizona.

“The proportion of drivers with typically Hispanic names labelled as Hispanic in the raw data was 70% in Colorado and 79% in Arizona, much higher than in Texas.”

May combine two claims.

Despite lower inferred search thresholds for black drivers, their hit rates are comparable to white drivers, ostensibly due to infra-marginality.

“Despite these lower inferred search thresholds, hit rates for black drivers are comparable to those for white drivers, ostensibly due to the problem of infra-marginality in outcome tests.”

May combine two claims.

The threshold faced by all groups decreased after legalization, although not all drops were statistically significant.

“Supplementary Fig. 3 also shows that the threshold faced by all groups decreases after legalization (although not all drops are statistically significant).”

May combine two claims.

In a difference-in-differences model, the race-specific legalization coefficients were large and negative for white, black, and Hispanic drivers, suggesting the observed drop in searches in Colorado and Washington was due to marijuana legalization.

“We found that α race r I is large and negative for white, black and Hispanic drivers, suggesting that the observed drop in searches in Colorado and Washington was due to marijuana legalization in those states.”

• AssumptionThe authors state this as an assumption: For each stop, a signal is drawn from the race-and-location-specific distribution; a search occurs if the signal exceeds the race-and-location-specific threshold; if searched, contraband is found with probability equal to the signal.

May combine two claims.

The outcome test suggests possible bias against Hispanic drivers in search decisions, but the evidence for black drivers is more ambiguous.

“The outcome test thus indicates that search decisions may be biased against Hispanic drivers, but the evidence is more ambiguous for black drivers.”

• AssumptionThe authors state this as an assumption: Officers exercise discretion in whom to search.
• AssumptionThe authors state this as an assumption: In the absence of discrimination, searched minority drivers should be found with contraband at the same rate as searched white drivers.

May combine two claims. The quote only partly states this claim.

The negative veil-of-darkness coefficients indicate discrimination against black drivers in stop decisions by both state patrols and municipal police departments.

“suggesting discrimination among both state patrols and municipal police departments.”

• AssumptionThe authors state this as an assumption: Officers who engage in racial profiling are less able to identify a driver's race after dark than during the day.
• AssumptionRests on an assumption: Race-specific driving behaviour is not related more to lighting than to time of day.
• AssumptionRests on an assumption: Artificial lighting does not substantially weaken the relationship between sunlight and visibility.
• AssumptionRests on an assumption: Violation type is not related to lighting in a way that biases the veil-of-darkness test.

May combine two claims.

Although overall search rates dropped in Washington and Colorado after legalization, black and Hispanic drivers still faced discrimination in search decisions.

“The data thus suggest that, although overall search rates dropped in Washington and Colorado, black and Hispanic drivers still faced discrimination in search decisions.”

• AssumptionRests on an assumption: Search policies do not differ within the geographic subdivisions considered.
• AssumptionRests on an assumption: Officers do not disproportionately suspect more serious criminal activity when searching black and Hispanic drivers than when searching white drivers.
• AssumptionRests on an assumption: The Bayesian threshold model's functional form and prior distributions are adequate for identifying search thresholds.

May combine two claims.

Prior work found that Texas State Patrol officers incorrectly recorded many Hispanic drivers as white, and the agency later corrected this.

“past work revealed that Texas State Patrol officers incorrectly recorded many Hispanic drivers as white, an error the agency subsequently corrected 36 .”

• cites [36] Friberg, B. et al. Texas Troopers Ticketing Hispanic Drivers as White (KXAN, 2015); https://www.kxan.com/investigations/texas-troopers-ticketing-hispanic-drivers-as-white/Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors used all available data that were sufficiently complete to conduct the analysis.

“We used all available data that were sufficiently complete to conduct our analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The raw data, processed data, and code are publicly available for inspection and reuse.

“The raw data we received from the states and cities, the processed data we used in this analysis and the code to clean and analyse the data are all available for public inspection and reuse.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The threshold test was introduced by Simoiu et al. to mitigate shortcomings of benchmark and outcome analysis.

“The threshold test for discrimination was introduced by Simoiu et al. 7 to mitigate the most serious shortcomings of benchmark and outcome analysis.”

• cites Simoiu et al. 7Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Under the threshold model, lower search thresholds for one group relative to another are interpreted as evidence of discrimination.

“Under this model of officer behaviour, we interpret lower search thresholds for one group relative to another as evidence of discrimination.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The marijuana search analysis included 1,674,619 stops in Colorado and 3,985,677 stops in Washington.

“The left panel (Colorado) represents 1,674,619 stops and the right panel (Washington)represents 3,985,677 stops.”

Classified as probably not a substantial claim.

The authors applied a computationally fast variant of the threshold test developed by Pierson et al., modified to include an additional hierarchical component.

“we applied a computationally fast variant of the threshold test developed by Pierson et al. 22 , which we fit separately on both state patrol stops and municipal police stops.”

• cites Pierson et al. 22Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

All Texas drivers with Hispanic names originally labelled white or with missing race data were re-categorized as Hispanic, adding about 1.9 million stops over 2011-2015.

“we re-categorized as 'Hispanic' all drivers in Texas with Hispanic names who were originally labelled 'white' or who had missing race data; this method adds about 1.9 million stops of Hispanic drivers over the period 2011-2015.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

If officers disproportionately patrol black and Hispanic neighbourhoods, downstream effects can be injurious even if individual stop decisions are not directly affected by driver race.

“If, for example, officers disproportionately patrol black and Hispanic neighbourhoods, the downstream effects can be injurious even if individual stop decisions are not directly affected by the colour of one's skin.”

May combine two claims.

Enforcement of minor traffic violations can place heavy burdens on black and Hispanic drivers without improving public safety.

“can place heavy burdens on black and Hispanic drivers without improving public safety 17 .”

• cites [17] Chohlas-Wood, A., Goel, S., Shoemaker, A. & Shroff, R. An Analysis of the Metropolitan Nashville Police Department’s Traffic Stop Practices https://policylab.stanford.edu/media/nashville-traffic-stops.pdf (Stanford Computational Policy Lab, 2018).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Because black drivers were more likely to be charged with drug-related offences before legalization, they were disproportionately affected by the policy change.

“since black drivers were more likely to be charged with such offences previous to legalization, these drivers were also disproportionately impacted by the policy change.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Automated checks were run to detect and correct errors in the raw data, though some errors probably remain.

“We ran numerous automated checks to detect and correct these where possible, although some errors probably remain due to the complex nature of the data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors did not exclude any data deemed sufficient to conduct the analysis.

“We did not exclude any data that were deemed sufficient to conduct our analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The KPT model requires strong assumptions, including that drivers and officers are rational actors and that every driver has perfect knowledge of the likelihood of being searched.

“the KPT model of behaviour requires strong assumptions, including that drivers and officers are rational actors and that every driver has perfect knowledge of the likelihood that he or she will be searched.”

• cites [28] Engel, R. S. & Tillyer, R. Searching for equilibrium: the tenuous nature of the outcome test. Justice Q. 25, 54–71 (2008). sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Following previous studies, a name was defined as 'typically' Hispanic if at least 75% of people with that name identified as Hispanic.

“Following previous studies 38,39 , we defined a name as 'typically' Hispanic if at least 75% of people with that name identified as Hispanic”

• cites [38] Word, D. & Perkins, C. Building a Spanish Surname List for the 1990s: A New Approach to an Old Problem (Population Division, US Census Bureau, 1996).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [39] Melendres v Arpaio (2009). 598 F. Supp. 2d 1025 (D. Ariz., 2009).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Posterior predictive checks showed minimal and similar prediction errors across race groups for both state patrol and municipal models.

“For both the state patrol model and the municipal department model, the prediction errors are minimal and similar across race groups.”

Classified as probably not a substantial claim. May combine two claims.

The threshold test captures only taste-based discrimination, not statistical discrimination.

“The threshold test, however, aims to capture only taste-based discrimination, as is common in the empirical literature on discrimination.”

Classified as probably not a substantial claim.

Drivers in other jurisdictions were not re-categorized.

“We did not re-categorize drivers in any other jurisdiction.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors found results qualitatively similar under alternative averaging schemes for overall thresholds.

“although our results are qualitatively similar under alternative averaging schemes.”

Classified as probably not a substantial claim.

Analysis was limited to counties and districts with at least 50 searches per race group.

“we limit our analysis to counties and districts with at least 50 searches per race group.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The data-cleaning pipeline was extensive and required subjective decisions.

“Our complete data-cleaning pipeline is extensive and required subjective decisions”

May combine two claims.

In the veil-of-darkness analysis, results were statistically significant and qualitatively similar across variations of the primary model.

“The results were statistically significant and qualitatively similar in all cases.”