CoChat
Sign Up
Cited by 26Ethics and Social Impacts of AINames, Identity, and Discrimination ResearchDecision-Making and Behavioral Economics

Automating Automaticity: How the Context of Human Choice Affects the Extent of Algorithmic Bias

Open access
Agan, Amanda, Davenport, Diag, Ludwig, Jens, Mullainathan, Sendhil·2023·National Bureau of Economic Research
DOI 10.3386/w30981
Open this paper in CoChat — free account

Abstract

Consumer choices are increasingly mediated by algorithms, which use data on those past choices to infer consumer preferences and then curate future choice sets.Behavioral economics suggests one reason these algorithms so often fail: choices can systematically deviate from preferences.For example, research shows that prejudice can arise not just from preferences and beliefs, but also from the context in which people choose.When people behave automatically, biases creep in; snap decisions are typically more prejudiced than slow, deliberate ones, and can lead to behaviors that users themselves do not consciously want or intend.As a result, algorithms trained on automatic behaviors can misunderstand the prejudice of users: the more automatic the behavior, the greater the error.We empirically test these ideas in a lab experiment, and find that more automatic behavior does indeed seem to lead to more biased algorithms.We then explore the large-scale consequences of this idea by carrying out algorithmic audits of Facebook in its two biggest markets, the US and India, focusing on two algorithms that differ in how users engage with them: News Feed (people interact with friends' posts fairly automatically) and People You May Know (people choose friends fairly deliberately).We find significant out-group bias in the News Feed algorithm (e.g., whites are less likely to be shown Black friends' posts, and Muslims less likely to be shown Hindu friends' posts), but no detectable bias in the PYMK algorithm.Together, these results suggest a need to rethink how large-scale algorithms use data on human behavior, especially in online contexts where so much of the measured behavior might be quite automatic.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3386/w30981

Cites

108 of 108 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Insko, C. A., J. Schopler, and C. Sedikides (1998). Personal control, entitativity, and evo- lution. In C. Sedikides, J. Schopler, and C. A. Insko (Eds.), Intergroup cognition and intergroup behavior, pp. 109-120.
  2. In B. Wittenbrink and S. N. (Eds.), Implicit measures of attitudes, pp. 59--102. The Guilford Press.
  3. LeVine, R. A. and D. T. Campbell (1972). Ethnocentrism: Theories of conflict, ethnic attitudes, and group behavior. John Wiley & Sons.
  4. Williams, D. H. (1948). The effects of an interracial project upon the attitudes of negro and white girls within the YWCA. In A. Rose (Ed.), Studies in Reduction of Prejudice. American Council of Race Relations.

Cited by

Sorted by citations

Related

Authors
Agan, A., Davenport, D., Ludwig, J., & Mullainathan, S.
Affiliations
Rutgers, The State University of New Jersey (US)
Corresponding author
Agan, Amanda
ORCID
Venue
National Bureau of Economic Research
Publisher
National Bureau of Economic Research
Published
2023-02
DOI
10.3386/w30981
OpenAlex
W4321366289
Cited by
25
References
105
Citations by year
2026: 4 · 2025: 6 · 2024: 8 · 2023: 7
Type
report
Language
en
Open access
Locations
  • National Bureau of Economic Research (publishedVersion)
  • SSRN Electronic Journal (acceptedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
Keywords
algorithmic bias, automatism, prejudice, out-group bias, Facebook, algorithmic auditing, social media, user preferences, racial bias, religious bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review