CoChat
Sign Up
Cited by 118Ethics and Social Impacts of AIMachine Learning and AlgorithmsHate Speech and Cyberbullying Detection

The DOI matches its Crossref record; 166 of the 173 references it cites were found in a registry or an index; none of the 23 sources its claims cite is checked for support yet.

Evolution and impact of bias in human and machine learning algorithm interaction

Open access
Sun, Wenlong, Nasraoui, Olfa, Shafto, Patrick·2020·PLOS ONE
DOI 10.1371/journal.pone.0235502PMC PMC7425868
Open this paper in CoChat — free accountOpen PDF

Abstract

Traditionally, machine learning algorithms relied on reliable labels from experts to build predictions. More recently however, algorithms have been receiving data from the general population in the form of labeling, annotations, etc. The result is that algorithms are subject to bias that is born from ingesting unchecked information, such as biased samples and biased labels. Furthermore, people and algorithms are increasingly engaged in interactive processes wherein neither the human nor the algorithms receive unbiased data. Algorithms can also make biased predictions, leading to what is now known as algorithmic bias. On the other hand, human's reaction to the output of machine learning methods with algorithmic bias worsen the situations by making decision based on biased information, which will probably be consumed by algorithms later. Some recent research has focused on the ethical and moral implication of machine learning algorithmic bias on society. However, most research has so far treated algorithmic bias as a static factor, which fails to capture the dynamic and iterative properties of bias. We argue that algorithmic bias interacts with humans in an iterative manner, which has a long-term effect on algorithms' performance. For this purpose, we present an iterated-learning framework that is inspired from human language evolution to study the interaction between machine learning algorithms and humans. Our goal is to study two sources of bias that interact: the process by which people select information to label (human action); and the process by which an algorithm selects the subset of information to present to people (iterated algorithmic bias mode). We investigate three forms of iterated algorithmic bias (personalization filter, active learning, and random) and how they affect the performance of machine learning algorithms by formulating research questions about the impact of each type of bias. Based on statistical analyses of the results of several controlled experiments, we found that the three different iterated bias modes, as well as initial training data class imbalance and human action, do affect the models learned by machine learning algorithms. We also found that iterated filter bias, which is prominent in personalized user interfaces, can lead to more inequality in estimated relevance and to a limited human ability to discover relevant data. Our findings indicate that the relevance blind spot (items from the testing set whose predicted relevance probability is less than 0.5 and who thus risk being hidden from humans) amounted to 4% of all relevant items when using a content-based filter that predicts relevant items. A similar simulation using a real-life rating data set found that the same filter resulted in a blind spot size of 75% of the relevant testing set.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes an iterated-learning framework to study how algorithmic and human biases interact and evolve in recommendation/filtering loops. Across synthetic and MovieLens simulations, iterated filter bias consistently shifted learned relevance boundaries, increased prediction inequality, and enlarged blind spots that hide relevant items, while random selection and active learning often did not.

Key findings

  • Iterated filter bias had significant effects in 2D synthetic data: boundary shift p<.001, effect size = 1.96; class-1 blind spot p=2.4e-10, effect size = -1.22; all-classes blind spot p=5.9e-12, effect size = -1.44. Active learning and random selection did not significantly affect boundary shift (p=.15 and .77, respectively).
  • On MovieLens real data, filter bias significantly decreased points predicted as class y=1 (p=7.1e-15, ES=1.88), increased class-1 blind spot (p=7.0e-15, ES=-1.89), and increased Gini inequality (p=7.7e-15, ES=-1.83); active learning also affected boundary shift and blind spot, while random selection did not significantly affect boundary or blind spot but decreased inequality.
  • The abstract reports a relevance blind spot of 4% of all relevant items for a content-based filter in one simulation, and a blind spot size of 75% of the relevant testing set in a similar simulation using a real-life rating data set.
  • Initial class imbalance and human action mattered: imbalanced initialization significantly affected boundary shift for all three bias modes (Kruskal-Wallis p=5.5e-19 filter, 3.27e-17 active learning, 1.34e-15 random); higher human action probability increased filter-bias boundary shift (p_action=1: p=0.4e-11, ES=3.087; p_action=0.5: p=1.8e-6, ES=0.87; p_action=0.01: p=.259). With class-dependent human action ratio 10:1, filter bias increased class-1 blind spot (p=1.1e-10, ES=-1.2) and random selection decreased it (p=.0008, ES=0.7).

Methods

  • The authors use an iterated-learning framework inspired by language evolution, simplifying recommendation into a two-class classification problem (relevant/non-relevant) with a Naive Bayes classifier. They compare three selection mechanisms: random, active-bias, and filter-bias; human action probabilities of 1, 0.5, and 0.01; and a class-dependent action ratio of 10:1, running 200 iterations and 40 independent simulations per condition.
  • Synthetic data: 2D Gaussian classes centered at {-2,0} and {2,0} with σ=1 and 1000 points per class, split into testing (200/class), validation (200/class), initialization ratios 10/100, 100/100, and 100/10, plus 3D, 4D, and 10D synthetic sets. Real data: MovieLens with 100,004 ratings on 9,125 movies from 671 users; user 547 (2,312 movies rated; 18 genres after filtering; PCA to 11 features) was studied first, then 7 more users; ratings were discretized as [1,3]=class 0 and (3,5]=class 1, with unbiased initialization only.
  • Main analyses measure boundary shift, Gini coefficient, and blind spot size (δ=0.5), using Shapiro-Wilk, Mann-Whitney U, t-test, ANOVA, Kruskal-Wallis, and effect sizes.

Limitations

  • The paper states future work will extend the framework to more users in a collaborative filtering setting, test additional parameters/configurations such as top-N recommendation list size and different active learning types beyond uncertainty-based active learning, and conduct human experiments similar to the simulated research questions.
  • Evident from the text: the findings rely on simulations rather than human experiments, use a simplified two-class content-based recommendation setup with Naive Bayes and independent-feature assumptions, and the real-data experiments focus on one user at a time and only the 8 most active MovieLens users, with genres as item features and PCA.
  • Evident from the text: for the real-life data set, the authors only explore unbiased initialization and assume action probabilities of 1, so human-action and class-imbalance effects are not tested on the real data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0235502

Cites

166 of 166 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 23 cited sources checked · 0 supported · 0 flagged

Show all 89 claims

  • It may state two separate claims as one. 33
  • The quote only partly states this claim. 5
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • Probably not a substantial claim. 11
44 of 44

  • The paper relies on this without support.

    People and algorithms are increasingly engaged in interactive processes wherein neither the human nor the algorithms receive unbiased data.

    “Furthermore, people and algorithms are increasingly engaged in interactive processes wherein neither the human nor the algorithms receive unbiased data.”

  • The authors state this as an assumption.

    A Bayesian model for prediction is assumed for the algorithm.

    “Following Griffiths’ model for human learners, we assume a Bayesian model for prediction.”

  • The authors state this as an assumption.

    People's response is always observed in the above analysis.

    “The above analysis assumes that people’s response is always observed.”

  • The authors state this as an assumption.

    People have some target hypothesis h*, which represents optimal performance for the algorithm.

    “Assume that people have some target hypothesis, h*, which represents optimal performance for the algorithm.”

  • The authors state this as an assumption.

    A simplified algorithm where only the very uncertain data are selected, and a mixture of random sampling and active learning, is assumed.

    “Assuming a simplified algorithm where only the very uncertain data are selected, we can investigate the limiting behavior of an algorithm with the active learning bias. Assuming a mixture of random sampling and active learning, we obtain:”

  • The authors state this as an assumption.

    The choice of whether to act can be formalized using Luce choice, a special case of softmax.

    “We can formalize this choice using Luce choice [56], a special case of softmax [137]”

  • The paper relies on this without support.

    The synthetic dataset represents a typical classification problem and is comparable to the real-life dataset after selecting one user and corresponding items.

    “The synthetic dataset represents a typical classification problem, it is the same for our real-life dataset after selecting one user and corresponding items.”

  • The paper relies on this without support.

    In the real world, some data can be invisible to some users because of bias either from users or from the algorithm itself.

    “In the real world, some data can be invisible to some users because of bias either from users or from the algorithm itself.”

  • The paper relies on this without support.

    Class imbalance can emerge from the algorithm intentionally asking users to rate only the most popular items, a common strategy used to collect initial ratings for new users.

    “Class imbalance can also emerge from the algorithm intentionally asking users to rate only the most popular items, a common strategy used to collect initial ratings for new users.”

  • The authors state this as an assumption.

    In RQ2, humans have a prior probability to act that is not dependent on the true label of the item presented.

    “In research question 2 (RQ 2), we assumed that humans have a prior probability to act which is not dependent on the true label of the item presented.”

  • The authors state this as an assumption.

    Humans have a higher probability to label or rate when the item presented is from relevant class y = 1.

    “we try to simulate this by assuming that humans have a higher probability to label or rate when the item presented is from relevant class y = 1 (see Eq 15).”

  • The paper relies on this without support.

    Filter bias already prefers points from class y = 1.

    “since filter bias already prefers points from class y = 1.”

  • The paper relies on this without support.

    Active learning bias already prefers points that are close to the boundary.

    “since this bias already prefers points that are close to the boundary.”

  • The paper relies on this without support.

    Both random selection and active learning are used to build an accurate model in machine learning.

    “both random selection and active learning are used to build an accurate model in machine learning.”

  • The paper relies on this without support.

    Random selection with class-dependent human action probability prefers points from class y = 1, however randomly.

    “Random selection with class-dependent human action probability prefer points from class y = 1, however randomly.”

  • The paper relies on this without support.

    Active learning aims to help learn the correct boundary.

    “Active learning aims to help learn correct boundary”

  • The paper relies on this without support.

    Humans and AI algorithms are both increasingly intertwined in feedback loops that end up feeding biased data to both humans and algorithms.

    “Humans and AI algorithms are both increasingly intertwined in feedback loops that end up feeding biased data to both humans and algorithms.”
17 of 17

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • MovieLens active-learning boundary shift is both nonsignificant and significant

    Major

    The boundary-shift discussion calls active learning nonsignificant, but Table 15 reports p = 0.002 and p = 0.0004; the following paragraph also concludes that its effect is significant.

  • MovieLens active-learning blind-spot result has conflicting direction and significance

    Major

    The text says active learning significantly increases the class-1 blind spot, but Table 16 reports a positive effect size of 0.66, which means a decrease under the stated initial-minus-final definition. The table caption instead says active learning has no significant impact, despite its p-values of 0.0002 and 0.0001.

  • User 547's movie count differs between text and Table 18

    Minor

    The dataset description says user 547 rated 2312 movies, whereas Table 18 lists 2391 for that user.

  • Active learning's class-1 blind-spot effect is called nonsignificant despite p = 0.03

    Minor

    The text says active learning has no significant effect, but Table 2 reports p = 0.03 in both tests, below the paper's stated p < 0.05 threshold.

  • Active learning has a significant boundary shift in Table 5

    Minor

    The conclusion says the non-filter modes are not affected, but for active learning at paction = 1, Table 5 reports significant Mann-Whitney and t-test p-values.

  • Table 3 discussion calls active learning's blind-spot decrease significant

    Minor

    The discussion describes a significant decrease for active learning, whereas Table 3 gives nonsignificant p-values of 0.5 and 0.18. Its reported negative effect size also indicates an increase, not a decrease, under the table's stated definition.

  • Table 8 misidentifies the sign of filter bias's effect size

    Minor

    The caption calls the filter-bias effect size negative, but the table reports a positive value of 1.6.

7 of 7

Citations

Cites

Flagged first, then by citations

Show all 173 references

7 of the 173 references couldn’t be matched to a known paper.

Research works printed but not linked: 11

  1. Spark KJ. Artificial Intelligence: What can it offer to Information Retrieval. Proceedings of the Informatics 3, Aslib, ed, London. 1978;.
  2. McFadden DL. Quantal choice analaysis: A survey. In: Annals of Economic and Social Measurement, Volume 5, number 4. NBER; 1976. p. 363–390.
  3. TL Griffiths, 2005, Proceedings of the Cognitive Science Society
  4. Smith K. Iterated learning in populations of Bayesian agents. In: Proceedings of the 31st annual conference of the cognitive science society. Citeseer; 2009. p. 697–702.
  5. KJ Rothman, 2008, Modern epidemiology
  6. Dwork C, Immorlica N, Kalai AT, Leiserson MD. Decoupled classifiers for group-fair and efficient machine learning. In: Conference on Fairness, Accountability and Transparency; 2018. p. 119–133.
  7. Barocas S. Data mining and the discourse on discrimination. In: Data Ethics Workshop, Conference on Knowledge Discovery and Data Mining; 2014.
  8. Schnabel T, Swaminathan A, Singh A, Chandak N, Joachims T. Recommendations as treatments: Debiasing learning and evaluation. arXiv preprint arXiv:160205352. 2016;.
  9. A Sinha, 2016, Advances in neural information processing systems, 3243
  10. S Milano, 2020, AI & SOCIETY, 1
  11. A Stuart, 1994, Distribution theory

Cited by

Sorted by citations

Items 101–118 of 118

Related

See all

Metadata

Authors
Sun, W., Nasraoui, O., & Shafto, P.
Affiliations
  • University of Louisville (US)
  • Rutgers, The State University of New Jersey (US)
Corresponding author
Nasraoui, Olfa
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2020-08-13
Volume
15
Issue
8
Pages
e0235502
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0235502
PMC
PMC7425868
OpenAlex
W3048367007
Semantic Scholar
fba50a0f6e7f1bb47e80ffa8c523396aa747a6ae
Cited by
116
References
173
Citations by year
2026: 15 · 2025: 29 · 2024: 16 · 2023: 21 · 2022: 26 · 2021: 8 · 2020: 1
FWCI
19.26
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Division of Information and Intelligent Systems (1549981)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
algorithmic bias, human-machine interaction, iterated learning, filter bubbles, personalization, active learning (machine learning), class imbalance, personalized interfaces, information filtering, relevance prediction
MeSH terms
Machine Learning, Humans, Learning, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review