CoChat
Sign Up
Cited by 118Ethics and Social Impacts of AIMachine Learning and AlgorithmsHate Speech and Cyberbullying Detection

Evolution and impact of bias in human and machine learning algorithm interaction

Open access
Sun, Wenlong, Nasraoui, Olfa, Shafto, Patrick·2020·PLOS ONE
DOI 10.1371/journal.pone.0235502PMC PMC7425868
Open this paper in CoChat — free accountOpen PDF

Abstract

Traditionally, machine learning algorithms relied on reliable labels from experts to build predictions. More recently however, algorithms have been receiving data from the general population in the form of labeling, annotations, etc. The result is that algorithms are subject to bias that is born from ingesting unchecked information, such as biased samples and biased labels. Furthermore, people and algorithms are increasingly engaged in interactive processes wherein neither the human nor the algorithms receive unbiased data. Algorithms can also make biased predictions, leading to what is now known as algorithmic bias. On the other hand, human's reaction to the output of machine learning methods with algorithmic bias worsen the situations by making decision based on biased information, which will probably be consumed by algorithms later. Some recent research has focused on the ethical and moral implication of machine learning algorithmic bias on society. However, most research has so far treated algorithmic bias as a static factor, which fails to capture the dynamic and iterative properties of bias. We argue that algorithmic bias interacts with humans in an iterative manner, which has a long-term effect on algorithms' performance. For this purpose, we present an iterated-learning framework that is inspired from human language evolution to study the interaction between machine learning algorithms and humans. Our goal is to study two sources of bias that interact: the process by which people select information to label (human action); and the process by which an algorithm selects the subset of information to present to people (iterated algorithmic bias mode). We investigate three forms of iterated algorithmic bias (personalization filter, active learning, and random) and how they affect the performance of machine learning algorithms by formulating research questions about the impact of each type of bias. Based on statistical analyses of the results of several controlled experiments, we found that the three different iterated bias modes, as well as initial training data class imbalance and human action, do affect the models learned by machine learning algorithms. We also found that iterated filter bias, which is prominent in personalized user interfaces, can lead to more inequality in estimated relevance and to a limited human ability to discover relevant data. Our findings indicate that the relevance blind spot (items from the testing set whose predicted relevance probability is less than 0.5 and who thus risk being hidden from humans) amounted to 4% of all relevant items when using a content-based filter that predicts relevant items. A similar simulation using a real-life rating data set found that the same filter resulted in a blind spot size of 75% of the relevant testing set.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes an iterated-learning framework to study how algorithmic and human biases interact and evolve in recommendation/filtering loops. Across synthetic and MovieLens simulations, iterated filter bias consistently shifted learned relevance boundaries, increased prediction inequality, and enlarged blind spots that hide relevant items, while random selection and active learning often did not.

Key findings

  • Iterated filter bias had significant effects in 2D synthetic data: boundary shift p<.001, effect size = 1.96; class-1 blind spot p=2.4e-10, effect size = -1.22; all-classes blind spot p=5.9e-12, effect size = -1.44. Active learning and random selection did not significantly affect boundary shift (p=.15 and .77, respectively).
  • On MovieLens real data, filter bias significantly decreased points predicted as class y=1 (p=7.1e-15, ES=1.88), increased class-1 blind spot (p=7.0e-15, ES=-1.89), and increased Gini inequality (p=7.7e-15, ES=-1.83); active learning also affected boundary shift and blind spot, while random selection did not significantly affect boundary or blind spot but decreased inequality.
  • The abstract reports a relevance blind spot of 4% of all relevant items for a content-based filter in one simulation, and a blind spot size of 75% of the relevant testing set in a similar simulation using a real-life rating data set.
  • Initial class imbalance and human action mattered: imbalanced initialization significantly affected boundary shift for all three bias modes (Kruskal-Wallis p=5.5e-19 filter, 3.27e-17 active learning, 1.34e-15 random); higher human action probability increased filter-bias boundary shift (p_action=1: p=0.4e-11, ES=3.087; p_action=0.5: p=1.8e-6, ES=0.87; p_action=0.01: p=.259). With class-dependent human action ratio 10:1, filter bias increased class-1 blind spot (p=1.1e-10, ES=-1.2) and random selection decreased it (p=.0008, ES=0.7).

Methods

  • The authors use an iterated-learning framework inspired by language evolution, simplifying recommendation into a two-class classification problem (relevant/non-relevant) with a Naive Bayes classifier. They compare three selection mechanisms: random, active-bias, and filter-bias; human action probabilities of 1, 0.5, and 0.01; and a class-dependent action ratio of 10:1, running 200 iterations and 40 independent simulations per condition.
  • Synthetic data: 2D Gaussian classes centered at {-2,0} and {2,0} with σ=1 and 1000 points per class, split into testing (200/class), validation (200/class), initialization ratios 10/100, 100/100, and 100/10, plus 3D, 4D, and 10D synthetic sets. Real data: MovieLens with 100,004 ratings on 9,125 movies from 671 users; user 547 (2,312 movies rated; 18 genres after filtering; PCA to 11 features) was studied first, then 7 more users; ratings were discretized as [1,3]=class 0 and (3,5]=class 1, with unbiased initialization only.
  • Main analyses measure boundary shift, Gini coefficient, and blind spot size (δ=0.5), using Shapiro-Wilk, Mann-Whitney U, t-test, ANOVA, Kruskal-Wallis, and effect sizes.

Limitations

  • The paper states future work will extend the framework to more users in a collaborative filtering setting, test additional parameters/configurations such as top-N recommendation list size and different active learning types beyond uncertainty-based active learning, and conduct human experiments similar to the simulated research questions.
  • Evident from the text: the findings rely on simulations rather than human experiments, use a simplified two-class content-based recommendation setup with Naive Bayes and independent-feature assumptions, and the real-data experiments focus on one user at a time and only the 8 most active MovieLens users, with genres as item features and PCA.
  • Evident from the text: for the real-life data set, the authors only explore unbiased initialization and assume action probabilities of 1, so human-action and class-imbalance effects are not tested on the real data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0235502

Cites

166 of 166 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 23 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    A similar simulation using a real-life rating data set found that the same filter resulted in a blind spot size of 75% of the relevant testing set.

    “A similar simulation using a real-life rating data set found that the same filter resulted in a blind spot size of 75% of the relevant testing set.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Based on statistical analyses of controlled experiments, the three iterated bias modes (personalization filter, active learning, random), initial training data class imbalance, and human action affect the models learned by machine learning algorithms.

    “Based on statistical analyses of the results of several controlled experiments, we found that the three different iterated bias modes, as well as initial training data class imbalance and human action, do affect the models learned by machine learning algorithms.”
    Quote in the paperSubstantial claim
  • Findingown data

    The relevance blind spot (items from the testing set whose predicted relevance probability is less than 0.5 and thus risk being hidden from humans) amounted to 4% of all relevant items when using a content-based filter that predicts relevant items.

    “Our findings indicate that the relevance blind spot (items from the testing set whose predicted relevance probability is less than 0.5 and who thus risk being hidden from humans) amounted to 4% of all relevant items when using a content-based filter that predicts relevant items.”
    Quote in the paperSubstantial claim
  • Findingown data

    Iterated filter bias causes the largest heterogeneity level in the Gini coefficient among the three bias forms.

    “Despite the small effect, the iterated algorithmic bias forms affect this distribution in different ways, and iterated filter bias causes the largest heterogeneity level as can be seen in Fig 6.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Filter bias has a dramatic impact on boundary shift when the imbalanced initialization ratio is high and more points are from the irrelevant class (ratio 1:10).

    “On the other hand, the boundary shift of filter bias has a big difference when the ratio is 1:10, indicating that filter bias has a dramatic impact on the boundary shift when the imbalanced initialization ratio is high and more points are from the irrelevant class.”
    Quote in the paperSubstantial claim
  • May combine two claims. 33
  • The quote only partly states this claim. 5
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Findingown data

    Iterated filter bias, prominent in personalized user interfaces, can lead to more inequality in estimated relevance and to a limited human ability to discover relevant data.

    “We also found that iterated filter bias, which is prominent in personalized user interfaces, can lead to more inequality in estimated relevance and to a limited human ability to discover relevant data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The number of label-changed points with ratio = 10:1 is higher than for ratio = 1:1 in iterated filter bias mode, and similar for active learning and random selection.

    “The Number of label-changed points with ratio = 10:1 is higher than 2 for ratio = 1:1 in the iterated filter bias mode, and a similar result can be seen with iterated active learning bias and iterated random selection (see Fig 9).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Random selection and active learning bias converge to the ground-truth boundary.

    “Fig 5 shows that random selection and active learning bias converge to the ground-truth boundary.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Both filter bias and active learning bias show no significant difference in class-1 blind spot size between ratios 10:1 and 1:1.

    “Both Filter bias and active learning bias show no significant difference between the two ratios.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both active learning and random selection decrease the inequality.

    “Both active learning and random selection decrease the inequality.”
    • Also stated in: § Experiments on 2D synthetic data · boosted
      “random selection significantly decrease the Gini coefficient, as well as the active learning.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    There is a significant increase in the class-1 blind spot sizes in the testing set for iterated filter bias and active learning bias with the MovieLens data set.

    “Table 16 shows that there is a significant increase in the class-1 blind spot sizes in the testing set for iterated filter bias and active learning bias.”
    Quote in the paperSubstantial claim

  • Borderline on S3 (load-bearing), S4 (based): unclear from the text.

    Findingnone given

    A similar impact was found when the initial labeled (training) set is class imbalanced with more relevant items than non relevant items.

    “A similar impact was found when the initial labeled (training) set is class imbalanced with more relevant items than non relevant items.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Random selection decreases the class-1 blind spot size when comparing ratios 1:1 and 10:1.

    “On the other hand, random selection decreases the class-1 blind spot size.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both iterated filter bias and iterated active learning bias have a significant effect on the boundary shift.

    “Both iterated filter bias and iterated active learning bias have a significantly effect on the boundary shift”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both iterated filter bias and iterated active learning bias have a significant effect on the boundary shift, while random selection does not have a significant effect.

    “We conclude that both iterated filter bias and iterated active learning bias have a significantly effect on the boundary shift, while random selection does not have a significant effect.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both active learning and random selection result in a significant decrease in the Gini coefficient with the MovieLens data set.

    “while both active learning and random selection result in a significant decrease in the Gini coefficient.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both filter bias and active learning bias show no significant difference in inequality between ratios 10:1 and 1:1.

    “Both Filter bias and active learning bias show no significant difference between the two ratios.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    For both filter bias and active learning, the number of points predicted as class y = 1, with and without the class-dependent human action probability, show no significant difference.

    “For both filter bias and active learning, the number of points predicted as class y = 1, with and without the class-dependent human action probability, show no significant difference.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The bias modes have different effects on the class-1 blind spot (Kruskal-wallis p = 3.4e-15 < 0.01).

    “The p-value is 3.4e-15 (<0.01), which indicates that they have different effects on the boundary shift (see Fig 12).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both active learning and random selection show a significant decrease in the Gini coefficient.

    “while both active learning and random selection show a significant decrease in the Gini coefficient”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    Common to both recommender systems and information filters is selection of a subset of data about which people express their preference by a process that is not random sampling, and an iterative learning process in which people's responses to the selected subset are used to train the algorithm for subsequent iterations.

    “Common to both recommender systems and information filters is: (1) selection, of a subset of data about which people express their preference, by a process that is not random sampling, and (2) an iterative learning process in which people’s responses to the selected subset are used to train the algorithm for subsequent iterations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    The data used to train and optimize performance of recommender systems and information filters are based on human actions; thus, data observed and omitted are not randomly selected but are consequences of people's choices.

    “The data used to train and optimize performance of these systems are based on human actions. Thus, data that are observed and omitted are not randomly selected, but are the consequences of people’s choices.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Most previous research has treated algorithmic bias as a static factor, which fails to capture the iterative nature of bias borne from continuous interaction between humans and algorithms.

    “However, most research has so far treated algorithmic bias as a static factor, which fails to capture the dynamic and iterative properties of bias.”
    • Also stated in: § Relationship between iterated algorithmic bias and other types of bias · plain
      “we observe that most previous research has treated algorithmic bias as a static factor, which fails to capture the iterative nature of bias that is borne from continuous interaction between humans and algorithms.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Algorithmic bias evolves with human interaction in an iterative manner, which may have a long-term effect on algorithm performance and humans' discovery and learning.

    “We argue that algorithmic bias interacts with humans in an iterative manner, which has a long-term effect on algorithms’ performance.”
    • Also stated in: § Relationship between iterated algorithmic bias and other types of bias · hedged
      “We argue that algorithmic bias evolves with human interaction in an iterative manner, which may have a long-term effect on algorithm performance and humans’ discovery and learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Human reaction to the output of machine learning methods with algorithmic bias worsens the situations by making decisions based on biased information, which will probably be consumed by algorithms later.

    “On the other hand, human’s reaction to the output of machine learning methods with algorithmic bias worsen the situations by making decision based on biased information, which will probably be consumed by algorithms later.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    When certain assumptions about the independence of language and the world are abandoned, learners converge to languages that depend on the structure as well as their prior biases.

    “Perfors et al. showed that, when certain assumptions about the independence of language and the world are abandoned, the learners converge to languages that depend on the structure as well as their prior biases [78].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [78] A Perfors, 2014, Language evolution can be shaped by the structure of the world, Cognitive science source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    In the paper's setting, there is a dependency between the previous output and the next input, and it is this dependency that can generate iterated bias.

    “More specifically in our setting, there is a dependency between the previous output and the next input, and it is this dependency that can generate iterated bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Selection of relevant data allows the possibility of learning that an input predicted to be relevant is not, but does not allow the possibility of learning that an input predicted to be irrelevant is actually relevant.

    “Specifically, the selection of relevant data allows the possibility of learning that an input that is predicted to be relevant is not, but does not allow the possibility of learning that an input that is predicted to be irrelevant is actually relevant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Both active learning and filtering have different goals, but focus on an ever more extreme (and therefore not representative) subset of data.

    “By deriving the transition matrices implied by iterated application of data selection biases, we can see that both active learning and filtering have different goals, but focus on an ever more extreme (and therefore not representative) subset of data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    After sufficient iterations, iterated learning generates a certain hypothesis; this iterated learning chain is expected to converge the hypothesis to the prior distribution of all hypotheses in case the learner is a Bayesian learner.

    “After sufficient iterations, this process generates a certain hypothesis. This iterated learning chain is expected to converge the hypothesis to the prior distribution of all hypotheses in case that the learner is a Bayesian learner [69].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] TL Griffiths, 2005

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Selecting biased training samples generally leads to a biased model.

    “Generally, selecting biased training samples leads to a biased model [96].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [96] Heckman JJ. Sample selection bias as a specification error (with an application to the estimation of labor supply functions); 1977. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    For a rating based recommendation system, the ranking is based on the prediction from the system, or the probability of relevance from prediction; in both situations, the selected data point x is likely to be highly rated or relevant, given h.

    “For a rating based recommendation system, the ranking is based on the prediction from the system, or the probability of relevance from prediction. In both situations, the selected data point x is likely to be highly rated or relevant, given h.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2] SE Robertson, 1977, The probability ranking principle in IR, Journal of documentation source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    The utility of acting may depend on the value of y*; there may be greater perceived utility in acting when the value of y* is very low, as in the case of an angry customer or disappointed user.

    “Alternatively, the utility of acting may depend on the value of y*. For example, it may be that there is greater perceived utility in acting when the value of y* is very low, as in the case of an angry customer or disappointed user.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Post-algorithmic bias emerges when users interpret the output of machine learning algorithms.

    “The post-algorithmic bias emerges when users interpret the output of machine learning algorithms [97–100].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [97] Baeza-Yates R. Data and algorithmic bias in the web. In: Proceedings of the 8th ACM Conference on Web Science. ACM; 2016. p. 1–1. source

      Not checked in the free run.

    • cites [98] Pedreshi D, Ruggieri S, Turini F. Discrimination-aware data mining. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM; 2008. p. 560–568. source

      Not checked in the free run.

    • cites [99] White R. Beliefs and biases in web search. In: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. ACM; 2013. p. 3–12. source

      Not checked in the free run.

    • cites [100] M Ayub, 2019, Modeling user rating preference behavior to improve the performance of the collaborative filtering based recommender systems, PloS one source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Iterated learning was found to produce meaningful structure patterns in language learning.

    “Iterated learning was found to produce meaningful structure patterns in language learning [72–76].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [72] S Kirby, 2014, Iterated learning and the evolution of language, Current opinion in neurobiology source

      Not checked in the free run.

    • cites [73] S Kirby, 2008, Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language, Proceedings of the National Academy of Sciences source

      Not checked in the free run.

    • cites [74] ML Kalish, 2007, Iterated learning: Intergenerational knowledge transmission reveals inductive biases, Psychonomic Bulletin & Review source

      Not checked in the free run.

    • cites [75] Smith K. Iterated learning in populations of Bayesian agents. In: Proceedings of the 31st annual conference of the cognitive science society. Citeseer; 2009. p. 697–702.

      Not checked in the free run.

    • cites [76] K Smith, 2003, Iterated learning: A framework for the emergence of language, Artificial life source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The selection of inputs depends on the hypothesis, and therefore information is not unbiased, p(x|hn)≠q(x).

    “The selection of inputs depends on the hypothesis, and therefore information is not unbiased, p(x|hn)≠q(x).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The limiting behavior of an algorithm with active learning bias depends on the iterated active learning bias; in most cases, in opposition to the goal of filtering, the algorithm will only select data point(s) closest to the learned model's boundary, if learning a classifier.

    “The limiting behavior depends on the iterated active learning bias, . This is, in most cases, in opposition to the goal of filtering, the algorithm will only select data point(s) which are closest to the learned model’s boundary, if we are learning a classifier for example.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: A simplified algorithm where only the very uncertain data are selected, and a mixture of random sampling and active learning, is assumed.

  • May combine two claims.

    Premisereasoning

    Humans' choices when labelling data are highly non-random and reflect not only their opinions about the presented content but also inferences about why the content was presented.

    “Humans’ choices, when labelling data, are also highly non-random, and would reflect not only their opinions about the presented content, but also inferences about why the content was presented.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Bias introduced into the data at any point can be amplified after retraining models and this in turn would further impact the predictions or recommendations.

    “Finally, the bias that is introduced into the data at any point can be amplified after retraining models and this in turn would further impact the predictions or recommendations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Language learning and machine learning have several properties in common; a 'hypothesis' in language is analogous to a 'model' in machine learning, and learning a language transmitted throughout consecutive generations is analogous to learning an online model throughout consecutive iterations of machine learning.

    “Language learning and machine learning have several properties in common (see Fig 1). For example, a ‘hypothesis’ in language is analogous to a ‘model’ in machine learning. Learning a language which gets transmitted throughout consecutive generations of humans is analogous to learning an online model throughout consecutive iterations of machine learning [69].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] TL Griffiths, 2005

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Recommendation systems have more direct interaction options with humans, while information retrieval focuses on getting relevant information only.

    “One reason is that recommendation systems have more direct interaction options with humans, while information retrieval focuses on getting relevant information only.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Unlike standard learning theory, the training data in human-algorithm interactive systems are no longer randomly sampled, which puts in question the guarantees about learning from such data.

    “Unlike standard learning theory, the training data are no longer randomly sampled. This in return puts in questions the guarantees about learning from such data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Either of the individual sources of bias (algorithmic selection and human labeling) could yield instability in human-algorithm interactive systems.

    “Either of the individual sources of bias could yield instability, thus motivating the need for a better understanding of the performance of such systems in the context of human-algorithm interactions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    The choice of whether to act depends on the relative utility of acting as opposed to not acting; if it is especially effortful to act, then people will be biased against acting.

    “where the choice of whether to act depends on the relative utility of acting as opposed to not acting. For example, if it is especially effortful to act, then people will be biased against acting.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The choice of whether to act can be formalized using Luce choice, a special case of softmax.

  • May combine two claims.

    Interpretationown data

    The nature of the model, and hence which items will be judged to be relevant to the user, changes depending on the iterated algorithmic bias, with filtering bias exerting the biggest influence.

    “This means that the nature of the model, and hence which items will be judged to be relevant to the user, changes depending on the iterated algorithmic bias, with filtering bias exerting the biggest influence.”
    • Also stated in: § Conclusion · hedged
      “indicating that the nature of the model, and hence which items will be judged to be relevant to the user, changes depending on the iterated algorithmic bias, with filtering bias exerting the biggest influence.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Active learning highly depends on the initial data points because it aims to help learn the correct boundary.

    “Active learning aims to help learn correct boundary, therefore it highly depends on the initial data points.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Active learning aims to help learn the correct boundary.

  • May combine two claims.

    Interpretationreasoning

    Active learning affects points close to the boundary, thus it has limited effects overall.

    “Active learning affects points close to the boundary, thus it has limited effects overall.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Active learning bias already prefers points that are close to the boundary.

  • The quote only partly states this claim.

    Interpretationreasoning

    Random selection with class-dependent human action probability ratio 10:1 slightly decreases the inequality.

    “Therefore, it slightly decreases the inequality.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Random selection with class-dependent human action probability prefers points from class y = 1, however randomly.

  • May combine two claims.

    Interpretationown data

    Human action affects the boundary shift: the more frequent human action is, the more significant is the effect on the boundary shift for only the iterated filter algorithmic bias.

    “Thus we conclude that human action affects the boundary shift: the more frequent human action is, the more significant is the effect on the boundary shift for only the iterated filter algorithmic bias.”
    Quote in the paperSubstantial claim
44 of 44

  • The paper relies on this without support.

    People and algorithms are increasingly engaged in interactive processes wherein neither the human nor the algorithms receive unbiased data.

    “Furthermore, people and algorithms are increasingly engaged in interactive processes wherein neither the human nor the algorithms receive unbiased data.”

  • The authors state this as an assumption.

    A Bayesian model for prediction is assumed for the algorithm.

    “Following Griffiths’ model for human learners, we assume a Bayesian model for prediction.”

  • The authors state this as an assumption.

    People's response is always observed in the above analysis.

    “The above analysis assumes that people’s response is always observed.”

  • The authors state this as an assumption.

    People have some target hypothesis h*, which represents optimal performance for the algorithm.

    “Assume that people have some target hypothesis, h*, which represents optimal performance for the algorithm.”

  • The authors state this as an assumption.

    A simplified algorithm where only the very uncertain data are selected, and a mixture of random sampling and active learning, is assumed.

    “Assuming a simplified algorithm where only the very uncertain data are selected, we can investigate the limiting behavior of an algorithm with the active learning bias. Assuming a mixture of random sampling and active learning, we obtain:”

  • The authors state this as an assumption.

    The choice of whether to act can be formalized using Luce choice, a special case of softmax.

    “We can formalize this choice using Luce choice [56], a special case of softmax [137]”

  • The paper relies on this without support.

    The synthetic dataset represents a typical classification problem and is comparable to the real-life dataset after selecting one user and corresponding items.

    “The synthetic dataset represents a typical classification problem, it is the same for our real-life dataset after selecting one user and corresponding items.”

  • The paper relies on this without support.

    In the real world, some data can be invisible to some users because of bias either from users or from the algorithm itself.

    “In the real world, some data can be invisible to some users because of bias either from users or from the algorithm itself.”

  • The paper relies on this without support.

    Class imbalance can emerge from the algorithm intentionally asking users to rate only the most popular items, a common strategy used to collect initial ratings for new users.

    “Class imbalance can also emerge from the algorithm intentionally asking users to rate only the most popular items, a common strategy used to collect initial ratings for new users.”

  • The authors state this as an assumption.

    In RQ2, humans have a prior probability to act that is not dependent on the true label of the item presented.

    “In research question 2 (RQ 2), we assumed that humans have a prior probability to act which is not dependent on the true label of the item presented.”

  • The authors state this as an assumption.

    Humans have a higher probability to label or rate when the item presented is from relevant class y = 1.

    “we try to simulate this by assuming that humans have a higher probability to label or rate when the item presented is from relevant class y = 1 (see Eq 15).”

  • The paper relies on this without support.

    Filter bias already prefers points from class y = 1.

    “since filter bias already prefers points from class y = 1.”

  • The paper relies on this without support.

    Active learning bias already prefers points that are close to the boundary.

    “since this bias already prefers points that are close to the boundary.”

  • The paper relies on this without support.

    Both random selection and active learning are used to build an accurate model in machine learning.

    “both random selection and active learning are used to build an accurate model in machine learning.”

  • The paper relies on this without support.

    Random selection with class-dependent human action probability prefers points from class y = 1, however randomly.

    “Random selection with class-dependent human action probability prefer points from class y = 1, however randomly.”

  • The paper relies on this without support.

    Active learning aims to help learn the correct boundary.

    “Active learning aims to help learn correct boundary”

  • The paper relies on this without support.

    Humans and AI algorithms are both increasingly intertwined in feedback loops that end up feeding biased data to both humans and algorithms.

    “Humans and AI algorithms are both increasingly intertwined in feedback loops that end up feeding biased data to both humans and algorithms.”
17 of 17

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • It is possible for algorithmic bias to occur even with unbiased training data if the model's design or decision-making mechanisms introduce bias.

    “but it is possible for algorithmic bias to occur even with unbiased training data if the model’s design or decision-making mechanisms introduce bias [54,55].”

    From Ethical Considerations of Using ChatGPT in Health Care (2023) · cites this paper as [54]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • MovieLens active-learning boundary shift is both nonsignificant and significant

    Major

    The boundary-shift discussion calls active learning nonsignificant, but Table 15 reports p = 0.002 and p = 0.0004; the following paragraph also concludes that its effect is significant.

  • MovieLens active-learning blind-spot result has conflicting direction and significance

    Major

    The text says active learning significantly increases the class-1 blind spot, but Table 16 reports a positive effect size of 0.66, which means a decrease under the stated initial-minus-final definition. The table caption instead says active learning has no significant impact, despite its p-values of 0.0002 and 0.0001.

  • User 547's movie count differs between text and Table 18

    Minor

    The dataset description says user 547 rated 2312 movies, whereas Table 18 lists 2391 for that user.

  • Active learning's class-1 blind-spot effect is called nonsignificant despite p = 0.03

    Minor

    The text says active learning has no significant effect, but Table 2 reports p = 0.03 in both tests, below the paper's stated p < 0.05 threshold.

  • Active learning has a significant boundary shift in Table 5

    Minor

    The conclusion says the non-filter modes are not affected, but for active learning at paction = 1, Table 5 reports significant Mann-Whitney and t-test p-values.

  • Table 3 discussion calls active learning's blind-spot decrease significant

    Minor

    The discussion describes a significant decrease for active learning, whereas Table 3 gives nonsignificant p-values of 0.5 and 0.18. Its reported negative effect size also indicates an increase, not a decrease, under the table's stated definition.

  • Table 8 misidentifies the sign of filter bias's effect size

    Minor

    The caption calls the filter-bias effect size negative, but the table reports a positive value of 1.6.

7 of 7

Cites

Flagged first, then by citations

7 of the 173 references couldn’t be matched to a known paper.

Research works printed but not linked: 11

  1. Spark KJ. Artificial Intelligence: What can it offer to Information Retrieval. Proceedings of the Informatics 3, Aslib, ed, London. 1978;.
  2. McFadden DL. Quantal choice analaysis: A survey. In: Annals of Economic and Social Measurement, Volume 5, number 4. NBER; 1976. p. 363–390.
  3. TL Griffiths, 2005, Proceedings of the Cognitive Science Society
  4. Smith K. Iterated learning in populations of Bayesian agents. In: Proceedings of the 31st annual conference of the cognitive science society. Citeseer; 2009. p. 697–702.
  5. KJ Rothman, 2008, Modern epidemiology
  6. Dwork C, Immorlica N, Kalai AT, Leiserson MD. Decoupled classifiers for group-fair and efficient machine learning. In: Conference on Fairness, Accountability and Transparency; 2018. p. 119–133.
  7. Barocas S. Data mining and the discourse on discrimination. In: Data Ethics Workshop, Conference on Knowledge Discovery and Data Mining; 2014.
  8. Schnabel T, Swaminathan A, Singh A, Chandak N, Joachims T. Recommendations as treatments: Debiasing learning and evaluation. arXiv preprint arXiv:160205352. 2016;.
  9. A Sinha, 2016, Advances in neural information processing systems, 3243
  10. S Milano, 2020, AI & SOCIETY, 1
  11. A Stuart, 1994, Distribution theory

Cited by

Sorted by citations

Related

Authors
Sun, W., Nasraoui, O., & Shafto, P.
Affiliations
  • University of Louisville (US)
  • Rutgers, The State University of New Jersey (US)
Corresponding author
Nasraoui, Olfa
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2020-08-13
Volume
15
Issue
8
Pages
e0235502
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0235502
PMC
PMC7425868
OpenAlex
W3048367007
Semantic Scholar
fba50a0f6e7f1bb47e80ffa8c523396aa747a6ae
Cited by
116
References
173
Citations by year
2026: 15 · 2025: 29 · 2024: 16 · 2023: 21 · 2022: 26 · 2021: 8 · 2020: 1
FWCI
19.26
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Division of Information and Intelligent Systems (1549981)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
algorithmic bias, human-machine interaction, iterated learning, filter bubbles, personalization, active learning (machine learning), class imbalance, personalized interfaces, information filtering, relevance prediction
MeSH terms
Machine Learning, Humans, Learning, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review