Bias in data‐driven artificial intelligence systems—An introductory survey

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

AI-based decision making may magnify pre-existing biases and evolve new classifications and criteria with huge potential for new types of biases.

“However, AI-based decision making may magnify pre-existing biases and evolve new classifications and criteria with huge potential for new types of biases.”

May combine two claims.

Algorithms may reproduce or even increase existing inequalities or discriminations.

“As a result, algorithms may reproduce (or even increase) existing inequalities or discriminations (Karimi, Génois, Wagner, Singer, & Strohmaier, 2018).”

• cites Karimi, Génois, Wagner, Singer, & Strohmaier, 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The problems of bias and discrimination are not limited to AI and the technology can be deployed in ways that reflect, amplify or distort real world perception and status quo.

“We are aware that the problems of bias and discrimination are not limited to AI and that the technology can be deployed (consciously or unconsciously) in ways that reflect, amplify or distort real world perception and status quo.”

May combine two claims.

Biases existing in humans enter AI systems and are amplified due to complex sociotechnical systems.

“Therefore, whatever biases exist in humans enter our systems and even worse, they are amplified due to the complex sociotechnical systems, such as the Web.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There has been an exceptional growth of research literature on interpretable ML in recent years, with emerging keywords such as explainable AI and black box explanation.

“Although attempts to tackle interpretable ML have existed for some time (Hoffman & Klein, 2017), there has been an exceptional growth of research literature in the last years with emerging keywords such as explainable AI (Adadi & Berrada, 2018) and black box explanation (Guidotti et al., 2019).”

• cites Hoffman & Klein, 2017Not checked yet

  Not checked in the free run.

• cites Adadi & Berrada, 2018Not checked yet

  Not checked in the free run.

• cites Guidotti et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Expressive ontological formalisms are demanded to represent the contextual dimensions of various types of sources, as well as annotations denoting causality and provenance of the represented knowledge.

“Therefore, expressive ontological formalisms are demanded to represent the contextual dimensions of various types of sources, e.g., data collections, knowledge bases, or ontologies, as well as annotations denoting causality and provenance of the represented knowledge.”

Classified as probably not a substantial claim.

The difficulty of evaluating bias mitigation methods lies in the fact that different methods work with different fairness notions and are applicable to different AI models.

“The difficulty of the evaluation lies on the fact that different methods work with different fairness notions and are applicable to different AI models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Knowledge bases and ontologies can also suffer from various types of bias or knowledge imbalance.

“Moreover, as any type of data source, knowledge bases and ontologies can also suffer from various types of bias or knowledge imbalance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Local explanation approaches are either model-dependent or model-agnostic.

“Such approaches are either model-dependent, e.g., Taylor approximations (Kasneci & Gottron, 2016), saliency masks (the image regions that are mainly responsible for the decision) for neural network decisions (Ma, Yu, & Yue, 2015), and attention models for recurrent networks (Choi et al., 2016), or model-agnostic, such as those started by the LIME method (Ribeiro, Singh, & Guestrin, 2016).”

• cites Kasneci & Gottron, 2016Not checked yet

  Not checked in the free run.

• cites Ribeiro, Singh, & Guestrin, 2016Not checked yet

  Not checked in the free run.

May combine two claims.

Existing EU regulation comes into play when discriminatory decisions have been taken, while provisions tackling the quality of selected data are rare.

“Generally speaking, existing EU regulation comes into play when (discriminatory) decisions have been taken, while provisions tackling the quality of selected data are rare.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A third stream aims at bridging the local and the global ones by defining a strategy for combining local models in an incremental way.

“A third stream aims at bridging the local and the global ones by defining a strategy for combining local models in an incremental way (Pedreschi et al., 2019).”

• cites Pedreschi et al., 2019Not checked yet

  Not checked in the free run.

May combine two claims.

Algorithms shape societal institutions and potential interventions, and vice versa.

“Algorithms thus shape societal institutions and potential interventions, and vice versa.”

Classified as probably not a substantial claim.

The intuition behind pre-processing approaches is that the fairer the training data is, the less discriminative the resulting model will be.

“The intuition behind these approaches is that the fairer the training data is, the less discriminative the resulting model will be.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

White-box post-processing approaches have not been further developed in recent years, being superseded by in-processing methods.

“White-box approaches have not been further developed in recent years, being superseded by in-processing methods.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In-processing approaches reformulate the classification problem by explicitly incorporating the model's discrimination behaviour in the objective function through regularization or constraints, or by training on latent target labels.

“In-processing approaches reformulate the classification problem by explicitly incorporating the model's discrimination behaviour in the objective function through regularization or constraints, or by training on latent target labels.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A different stream of approaches focuses on the local behavior of a model, searching for an explanation of the decision made for a specific instance.

“A different stream of approaches focuses on the local behavior of a model, searching for an explanation of the decision made for a specific instance (Guidotti et al., 2019).”

• cites Guidotti et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The problem of formalising fairness is still open, as is the discussion about the merits and demerits of the different measures.

“Despite the many formal, mathematical definitions of fairness proposed over the last years the problem of formalising fairness is still open as well as the discussion about the merits and demerits of the different measures.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

There are still no conclusive results regarding what is the state of the art method for each category of bias mitigation, which fairness-related interventions perform best, or whether category-specific interventions perform better than holistic approaches.

“First, despite the large number of methods for mitigating bias, there are still no conclusive results regarding what is the state of the art method for each category, which of the fairness-related interventions perform best, or whether category-specific interventions perform better comparing to holistic approaches that tackle bias at all stages of the analysis process.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Bias can be manifested in data through sensitive features and their causal influences, or through under/over-representation of certain groups.

“Bias can be manifested in (multi-modal) data through sensitive features and their causal influences, or through under/over-representation of certain groups.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The main idea of LIME is to derive a local explanation for a decision outcome on a specific instance by learning an interpretable model from a randomly generated neighbourhood of the instance.

“The main idea is to derive a local explanation for a decision outcome on a specific instance by learning an interpretable model from a randomly generated neighbourhood of the instance.”

• cites Ribeiro, Singh, & Guestrin, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Approaches for bias mitigation can be categorised into pre-processing methods focusing on the data, in-processing methods focusing on the ML algorithm, and post-processing methods focusing on the ML model.

“Approaches for bias mitigation can be categorised into: i) pre-processing methods focusing on the data, ii) in-processing methods focusing on the ML algorithm, and iii) post-processing methods focusing on the ML model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent work revealed that DNNs are vulnerable to adversarial attacks, i.e., intentional perturbations of the input examples.

“In the same topic, recent work revealed that DNNs are vulnerable to adversarial attacks, i.e., intentional perturbations of the input examples, and therefore there is a need for methods to enhance their resilience (Song et al., 2018).”

• cites Song et al., 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

More than 20 different definitions of fairness have appeared in the computer science literature.

“More than 20 different definitions of fairness have appeared thus far in the computer science literature (Verma & Rubin, 2018;Zliobaite, 2017); and some of these definitions and others were proposed and investigated in work on formalizing fairness from other disciplines, such as education, over the past 50 years (Hutchinson & Mitchell, 2019).”

• cites Verma & Rubin, 2018Not checked yet

  Not checked in the free run.

• cites Zliobaite, 2017Not checked yet

  Not checked in the free run.

May combine two claims.

Pre-processing methods use heuristics aiming to balance protected and unprotected groups in the training set; however, their impact is not well controlled despite their efforts for minimal data interventions.

“These methods use heuristics aiming to balance the protected and unprotected groups in the training set; however, their impact is not well controlled despite their efforts for minimal data interventions.”

Probably not a substantial claim: may not pass S2 (checkable).

Domain modelling using ontologies will support context-aware bias description and interpretability.

“Thus, domain modelling using ontologies will support context-aware bias description and interpretability.”

May combine two claims.

Corbett-Davies & Goel (2018) show the statistical limitations of prevailing mathematical definitions of fairness and the negative effect of enforcing such fairness-measures on group well-being.

“(Corbett-Davies & Goel, 2018) show the statistical limitations of prevailing mathematical definitions of fairness and the (negative) effect of enforcing such fairness-measures on group well-being and urge the community to explicitly focus on consequences of potential interventions.”

• cites Corbett-Davies & Goel, 2018Not checked yet

  Not checked in the free run.

May combine two claims.

Once introduced, algorithmic systems encourage the creation of very specific data collection infrastructures and policies, which then change or amplify power relations.

“Once introduced, algorithmic systems encourage the creation of very specific data collection infrastructures and policies, for example, they may suggest tracking and surveillance (Introna & Wood, 2004) which then change or amplify power relations.”

• cites Introna & Wood, 2004Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent work has examined fairness in the reinforcement learning context, where one needs to reconsider the effects of short-term actions on long-term rewards.

“Recent works have emerged in this direction, e.g., (Jabbari, Joseph, Kearns, Morgenstern, & Roth, 2017) examine fairness in the RL context where one needs to reconsider the effects of short-term actions on long-term rewards.”

• cites Jabbari, Joseph, Kearns, Morgenstern, & Roth, 2017Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

In case of AI algorithms the problem of accountability is aggravated as we do not codify the solution, rather the solution is inferred via machine learning algorithms and complex data.

“In case of AI algorithms the problem is aggravated as we do not codify the solution, rather the solution is inferred via machine learning algorithms and complex data.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

The notion of fairness varies across countries, cultures and application domains.

“Also,”

• cites Schäfer, Haun, & Tomasello, 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Certain data may be easier to access and analyse than others, which has caused the role of Twitter for various societal phenomena to be overemphasized.

“This problem is exacerbated by the fact that certain data may be easier to access and analyse than others, which has caused, for example, the role of Twitter for various societal phenomena to be overemphasized (Tufekci, 2014).”

• cites Tufekci, 2014Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Post-processing approaches alter the classification model after it has been learned, either by altering the model's internals (white-box) or its predictions (black-box).

“The third strategy is to post-process the classification model once it has been learned from data.This consists of altering the model's internals (white-box approaches) or its predictions (black-box approaches).”

Probably not a substantial claim: may not pass S4 (based).

The problem of representativeness of the training data and its impact on the representativeness of the generated data might aggravate issues of fairness and discrimination.

“Still however, the problem of representativeness of the training data and its impact on the representativeness of the generated data might aggravate issues of fairness and discrimination.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The additional layer of feature extraction typically used within AI-based multimodal analysis systems makes it even more challenging to trace the source of bias in such systems.

“Overall, the additional layer of feature extraction that is typically used within AI-based multimodal analysis systems makes it even more challenging to trace the source of bias in such systems.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AI knowledge representation has been developing a rich theory of argumentation over the last 25 years, which recent approaches try to leverage for generalizing the reasoning aspect of ML towards the use of computational models of argumentation.

“AI knowledge representation has been developing a rich theory of argumentation over the last 25 years (Dung, 1995), which recent approaches (Cocarascu & Toni, 2016) try to leverage for generalizing the reasoning aspect of ML towards the use of computational models of argumentation.”

• cites Dung, 1995Not checked yet

  Not checked in the free run.

• cites Cocarascu & Toni, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Recently an ISO standard on data quality (ISO 8000) was published, though not binding and not with regard to decisionmaking techniques.

“Recently an ISO standard on data quality (ISO 8000) was published, though not binding and not with regard to decisionmaking techniques.”

• cites ISO 8000Not checked yet

  Not checked in the free run.