Think about the stakeholders first! Toward an algorithmic transparency playbook for regulatory compliance

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In the long-term unemployment use case, there were two main purposes of transparency: to understand why an individual was assigned to a particular risk category, and to understand what could be done to help high-risk individuals lower their chances of remaining long-term unemployed.

“In the long-term unemployment use case, there were two main purposes of transparency: to understand why an individual was assigned to a particular risk category, and to understand what could be done to help high-risk individuals lower their chances of remaining longterm unemployed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the IEFP case study, the job counselor may use the explanation to offer recommendations for recourse against a high-risk score.

“Furthermore, the job counselor may also use the explanation to offer recommendations for recourse against a high-risk score.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The IEFP ADS generates explanations about what factors contribute to an individual being classified as low, medium, or high risk for long-term unemployment, which job counselors use to help make better treatment decisions.

“The ADS generates explanations about what factors contribute to an individual being classified as low, medium, or high risk for long-term unemployment, which job counselors use to help make better treatment decision.”

May combine two claims.

In the development of the IEFP AI, explanations were created to meet the varying goals of practitioners, policymakers, and job counselors, but affected individuals were not considered.

“In the development of the AI, explanations were created to meet the varying goals of many of these stakeholders including practitioners, policymakers, and the job counselors.Unfortunately, and significantly, affected individuals were not considered.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the IEFP case study, transparency is built into the ADS with the goal of offering learning and support to job counselors.

“In our case study, transparency is built into the ADS with the goal of offering learning and support to job counselors.”

May combine two claims.

There is no normative consensus around AI transparency, and most policy debates are focused on the risks of AI rather than the opportunities.

“there is no normative consensus around AI transparency, and most policy debates are focused on the risks of AI rather than the opportunities (Gasser and Almeida, 2017)”

• cites Gasser and Almeida, 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Although many regulations specifically mention AI systems, all the ideas they contain about transparency could be applied to all automated decision systems.

“Notably, while many regulations are written to specifically mention “AI systems,” all the ideas they contain about transparency could be applied to all ADS.”

May combine two claims.

The meaning and scope of the GDPR right to explanation are still debated by legal scholars, with little of the discussion resulting in concrete benefits for citizens.

“Despite being passed into law in 2016, the meaning and scope of the right are still being debated by legal scholars, with little of the discussion resulting in concrete benefits for citizens (Selbst and Powles, 2018).”

• cites Selbst and Powles, 2018Not checked yet

  Not checked in the free run.

May combine two claims.

A meta-analysis of over 80 AI ethics guidelines and soft laws found that 87% mention transparency and include an effort to increase the explainability of AI systems.

“A meta-analysis of over 80 AI ethics guidelines and soft laws found that 87% mention transparency, and include an effort to increase the explainability of AI systems (Jobin et al., 2019).”

• cites Jobin et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Transparency and its related terms cannot trivially be quantified or measured, and transparency for one stakeholder does not automatically imply the same for different stakeholders.

“Furthermore, transparency and its related terms cannot trivially be quantified or measured, and transparency for one stakeholder does not automatically imply the same for different stakeholders (Lipton, 2018;Hind, 2019;Larsson and Heintz, 2020).”

• cites Lipton, 2018Not checked yet

  Not checked in the free run.

• cites Hind, 2019Not checked yet

  Not checked in the free run.

• cites Larsson and Heintz, 2020Not checked yet

  Not checked in the free run.

May combine two claims.

Where AI regulation has specificity, it lacks mechanisms for enforcing laws and holding institutions using AI accountable.

“Where there is specificity, there is a lack of mechanisms for enforcing laws and holding institutions using AI accountable.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The knowledge of technologists will always outpace legislators' ability to draft meaningful legislation.

“Assuming any reasonable rate of advancement in AI technology, the knowledge of technologists will always outpace legislator's ability to draft meaningful legislation.”

• AssumptionThe authors state this as an assumption: AI technology will continue to advance at a reasonable rate.

May combine two claims.

Data visualizations have proved useful for informing users and making complex information more accessible and digestible, and have been found to have a powerful persuasive effect.

“Data visualizations have proved useful for informing users and making complex information more accessible and digestible, and have even been found to have a powerful persuasive effect (Pandey et al., 2014;Tal and Wansink, 2016).”

• cites Pandey et al., 2014Not checked yet

  Not checked in the free run.

• cites Tal and Wansink, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Methods like selectregress-round simplify black-box models into interpretable models that use a similar set of features.

“There are also methods like selectregress-round that simplify black-box models into interpretable models that use a similar set of features (Jung et al., 2017).”

• cites Jung et al., 2017Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Interpretable models like linear regression, decision trees, or rules-based models have intrinsic transparency mechanisms that offer algorithmic transparency.

“Interpretable models, like linear regression, decision trees, or rules-based models, have intrinsic transparency mechanisms that offer algorithmic transparency, like the linear formula, the tree diagram, and the set of rules, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Transparency methods have focused on generating explanations for three different points in an ADS pipeline: the data (preprocessing), the model/algorithm (inprocessing, intrinsic), or the outcome (postprocessing, post hoc).

“Transparency methods have focused on generating explanations for three different”

• cites Ventocilla et al., 2018Not checked yet

  Not checked in the free run.

• cites Arya et al., 2020Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The machine learning community accepts two classifications of models: intrinsically transparent (white-box) models and non-intrinsically transparent (black-box) models.

“The machine learning community accepts two classifications of models: those that are intrinsically transparent by their nature (sometimes called directly interpretable or white-box models), and those that are not (called black box models) (Marcinkevics and Vogt, 2020).”

• cites Marcinkevics and Vogt, 2020Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Both the proposed Algorithmic Accountability Act in the United States and the Artificial Intelligence Act in the European Union specifically mention that ADS should have transparency mechanisms that allow individuals to have recourse against a system outcome.

“For instance, both the proposed Algorithmic Accountability Act in the United States and the Artificial Intelligence Act in the European Union specifically mention that ADS should have transparency mechanisms that allow individuals to have recourse against a system outcome.”

May combine two claims.

There is no single approach to designing transparent systems for disparate stakeholder groups, and each stakeholder group has its own goals and purposes for wanting to understand an ADS.

“While there is some overlap in how these stakeholders may think about transparency, in general, there is no single approach to designing transparent systems for these disparate stakeholder groups, and each of them has their own goals and purposes for wanting to understand an ADS (Sokol and Flach, 2020).”

• cites Sokol and Flach, 2020Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Transparency questions are context-specific and should address a direct transparency goal of the stakeholder, and they can be mapped onto known methods for creating explanations, so a well-defined purpose for transparency acts as a bridge between goals and methods.

“These questions have two important characteristics.First, they are context-specific and should address a direct transparency goal of the stakeholder.Second, and importantly for technologists, these questions can be mapped onto known methods for creating explanations, meaning that a well-defined purpose for transparency acts as a bridge between the goals and methods.”

• cites Liao et al., 2020Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

These methods are considered model-agnostic because they can be used to create explanations for any model, from linear models to random forests to neural networks.

“These methods are considered model-agnostic because they can be used to create explanations for any model, from linear models to random forests to neural networks.”

May combine two claims.

Understanding of a machine learning model is negatively correlated with its complexity, and decision trees are among the model types most understood by users.

“A user study found that the understanding of a machine learning model is negatively correlated with its complexity, and found decision trees to be among the model types most understood by users (Allahyari and Lavesson, 2011).”

• cites Allahyari and Lavesson, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

In the IEFP ADS, there are four main stakeholders: the technologists who developed the ADS, the policymakers who reviewed the ADS and passed laws for its implementation, the job counselors who use the system, and the affected individuals who are assessed for long-term unemployment.

“In the ADS used by IEFP in Portugal, there are four main stakeholders: the technologists who developed the ADS, the policymakers who reviewed the ADS and passed laws for its implementation, the job counselors who use the system, and the affected individuals who are assessed for long-term unemployment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LIME creates a linear regression to approximate an underlying black-box model.

“For example, LIME creates a linear regression to approximate an underlying black-box model (Lundberg and Lee, 2017).”

• cites Lundberg and Lee, 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Some methods create a transparent surrogate model that mimics the behavior of a black-box model.

“Some methods create a transparent surrogate model that mimics the behavior of a blackbox model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Explanations can be classified on attributes including scope (local or global), intrinsic or post hoc, data or model, model-agnostic or model-specific, surrogate or model behavior, and static or interactive.

“There are several existing taxonomies of these methods, which show that explanations can be classified on a number of attributes like the scope (local or global), intrinsic or post hoc, data or model, model-agnostic or modelspecific, surrogate or model behavior, and static or interactive (Molnar, 2019;Arya et al., 2020;Marcinkevics and Vogt, 2020).”

• cites Molnar, 2019Not checked yet

  Not checked in the free run.

• cites Arya et al., 2020Not checked yet

  Not checked in the free run.

• cites Marcinkevics and Vogt, 2020Not checked yet

  Not checked in the free run.

May combine two claims.

The IEFP ADS assigns unemployed persons as low, medium, or high risk for remaining unemployed, and job counselors assign them to interventions such as reskilling, resume building, or job search training.

“The ADS assigns unemployed persons as low, medium, or high risk for remaining unemployed, and then job counselors have the responsibility of assigning them to interventions such as reskilling, resume building, or job search training (Zejnilović et al., 2020).”

• cites Zejnilović et al., 2020 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

If someone is rejected for a loan and the reason for that decision is the person's age, there is no action that they can effectively take for recourse against that decision.

“someone is rejected for a loan and the reason for that decision is the person's age, there is no action that they can effectively take for recourse against that decision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some of the most popular post hoc methods are LIME, SHAP, SAGE, and QII.

“Some of the most popular post hoc methods are LIME, SHAP, SAGE, and QII (Datta et al., 2016;Ribeiro et al., 2016;Lundberg and Lee, 2017;Covert et al., 2020).”

• cites Datta et al., 2016Not checked yet

  Not checked in the free run.

• cites Ribeiro et al., 2016Not checked yet

  Not checked in the free run.

• cites Lundberg and Lee, 2017Not checked yet

  Not checked in the free run.

May combine two claims.

Visualizations can be abused and can successfully misrepresent a message through techniques like exaggeration or understatement.

“However, visualizations should be deployed thoughtfully, as they have the ability to be abused and can successfully misrepresent a message through techniques like exaggeration or understatement (Pandey et al., 2015).”

• cites Pandey et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Researchers have created a number of different tools to accomplish transparency in ADS.

“Furthermore, researchers have created a number of different tools to accomplish transparency in ADS (Datta et al., 2016;Ribeiro et al., 2016;Lundberg and Lee, 2017;Covert et al., 2020).”

• cites Datta et al., 2016Not checked yet

  Not checked in the free run.

• cites Ribeiro et al., 2016Not checked yet

  Not checked in the free run.

• cites Lundberg and Lee, 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Having a human-in-the-loop is an increasingly common practice for implementing automated decision systems.

“Having a HITL is an increasingly common practice for implementing ADS (Raso, 2017;Gillingham, 2019;Wagner, 2019).”

• cites Raso, 2017Not checked yet

  Not checked in the free run.

• cites Gillingham, 2019Not checked yet

  Not checked in the free run.

• cites Wagner, 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Miller proposed that the purpose of transparency is to answer a why question.

“Miller proposed that the purpose of transparency is to answer a “why” question (Miller, 2017), and gives the following example:”

• cites Miller, 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Had the practitioners adopted a robust stakeholder-first approach to designing transparent systems, they could have better considered how to meet the goals of affected individuals in the IEFP ADS.

“Had the practitioners adopted a robust stakeholder-first approach to designing transparent systems they could have better considered how to meet the goals of this key stakeholder group.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Delgado et al. identified five dimensions containing critical questions that can be used to properly guide participatory design in XAI.

“Rather, they identified five dimensions containing critical questions like”

• cites Delgado et al., 2021Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Participatory design, where stakeholders are included directly in design conversations, is a meaningful low-level step that technologists can take to implement transparency into ADS.

“One such step is the use of participatory design, where stakeholders are included directly in design conversations (Eiband et al., 2018;Aizenberg and Van Den Hoven, 2020;Gupta and De Gasperis, 2020;Cech, 2021).”

• cites Eiband et al., 2018Not checked yet

  Not checked in the free run.

• cites Aizenberg and Van Den Hoven, 2020Not checked yet

  Not checked in the free run.

• cites Gupta and De Gasperis, 2020Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

There is considerable overlap between stakeholders' scope needs, but generally, it is important which scope an explanation has.

“Naturally, there is considerable overlap between stakeholders' scope needs (for example, an auditor may want to inspect a model globally and look at local cases), but generally, it is important which scope an explanation has.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The transparency provided by SHAP provided a local explanation about the outcome of the model.

“The transparency provided by SHAP provided a local explanation about the outcome of the model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Others have advocated for fully holistic views of stakeholders that include their values, interpersonal dynamics, and the social context of the AI system.

“Others have advocated for fully holistic views of stakeholders that include their values, interpersonal dynamics, and the social context of the AI system (Ehsan and Riedl, 2020;Kaur et al., 2022).”

• cites Ehsan and Riedl, 2020Not checked yet

  Not checked in the free run.

• cites Kaur et al., 2022Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Participatory design was used to successfully embed positive data-related values in nonprofit organizations in Australia.

“In another example, participatory design was used to successfully embed positive data-related values in nonprofit organizations in Australia (McCosker et al., 2022).”

• cites McCosker et al., 2022Not checked yet

  Not checked in the free run.

May combine two claims.

System transparency can improve user-system performance and can play a critical role in avoiding harmful societal risks.

“Researchers have demonstrated that system transparency can improve user-system performance, and can play a critical role in avoiding harmful societal risks (Zhou et al., 2019;Burgt, 2020).”

• cites Zhou et al., 2019Not checked yet

  Not checked in the free run.

• cites Burgt, 2020 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The scope of an explanation is highly relevant to the stakeholder and goals of an explanation, and is related to whether the stakeholder operates at a system or individual level.

“The scope of an explanation is highly relevant to the stakeholder and goals of an explanation, and is related to whether the stakeholder operates at a system or individual level.”

Classified as probably not a substantial claim.

Thinking about stakeholders implies considering a range of aspects about stakeholders like their socio-technical background, historically relevant information, and the relative power structures in which they exist.

“Primarily, it is the idea that”

• cites Ehsan and Riedl, 2020Not checked yet

  Not checked in the free run.

• cites Delgado et al., 2021Not checked yet

  Not checked in the free run.

• cites Ehsan et al., 2021Not checked yet

  Not checked in the free run.

May combine two claims.

Policymakers and ADS compliance officers are more apt to be concerned with system-level goals, while humans-in-the-loop and affected individuals are likely more interested in local explanations.

“Policymakers and ADS compliance officers are more apt to be concerned with systemlevel goals, like ensuring that the ADS is fair, respects privacy, and is valid overall, while humans-in-the-loop and those individuals affected by the outcome of an ADS are likely more interested in seeing local explanations to pertain to their specific cases.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Researchers found that the explanations improved the confidence of the decisions, but counterintuitively, had a somewhat negative effect on the quality of those decisions.

“Overall, researchers found that the explanations improved the confidence of the decisions, but counterintuitively, had a somewhat negative effect on the quality of those decisions (Zejnilović et al., 2020).”

• cites Zejnilović et al., 2020 sourceNot checked yet

  Not checked in the free run.