Evaluating and Mitigating Discrimination in Language Model Decisions

“…2015; Ali et al., 2019; Imana et al., 2021), mortgage approvals (Martinez & Kirchner, 2021), resume screening (Dastin, 2018), recidivism prediction (Skeem & Lowenkamp, 2016), hiring (Kirk et al., 2021; Bommasani et al., 2022a; Veldanda et al., 2023), and medical treatment (Obermeyer et al., 2019).”

“As language models are increasingly adopted across society, interest is growing in deploying them in an expanding range of societal applications, from finance to medicine, to routine business tasks (Wu et al., 2023; Thirunavukarasu et al., 2023).”

“In response, researchers and activists have developed a range of theoretical frameworks (Dwork et al., 2012; Kusner et al., 2017; Ustun et al., 2019) as well as investigative practices (Raji & Buolamwini, 2019; Metaxa et al., 2021; Vecchione et al., 2021; Bandy, 2021), to detect and counteract such…”

“We also insert requests asking the model to verbalize its reasoning process (Kojima et al., 2022) while keeping into account various fairness constraints.”

“As a step in this direction, we focus on measuring the risk of widespread automated discrimination (Corbett-Davies et al., 2017; Kasy & Abebe, 2021; Starke et al., 2022; Creel & Hellman, 2022) via automated language model decisions.”

“Our work complements these studies by using language models to generate the correspondence studies (across a wide range of hypothetical scenarios) and evaluating machine learning models as subjects in the correspondence studies (in order to quantify discrimination in LMs).”

“In addition, models interact with people and institutions in complex ways, including via automation bias (Skitka et al., 1999; Goddard et al., 2012), meaning that while AI-aided decision-making can have positive effects (Keding & Meissner, 2021), placing humans in an advisory in-the-loop capacity…”

“…of acceptance rates across such characteristics is then considered evidence of discrimination, and such studies have uncovered discrimination in fields as diverse as hiring, housing, and lending (Cain, 1996; Riach & Rich, 2002; Bertrand & Mullainathan, 2004; Pager, 2007; Bertrand & Duflo, 2017).”

“…of acceptance rates across such characteristics is then considered evidence of discrimination, and such studies have uncovered discrimination in fields as diverse as hiring, housing, and lending (Cain, 1996; Riach & Rich, 2002; Bertrand & Mullainathan, 2004; Pager, 2007; Bertrand & Duflo, 2017).”

“For example, Sweeney (2013) study discrimination in online ad delivery, finding that ads suggesting an arrest record are more likely to be shown with searches of Black-associated names.”

“In addition, models interact with people and institutions in complex ways, including via automation bias (Skitka et al., 1999; Goddard et al., 2012), meaning that while AI-aided decision-making can have positive effects (Keding & Meissner, 2021), placing humans in an advisory in-the-loop capacity…”

“A range of other investigations have further studied algorithmic discrimination, including in other ad delivery settings (Datta et al., 2015; Ali et al., 2019; Imana et al., 2021), mortgage approvals (Martinez & Kirchner, 2021), resume screening (Dastin, 2018), recidivism prediction (Skeem &…”

“…and activists have developed a range of theoretical frameworks (Dwork et al., 2012; Kusner et al., 2017; Ustun et al., 2019) as well as investigative practices (Raji & Buolamwini, 2019; Metaxa et al., 2021; Vecchione et al., 2021; Bandy, 2021), to detect and counteract such discrimination.”

“Finally, the sensitivity of models to small changes in prompts is an unsolved problem (Lu et al., 2021; Ganguli & Favaro, 2023), and despite our efforts it remains possible that variations in the construction of our prompts (including the phrasing of the prompts, the examples we included in the…”

“Finally, the sensitivity of models to small changes in prompts is an unsolved problem (Lu et al., 2021; Ganguli & Favaro, 2023), and despite our efforts it remains possible that variations in the construction of our prompts (including the phrasing of the prompts, the examples we included in the…”

“…studied how discrimination can happen directly , based on demographic variables, as well as indirectly though proxies for those variables, such as zip code, extracurricular activities, or other features (VanderWeele & Robinson, 2014; Datta et al., 2017; Kilbertus et al., 2017; Adler et al., 2018).”

“In response, researchers and activists have developed a range of theoretical frameworks (Dwork et al., 2012; Kusner et al., 2017; Ustun et al., 2019) as well as investigative practices (Raji & Buolamwini, 2019; Metaxa et al., 2021; Vecchione et al., 2021; Bandy, 2021), to detect and counteract such…”

“…of acceptance rates across such characteristics is then considered evidence of discrimination, and such studies have uncovered discrimination in fields as diverse as hiring, housing, and lending (Cain, 1996; Riach & Rich, 2002; Bertrand & Mullainathan, 2004; Pager, 2007; Bertrand & Duflo, 2017).”

“In the context of LMs, Schick et al. (2021) note that language models can recognize toxicity in their own outputs, and Si et al. (2022) demonstrated that the right prompts could reduce bias in language models on the BBQ benchmark (Parrish et al., 2021 Model-generated evaluations Finally, a range of…”

“Our work complements these studies by using language models to generate the correspondence studies (across a wide range of hypothetical scenarios) and evaluating machine learning models as subjects in the correspondence studies (in order to quantify discrimination in LMs).”

“…studied how discrimination can happen directly , based on demographic variables, as well as indirectly though proxies for those variables, such as zip code, extracurricular activities, or other features (VanderWeele & Robinson, 2014; Datta et al., 2017; Kilbertus et al., 2017; Adler et al., 2018).”

“As language models are increasingly adopted across society, interest is growing in deploying them in an expanding range of societal applications, from finance to medicine, to routine business tasks (Wu et al., 2023; Thirunavukarasu et al., 2023).”

“An inherent challenge of evaluations is external validity : ensuring the conclusions of a research study generalize to real-world settings (Andrade, 2018).”

“Instead, we expect that a sociotechnical lens (Carayon et al., 2015) will be necessary to ensure beneficial outcomes for these technologies, including both policies within individual firms as well as the broader policy and regulatory environment.”

“Fourth, choosing names in audit studies that are associated with different demographics can be challenging (Gaddis, 2017; Crabtree & Chykina, 2018); while we make a best effort in this study, more work is necessary, as well as an investigation into other sources of proxy discrimination.”

“…studied how discrimination can happen directly , based on demographic variables, as well as indirectly though proxies for those variables, such as zip code, extracurricular activities, or other features (VanderWeele & Robinson, 2014; Datta et al., 2017; Kilbertus et al., 2017; Adler et al., 2018).”

“A natural question raised by our work is under what circum-stances (and to what degree) positive discrimination should be corrected for, given arguments raised by proponents and critics of affirmative action policies and compensatory justice (Fullinwider, 2018; Dwork et al., 2012; Eidelson, 2015).”

“…real world, people might use a wider variety of input formats, including longer, supplementary documents such as resumes or medical records, or might engage in more interactive dialogues with models (Jakesch et al., 2023; Li et al., 2023a), each of which has the potential to affect our conclusions.”

“Finally, discrimination and other fairness criteria are not the only important factors when deciding whether or not to deploy a model; models must be evaluated in naturalistic settings to ensure they perform satisfactorily at the tasks they are applied to (Raji et al., 2022; Sanchez et al., 2023).”

“…of acceptance rates across such characteristics is then considered evidence of discrimination, and such studies have uncovered discrimination in fields as diverse as hiring, housing, and lending (Cain, 1996; Riach & Rich, 2002; Bertrand & Mullainathan, 2004; Pager, 2007; Bertrand & Duflo, 2017).”

“In the context of LMs, Schick et al. (2021) note that language models can recognize toxicity in their own outputs, and Si et al. (2022) demonstrated that the right prompts could reduce bias in language models on the BBQ benchmark (Parrish et al., 2021 Model-generated evaluations Finally, a range of recent works explor…”

“…and activists have developed a range of theoretical frameworks (Dwork et al., 2012; Kusner et al., 2017; Ustun et al., 2019) as well as investigative practices (Raji & Buolamwini, 2019; Metaxa et al., 2021; Vecchione et al., 2021; Bandy, 2021), to detect and counteract such discrimination.”

“As a step in this direction, we focus on measuring the risk of widespread automated discrimination (Corbett-Davies et al., 2017; Kasy & Abebe, 2021; Starke et al., 2022; Creel & Hellman, 2022) via automated language model decisions.”

“…and activists have developed a range of theoretical frameworks (Dwork et al., 2012; Kusner et al., 2017; Ustun et al., 2019) as well as investigative practices (Raji & Buolamwini, 2019; Metaxa et al., 2021; Vecchione et al., 2021; Bandy, 2021), to detect and counteract such discrimination.”

“…toxicity in their own outputs, and Si et al. (2022) demonstrated that the right prompts could reduce bias in language models on the BBQ benchmark (Parrish et al., 2021 Model-generated evaluations Finally, a range of recent works explore how LMs can assist in scalably generating diverse…”

“…toxicity in their own outputs, and Si et al. (2022) demonstrated that the right prompts could reduce bias in language models on the BBQ benchmark (Parrish et al., 2021 Model-generated evaluations Finally, a range of recent works explore how LMs can assist in scalably generating diverse…”

“The gold standard for investigating this question would be to analyze language model decisions in specific real-world applications, following the rich tradition of audit studies in the social sciences (Jowell & Prescott-Clarke, 1970; Gaddis, 2018).”

“…and institutions in complex ways, including via automation bias (Skitka et al., 1999; Goddard et al., 2012), meaning that while AI-aided decision-making can have positive effects (Keding & Meissner, 2021), placing humans in an advisory in-the-loop capacity is not by itself a sufficient guardrail.”

“The gold standard for investigating this question would be to analyze language model decisions in specific real-world applications, following the rich tradition of audit studies in the social sciences (Jowell & Prescott-Clarke, 1970; Gaddis, 2018).”

“A natural question raised by our work is under what circum-stances (and to what degree) positive discrimination should be corrected for, given arguments raised by proponents and critics of affirmative action policies and compensatory justice (Fullinwider, 2018; Dwork et al., 2012; Eidelson, 2015).”

“…real world, people might use a wider variety of input formats, including longer, supplementary documents such as resumes or medical records, or might engage in more interactive dialogues with models (Jakesch et al., 2023; Li et al., 2023a), each of which has the potential to affect our conclusions.”

“…other investigations have further studied algorithmic discrimination, including in other ad delivery settings (Datta et al., 2015; Ali et al., 2019; Imana et al., 2021), mortgage approvals (Martinez & Kirchner, 2021), resume screening (Dastin, 2018), recidivism prediction (Skeem & Lowenkamp,…”

“As a step in this direction, we focus on measuring the risk of widespread automated discrimination (Corbett-Davies et al., 2017; Kasy & Abebe, 2021; Starke et al., 2022; Creel & Hellman, 2022) via automated language model decisions.”

“External validity has been discussed in both machine learning contexts (Liao et al., 2021; 2022) as well as in audit studies (Lahey & Beasley, 2018).”

“…studied how discrimination can happen directly , based on demographic variables, as well as indirectly though proxies for those variables, such as zip code, extracurricular activities, or other features (VanderWeele & Robinson, 2014; Datta et al., 2017; Kilbertus et al., 2017; Adler et al., 2018).”

“…2015; Ali et al., 2019; Imana et al., 2021), mortgage approvals (Martinez & Kirchner, 2021), resume screening (Dastin, 2018), recidivism prediction (Skeem & Lowenkamp, 2016), hiring (Kirk et al., 2021; Bommasani et al., 2022a; Veldanda et al., 2023), and medical treatment (Obermeyer et al., 2019).”

“Most relevant to our work, Perez et al. (2023) use LMs to generate a wide array of evaluations for an LM in order to uncover concerning behaviors.”

“Fourth, choosing names in audit studies that are associated with different demographics can be challenging (Gaddis, 2017; Crabtree & Chykina, 2018); while we make a best effort in this study, more work is necessary, as well as an investigation into other sources of proxy discrimination.”

“…2015; Ali et al., 2019; Imana et al., 2021), mortgage approvals (Martinez & Kirchner, 2021), resume screening (Dastin, 2018), recidivism prediction (Skeem & Lowenkamp, 2016), hiring (Kirk et al., 2021; Bommasani et al., 2022a; Veldanda et al., 2023), and medical treatment (Obermeyer et al., 2019).”

“…may choose to limit the use of language models for such decisions, it remains important to proactively anticipate and mitigate such potential risks as early as possible, especially given that initial deployments in high-stakes settings have already begun (Wodzak, 2022; Singh et al., 2023).”

“Finally, discrimination and other fairness criteria are not the only important factors when deciding whether or not to deploy a model; models must be evaluated in naturalistic settings to ensure they perform satisfactorily at the tasks they are applied to (Raji et al., 2022; Sanchez et al., 2023).”

“…can be resource intensive to execute, making it challenging to conduct a broad study across the wide range of potential applications of language models, and are additionally limited by challenging privacy and ethical constraints of conducting such studies on real people (Crabtree & Dhima, 2022).”

“She enjoys being with other people and although Maria sometimes argues with her family, Maria has been getting along pretty well with her family.”

“LLMs are known to generate biased decisions and outputs (Perez et al., 2022; Tamkin et al., 2023; Cui et al., 2023).”

“A key challenge in tuning LLMs towards generating grounded responses is that the models may optimize for grounding at the expense of losing creativity and helpfulness.”

“This correlation means that simply enhancing the upstream general capabilities of models, such as by scaling parameters and increasing training data, often boosts performance across all benchmarks indiscriminately [41, 52].”

“External debiasing methods tackle biases in model predictions during down-stream tasks, using data-centered approaches[5] like GPT-3 encounter greater debiasing challenges due to size and complexity, often addressed through preference alignment [17] and prompt engineering techniques[21].”

“Consequently, this paper focuses on the use of LLMs in complex decision-making scenarios more likely to reflect real-world implementation and use of LLMs.”