Bias in computer systems
Open accessAbstract
From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent. Preexisting bias has its roots in social institutions, practices, and attitudes. Technical bias arises from technical constraints of considerations. Emergent bias arises in a context of use. Although others have pointed to bias inparticular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it. We conclude by suggesting that freedom from bias should by counted amoung the select set of criteria—including reliability, accuracy, and efficiency—according to which the quality of systems in use in society should be judged.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
6 of 22 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured
- Premisecitation
Section 50-(9) of the British Nationality Act, implemented in BNAP, specifies that a man is the father of only his legitimate children, whereas a woman is the mother of all her children, legitimate or not.
“As specified in Section 50-(9) of the British Nationality Act itself and imple-mented in BNAP,”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Sergot et al. 1986 source
- Premisecitation§ Bias in Computer Systems
The original NRMP Admissions Algorithm placed married couples at a disadvantage in achieving an optimal placement compared with their single peers.
“At this point, it was discovered that the original Admissions Algorithm placed married couples at a disadvantage in achieving an optimal placement as compared with their single peers [Roth 1984;1990].”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Roth 1984 source
Section IV, “Contemporary Issues” (page not supplied), says the couples algorithm used until 1983 could not accept preferences over pairs of positions and could produce outcomes unstable with respect to couples. That supports a disadvantage for couples, but the paper does not establish that couples achieved less optimal placements than single applicants. (LLM, conf=0.84)
Experimental — accuracy not yet measured - cites Roth 1990 source
- Premisereasoning
The MLSA gives processing attention to all processes beginning a new command as quickly as possible.
“In this manner the MLSA gives processing attention to all processes as quickly as possible that are beginning a new command.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Corbato et al. 1962 source
- Premisecitation
At the time of BNAP's initial implementation (1983), no mechanism was built into the program to incorporate relevant case law as it came into being.
“At the time of the BNAP's initial implementation (1983) no mechanism was built into the program to incorporate relevant case law as it came into being [Sergot et al. 1986].”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Sergot et al. 1986 source
- Findingnone given§ Bias in Computer Systems
The NRMP Admissions Algorithm does not take into account geographic distribution when determining a match, considering only the ranked preferences of hospitals and students.
“The Admissions Algorithm does not take into account geographic distribution when determining a match, considering only the ranked preferences of hospitals and students.”
Quote in the paperSubstantial claim
- Premisenone given
No comparable work examines bias in computer systems comprehensively and offers a framework for understanding and remedying it.
“Although others have pointed to bias in particular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it.”
- Also stated in: § INTRODUCTION · hedged
“Although others have pointed to bias in particular computer systems and have noted the general problem [Johnson and Mulvey 1993;Moor 1985], we know of no comparable work that focuses exclusively on this phenomenon and examines it comprehensively.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Johnson and Mulvey 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Moor 1985 source
- Also stated in: § INTRODUCTION · hedged
- Interpretationown data
From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent.
“From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent.”
- Also stated in: § INTRODUCTION · plain
“From an analysis of actual computer systems, we have developed three categories: preexisting bias, technical bias, and emergent bias.”
- Also stated in: § 2. FRAMEWORK FOR ANALYZING BIAS IN COMPUTER SYSTEMS · plain
“The framework that emerged from this methodology is comprised of three overarching categories-preexisting bias, technical bias, and emergent bias.”
Quote in the paperSubstantial claim- Rests on an assumption: The 17 computer systems examined are representative enough to generalize the framework to computer systems generally.
- Also stated in: § INTRODUCTION · plain
- Interpretationnone given
Emergent bias arises in a context of use.
“Emergent bias arises in a context of use.”
- Also stated in: § 2.3 Emergent Bias · plain
“emergent bias arises only in a context of use.”
Quote in the paperSubstantial claim - Also stated in: § 2.3 Emergent Bias · plain
- Interpretationreasoning
In that situation, individuals with long-running programs are systematically disadvantaged in overall service from the processor.
“Thus, in terms of overall service from the processor, these individuals with long-running programs are systematically disadvantaged.”
Quote in the paperSubstantial claim - Interpretationreasoning
The British Nationality Act is biased against the illegitimate descendants of British men.
“Thus the British Nationality Act is biased against the illegitimate descendants of British men.”
Quote in the paperSubstantial claim - Interpretationreasoning
User interfaces are likely to be particularly prone to emergent bias because interfaces by design seek to reflect the capacities, character, and habits of prospective users.
“User interfaces are likely to be particularly prone to emergent bias because interfaces by design seek to reflect the capacities, character, and habits of prospective users.”
Quote in the paperNot a substantial claim - Interpretationcitation
This study illustrates how preexisting biases, in the form of expectations about what software will appeal to each gender, coupled with the implicit assumption that the generic user of software is likely to be male, can influence design and give rise to bias in software.
“This study illustrates how preexisting biases, in the form of expectations about what software will appeal to each gender, coupled with the implicit assumption that the generic”
Quote in the paperSubstantial claim - Premisecitation
Some users with long-running programs uncovered the MLSA's strategy and developed a counterstrategy of manually stopping and restarting a long-running process to run it in small chunks, each placed in the top priority queue and executed speedily.
“some users with long-running programs uncovered the MLSA's strategy and developed a counterstrategy: by using a manual button to stop execution of a long-running process and a moment later restarting the process from where it left off, users effectively ran their long-running tasks in small chunks.Each small chunk, of course, was placed by MLSA into the top priority queue and executed speedily.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Corbato (electronic communication, December 17, 1993)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Huff and Cooper report that many of their subjects were aware of and had expressed open concern about how computers were frequently perceived to be in the male domain.
“Huff and Cooper report, furthermore, that many of their subjects were aware of and had expressed open concern about how computers were frequently perceived to be in the male domain.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Huff and Cooper 1987 source
- Interpretationreasoning
Nonexperts, particularly in developing countries, would be inclined to accept decisively the BNAP's response to their client's situation.
“Thus nonexperts-particularly in developing countrieswould be inclined to accept decisively the BNAP's response to their client's situation.”
Quote in the paperSubstantial claim- Rests on an assumption: BNAP's users are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.
- Rests on an assumption: Some of BNAP's queries require expert knowledge to answer.
- Rests on an assumption: Nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.
- Interpretationreasoning
Assuming the system is not saturated with too many users, short-running processes are speedily processed to completion under MLSA.
“Thus, assuming the system is not saturated with too many users, short-running processes are speedily processed to completion.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The MLSA system is not saturated with too many users.
- Premisecitation
The original design of the NRMP Admissions Algorithm did not deal well with the changing social conditions when significant numbers of dual-career couples participated in the match.
“Although the original design of the Admissions Algorithm did not deal well with the changing social conditions (when significant numbers of dual-career couples participated in the match)”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Roth 1990 source
- Interpretationreasoning§ Bias in Computer Systems
The NRMP embodies a preexisting bias.
“Thus, if the algorithm systematically and unfairly favors hospital over student preferences, it does so because of design specifications and organizational practices that predate the computer implementation.As such, the NRMP embodies a preexisting bias.”
Quote in the paperSubstantial claim- Rests on an assumption: In cases of conflict, there is no good rationale for favoring hospital programs at the expense of medical students.
- Interpretationreasoning
This BNAP bias is emergent because it depends on a shift in the social context concerning knowledge within the legal community.
“Again, we identify this bias as emergent because it depends on a shift in the social context, in this instance one concerning knowledge within the legal community.”
Quote in the paperSubstantial claim - Interpretationreasoning
In such ways, the BNAP comes to act as an instrument of bias against the nonexperts and their clients.
“In such ways, the BNAP comes to act as an instrument of bias against the nonexperts and their clients.”
Quote in the paperSubstantial claim- Rests on an assumption: BNAP's users are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.
- Rests on an assumption: Some of BNAP's queries require expert knowledge to answer.
- Rests on an assumption: Nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.
- Interpretationreasoning
Any expert system (independent of content area) that does not possess a reasonable mechanism for integrating new knowledge may be vulnerable to a similar form of emergent bias.
“Indeed, to varying degrees, any expert system (independent of content area) that does not possess a reasonable mechanism for integrating new knowledge may be vulnerable to a similar form of emergent bias.”
Quote in the paperSubstantial claim - Interpretationreasoning
This BNAP bias is emergent because it arises from a shift in the population using the system from the one apparently held in mind by the system's creators (from expert to nonexpert users).
“Because this bias arises from a shift in the population using the system from the one apparently held in mind by the system's creators (from expert to nonexpert users) we identify this bias as emergent.”
Quote in the paperSubstantial claim- Rests on an assumption: BNAP was designed in a research environment among people with sophisticated knowledge of immigration law.
- Rests on an assumption: BNAP's users are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.
- Interpretationreasoning
In a timeshare system with many users, most using small short tasks and one or two consistently using long-running programs, the long-running programs necessarily end up with lower priority than the short-running tasks under MLSA.
“According to the MLSA, the long-running programs of these individuals will necessarily end up with a lower priority than the short-running tasks of the other users.”
Quote in the paperSubstantial claim - Interpretationreasoning
To the extent BNAP accurately represents the British Nationality Act, the program embodies preexisting bias.
“Accordingly, to the extent the BNAP accurately represents the British Nationality Act, the program embodies preexisting bias.”
Quote in the paperSubstantial claim - Interpretationreasoning
In principle, a long-running process could wait all day to finish under MLSA.
“At the same time, however, in principle a longrunning process could wait all day to finish.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Bias in Computer Systems
The bias against couples in the NRMP primarily emerged when a shift occurred in social conditions, namely, when husband and wife medical students increasingly participated in the match process.
“The point here is that the bias-in this case emergent bias-against couples primarily emerged when a shift occurred in the social conditions, namely, when husband and wife medical students increasingly participated in the match process.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Bias in Computer Systems
Centralized computing systems with widespread use can hold users hostage to whatever biases are embedded within the system.
“In this manner, centralized computing systems with widespread use can hold users hostage to whatever biases are embedded within the system.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 18
- May combine two claims. 15
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationThe BNAP authors state their goal is to translate the British Nationality Act so that the consequences of the act can be determined mechanistically, and they place their work in the context of other legal expert systems designed for use with real users.
“For in the same text the authors also state (as noted above) that their goal is to translate the British Nationality Act so that”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Sergot et al. 1986 source
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).
PremisecitationThe NRMP Admissions Algorithm systematically favors hospital programs over medical students in cases of conflict.
“One charges that the Admissions Algorithm systematically favors hospital programs over medical students in cases of conflict [Graettinger and Peranson 1981b;Roth 1984;Sudarshan and Zisook 1981;Williams et al. 1981].”
Quote in the paperNot a substantial claim1 of 3 sources checkedProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenTechnical bias arises from technical constraints or considerations.
“Technical bias arises from technical constraints or considerations.”
- Also stated in: § 2.1 Preexisting Bias · plain
“Technical bias arises from technical constraints or technical considerations.”
Quote in the paperNot a substantial claim- Also stated in: § 2.1 Preexisting Bias · plain
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenPreexisting bias has its roots in social institutions, practices, and attitudes.
“Preexisting bias has its roots in social institutions, practices, and attitudes.”
- Also stated in: § INTRODUCTION · plain
“Preexisting bias has its roots in social institutions, practices, and attitudes.”
Quote in the paperNot a substantial claim- Also stated in: § INTRODUCTION · plain
May combine two claims.
Interpretationreasoning§ Bias in Computer SystemsThe NRMP is not biased against rural hospitals in favor of urban ones because the discrimination reflects the preferences of match participants and has a reasonable basis.
“In this case, we ourselves would shy away from viewing this as a bias in the system because we think this discrimination can be defended as having a reasonable basis.Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
Quote in the paperSubstantial claim- Rests on an assumption: The discrimination against rural hospitals reflects the preferences of match participants.
- Rests on an assumption: The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningEmergent bias poses a potential problem for any legal expert system, especially in a society where legal systems depend on evolving case law.
“Emergent bias poses a potential problem for any legal expert system, especially in a society where legal systems depend on evolving case law.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningIn this case the biased designs were unintended and reflected gender-biases deeply rooted in the culture at large.
“We thus infer that in this case the biased designs were unintended and, instead, reflected gender-biases deeply rooted in the culture at large.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningBiases in computer systems can be difficult to identify and remedy because computer systems are inexpensive to disseminate, can become standards, can hide biases in complex code, and offer no equivalent means for appeal.
“As Sabre and Apollo illustrate, biases in computer systems can be difficult to identify let alone remedy because of the way the technology engages and extenuates them.Computer systems, for instance, are comparatively inexpensive to disseminate, and thus, once developed, a biased system has the potential for widespread impact.If the system becomes a standard in the field, the bias becomes pervasive”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationHuff and Cooper report that along a number of dimensions the designs proposed by subjects in the gender-unspecified group closely resembled the designs proposed by subjects who designed for boys and were significantly different from the designs proposed by subjects who designed for girls.
“Huff and Cooper report that along a number of dimensions the designs proposed by subjects in the gender-unspecified group closely resembled the designs proposed by subjects who designed for boys and were significantly different from the designs proposed by subjects who designed for girls.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Huff and Cooper 1987 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisereasoningOne source of bias in Sabre and Apollo lies in their algorithms for controlling search and display functions, which give preference to on-line flights.
“One source of this alleged bias lies in Sabre's and Apollo's algorithms for controlling search and display functions.In the algorithms, preference is given to”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningA shift in context of use may well create difficulties for a new set of users.
“Thus, a shift in context of use may well create difficulties for a new set of users.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationSabre and Apollo systematically downgrade and are biased against international carriers who fly few, if any, internal U.S. flights, and against internal carriers who do not fly international flights.
“Thus, the computer systems systematically downgrade and, hence, are biased against international carriers who fly few, if any, internal U.S. flights, and against internal carriers who do not fly international flights [Fotos 1988;Ott 1988].”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Fotos 1988
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Ott 1988
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenEmergent bias typically emerges some time after a design is completed, as a result of changing societal knowledge, population, or cultural values.
“This bias typically emerges some time after a design is completed, as a result of changing societal knowledge, population, or cultural values.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThe interface design compounds the bias in reservation systems because lists of ranked flight options are displayed screen by screen, and 90% of tickets booked by travel agents are booked by the first screen display.
“One is that the interface design compounds the bias in the reservation systems.Lists of ranked flight options are displayed screen by screen.Each screen displays only two to five options.The advantage to a carrier of having its flights shown on the first screen is enormous since 90% of the tickets booked by travel agents are booked by the first screen display [Taib 1990].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Taib 1990
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningTechnical bias with the MLSA is embedded in the program, but the bias is a phenomenon of the system in use, in a context wherein users with short and long jobs outpace the system's capacity.
“Or consider again the case of technical bias with the MLSA (which favors users with short jobs over long jobs).While technical bias is embedded in the program, the bias is a phenomenon of the system in use, in a context wherein users with short and long jobs outpace the system's capacity.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationIn the Huff and Cooper study, subjects were asked to propose designs for software to teach seventh graders the correct use of commas; one group designed for seventh-grade boys, a second for seventh-grade girls, and a third for seventh-graders with gender unspecified.
“In their study, subjects were asked to propose designs for software to teach seventh graders the correct use of commas.One group of subjects was asked to design the software for seventh-grade boys, the second group to design for seventh-grade girls, and the third group to design for seventh-graders, gender unspecified.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Huff and Cooper 1987 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone givenPreexisting bias can enter a system either through explicit and conscious efforts of individuals or institutions, or implicitly and unconsciously, even in spite of the best of intentions.
“This type of bias can enter a system either through the explicit and conscious efforts of individuals or institutions, or implicitly and unconsciously, even in spite of the best of intentions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone givenSources of technical bias include limitations of computer tools such as hardware, software, and peripherals; the process of ascribing social meaning to algorithms developed out of context; imperfections in pseudorandom number generation; and the attempt to make human constructs amenable to computers.
“Sources of technical bias can be found in several aspects of the design process, including limitations of computer tools such as hardware, software, and peripherals; the process of ascribing social meaning to algorithms developed out of context; imperfections in pseudorandom number generation; and the attempt to make human constructs amenable to computers, when we quantify the qualitative, discret”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThe NRMP algorithm's preference for hospital programs in cases of conflict duplicates what happens in an actual admissions process without a computerized matching program.
“The constraint inherent in the NRMP algorithm, in which preference is given to hospital choices when conflicts in rankings occur, duplicates what happens in an actual admissions process without a computerized matching program”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Graettinger and Peranson 1981b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationnone given§ Bias in Computer SystemsWhatever ranking algorithm is used, if it systematically places certain airlines' flights on initial screens and other airlines' flights on later screens, the system will exhibit technical bias.
“Whatever ranking algorithm is used, if it systematically places certain airlines' flights on initial screens and other airlines' flights on later screens, the system will exhibit technical bias.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenPreexisting biases can reflect the personal biases of individuals who have significant input into the design of the system, such as the client or system designer.
“They can also reflect the personal biases of individuals who have significant input into the design of the system, such as the client or system designer.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThose responsible for maintaining the NRMP responded conscientiously to this societal change and modified the system's algorithm to place couples more fairly.
“those responsible for maintaining the system responded conscientiously to this societal change and modified the system's algorithm to place couples more fairly [Roth 1990].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Roth 1990 source
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone givenPreexisting biases may originate in society at large, in subcultures, and in formal or informal, private or public organizations and institutions.
“Preexisting biases may originate in society at large, in subcultures, and in formal or informal, private or public organizations and institutions.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationcitationShould the accumulation of case law lead to changes in the way the Act is interpreted, such as by granting new subgroups British citizenship, BNAP would systematically misinform members of this subgroup regarding their status as British citizens.
“Should the accumulation of case law lead to changes in the way the Act is interpreted-say, by granting new subgroups British citizenship-BNAP would systematically misinform members of this subgroup regarding their status as British citizens [Leith 1986].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Leith 1986 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataIn total, 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law were examined.
“In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductown dataThe framework was derived by examining actual computer systems for bias; instances were identified and characterized according to source, generalized to abstract categories, and refined by application to other instances.
“We derived our framework by examining actual computer systems for bias.Instances of bias were identified and characterized according to their source, and then the characterizations were generalized to more abstract categories.These categories were further refined by their application to other instances of bias in the same or additional computer systems.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningThe presence of bias is not so much a feature inherent in the system independent of the context of use, but an aspect of a system in use.
“although we have talked about bias in computer systems, the presence of bias is not so much a feature inherent in the system independent of the context of use, but an aspect of a system in use.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
In cases of conflict, there is no good rationale for favoring hospital programs at the expense of medical students.
“We are inclined to answer yes because in cases of conflict there does not appear to be a good rationale for favoring hospital programs at the expense of students.”
The paper relies on this without support.
The discrimination against rural hospitals reflects the preferences of match participants.
“Namely, the discrimination reflects the preferences of match participants, and it is reasonable in our view for employment decisions to be determined largely by the choices of employers and employees.”
The paper relies on this without support.
The 17 computer systems examined are representative enough to generalize the framework to computer systems generally.
“In total, we examined 17 computer systems from diverse fields including banking, commerce, computer science, education, medicine, and law.”
The paper relies on this without support.
The illustrative examples in Table I portray plausible cases of bias.
“The illustrative examples portray plausible cases of bias.”
The paper relies on this without support.
The best teaching hospitals tend to be in urban areas, and urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.
“Because the best teaching hospitals tend to be in urban areas, the urban areas tend to fill their positions far more effectively, and with better students, than rural hospitals.”
The authors state this as an assumption.
The MLSA system is not saturated with too many users.
“assuming the system is not saturated with too many users”
The paper relies on this without support.
BNAP was designed in a research environment among people with sophisticated knowledge of immigration law.
“First, the system was designed in a research environment, among people with sophisticated knowledge of immigration law.”
The paper relies on this without support.
BNAP's users are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.
“Its users, however, are likely to be at best paralegal or immigration counselors in Britain, if not lay persons in foreign countries with only limited access to British legal expertise.”
The paper relies on this without support.
Some of BNAP's queries require expert knowledge to answer.
“Some of the program's queries, for example, require expert knowledge to answer.”
The paper relies on this without support.
Nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.
“More generally, nonexperts advising British citizen hopefuls are not alerted to alternative legal frameworks that complement the British Nationality Act.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HELM's multi-metric approach makes potential trade-offs explicit and helps prevent nonaccuracy desiderata from being treated as second-class.
“The multi-metric approach (Figure S1) makes explicit potential trade-offs and helps ensure the nonaccuracy desiderata are not treated as second-class citizens to accuracy (see Ref. 13).”
From Holistic Evaluation of Language Models (2023) · cites this paper as [13]
The Conclusion argues that freedom from bias should be a criterion for judging computer systems alongside accuracy, and Section 4 describes weighing ease of access against equity of access. These passages support taking nonaccuracy concerns and trade-offs seriously, but the paper does not assess HELM’s multi-metric approach or show that it makes those trade-offs explicit. (LLM, conf=0.82)
Experimental — accuracy not yet measuredResearch has repeatedly shown that ADM systems can encode biases into decisions.
“research has repeatedly shown that these systems can in fact encode biases into decisions 124 .”
From Administrative law and the machines of government: judicial review of automated public-sector decision-making (2019) · cites this paper as B Friedman and H Nissenbaum 'Bias in Computer Systems' (1996)
The risk of emergent bias, by which a tool produces biased outputs through its own functioning, is documented in the literature.
“The risk of”
From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [25]
Bias and opacity in computer systems have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.
“Bias or opacity are far from newly observed characteristics of computer systems generally [26, 11, 24], but these issues have been exacerbated by the rapidly expanding numbers of data-driven information systems entering decision-making processes.”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [26]
AI-driven decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic.
“These decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic (Friedman and Nissenbaum 1996).”
From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Friedman and Nissenbaum 1996
The apparent problem to which data science is applied determines whether it serves morally defensible ends.
“On this account, the apparent problem to which data science is a solution determines whether it happens to serve morally defensible ends.”
From Problem Formulation and Fairness (2019) · cites this paper as [17]
The XING outcome systematically disadvantages individuals of one gender by preferring the other at the top-k positions.
“Corresponding with [14] this outcome systematically disadvantages individuals of one gender by preferring the other at the topk positions.”
From FA*IR: A Fair Top-k Ranking Algorithm (2017) · cites this paper as [14]
The underlying causes of harms from algorithmic bias are complex; technical factors such as biased data and design choices play an important role, and biases can also arise from the context of use, for example, how human users interpret system outputs or from a mismatch between the capabilities and values assumed in the design of the system and those of its actual users.
“While technical factors, such as biased data and design choices, play an important role, biases can also arise from the context of use, for example, how human users interpret system outputs or from a mismatch between the capabilities and values assumed in the design of the system and those of its actual users (Danks & London, 2017; Friedman & Nissenbaum, 1996).”
From Digital ageism: challenges and opportunities in artificial intelligence for older adults (2022) · cites this paper as Friedman & Nissenbaum, 1996
Machine learning systems are known to have issues relating to data protection, privacy, and other human rights issues.
“and to data protection, privacy, and other human rights issues 11”
From Administrative Law and the Machines of Government: Judicial Review of Automated Public-Sector Decision-Making (2018) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Table II gives conflicting years for its source
MinorThe table caption dates the Williams et al. source to 1991, while the paper's reference entry for that work dates it to 1981.
“Table II.The Simplest Case of Rank-Order Lists in which the Desires of Students and Programs are in Conflict (Reprinted from Williams et al. [1991], by permission of the New England Journal of Medicine)”
“18. WILLIAMS, K. J., WERTH, V. P., AND WOLFF, J. A. 1981. An analysis of the resident match.”
Cites
The evolution of the labor market for medical interns and residents: A case study in game theory
Roth, A. E. (1984) · Journal of Political Economy · doi:10.1086/261272 · Cited by 1,219 · see citing works
MOOR, J. H. (1985) · Metaphilosophy · doi:10.1111/j.1467-9973.1985.tb00173.x · Cited by 863 · see citing works
The British Nationality Act as a logic program
Sergot, M. J., Sadri, F., Kowalski, R. A., Kriwaczek, F., Hammond, P., & Cory, H. T. (1986) · Communications of the ACM · doi:10.1145/5689.5920 · Cited by 564 · see citing works
Moor, J. (1985) · The Ethics of Information Technologies · doi:10.4324/9781315259697-7 · Cited by 456
An experimental time-sharing system
Corbató, F. J., Merwin-Daggett, M., & Daley, R. C. (1962) · Proceedings of the May 1-3, 1962, spring joint computer conference on - AIEE-IRE '62 (Spring) · doi:10.1145/1460833.1460871 · Cited by 234 · see citing works
New physicians: A natural experiment in market organization
Roth, A. E. (1990) · Science · doi:10.1126/science.2274783 · Cited by 141 · see citing works
Huff, C., & Cooper, J. (1987) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1987.tb00328.x · Cited by 85 · see citing works
Fundamental Errors in Legal Logic Programming
Leith, P. (1986) · The Computer Journal · doi:10.1093/comjnl/29.6.545 · Cited by 51 · see citing works
Moor, J. H. (2020) · Routledge eBooks · doi:10.4324/9781003075011-1 · Cited by 21 · see citing works
National resident matching program
(1981) · New England Journal of Medicine · doi:10.1056/nejm198108273050923 · Cited by 10 · see citing works
Graettinger, J. S., & Peranson, E. (1981) · New England Journal of Medicine · doi:10.1056/nejm198105073041909 · Cited by 10 · see citing works
An analysis of the resident match
Williams, K. J., Werth, V. P., & Wolff, J. A. (1981) · New England Journal of Medicine · doi:10.1056/nejm198105073041910 · Cited by 9 · see citing works
(1990) · Aviation week & space technology · Cited by 6 · see citing works
Results of the National Resident Matching Program for 1982
Graettinger, J. S. (1982) · Academic Medicine · doi:10.1097/00001888-198205000-00018
Flagged first, then by citations
Research works printed but not linked: 4
- FISHLOCK , T. 1981. Delhi press detect racism in Nationality Bill. The London Times ( Jan. 20 ) . FISHLOCK, T. 1981. Delhi press detect racism in Nationality Bill. The London Times (Jan. 20).
- GAO . 1992. Patriot Missile defense: Software problem led to system failure at Dhahran , Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C. GAO. 1992. Patriot Missile defense: Software problem led to system failure at Dhahran, Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C.
- JOHNSON , D.G. AND MULVEY , J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11 , Dept. of Civil Engineering and Operations Research , Princeton Univ ., Princeton, N.J. JOHNSON,D.G.AND MULVEY, J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11, Dept. of Civil Engineering and Operations Research, Princeton Univ., Princeton, N.J.
- OTT , J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. ( July 18 ) . OTT, J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. (July 18).
Cited by
Sorted by citations
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 3,432 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Value Sensitive Design and Information Systems
Friedman, B., Kahn, P. H., & Borning, A. H. (2008) · Wiley eBooks · doi:10.1002/9780470281819.ch4 · Cited by 1,149 · see citing works
Self-citationFriedman, B. (1996) · Interactions · doi:10.1145/242485.242493 · Cited by 966 · see citing works
Self-citationBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · see citing works
“Friedman and Nissenbaum (1996) show that in some cases, system bias reflects biases in society; these are pre-existing biases with roots in social institutions, practices and attitudes.”
BackgroundMethodSelf-citationBaker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993 · see citing works
“Landmark early work by Friedman and Nissenbaum (1996) set a precedent for this sequential framing of bias, categorizing biases based on the context where they arise, whether from preexisting social biases, technical limitations, or use of the models by real users.”
BackgroundImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
“The design, prototyping, and maintenance of machine learning systems raises many unique challenges [27, 63, 78, 87] not commonly faced with other kinds of intelligent systems or computing systems more broadly [34, 66, 67].”
BackgroundShneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768 · see citing works
“An early review by Friedman and Nissenbaum [39] described three kind of bias, which remain as a helpful guide: preexisting biases based on social practices and attitudes, technical bias based on design constraints in hardware and software, and emergent bias that arises from changing the use context.”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
“Broadly discussing bias in computer systems, Friedman and Nissenbaum (1996) characterize it according to its source such as societal, technical, or usage related.”
BackgroundEthical Implications and Accountability of Algorithms
Martin, K. (2019) · Journal of Business Ethics · doi:10.1007/s10551-018-3921-3 · Cited by 757 · see citing works
“7 These technologies survive to have biases that are valueladen (Friedman and Nissenbaum 1996; Johnson 2004) or have politics (Winner 1980).”
BackgroundAlgorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733 · see citing works
“Researchers have been aware of and have critiqued the biases embedded in computational systems for many years (Friedman and Nissenbaum 1996).”
BackgroundValue Sensitive Design and Information Systems
Friedman, B., Kahn, P. H., Borning, A., & Huldtgren, A. (2013) · Philosophy of Engineering and Technology · doi:10.1007/978-94-007-7844-3_4 · Cited by 678 · see citing works
“bias (Friedman and Nissenbaum, 1996), universal usability (Shneiderman, 1999, 2000;Thomas, 1997),autonomy (Suchman,1994;Winograd, 1994), informedconsent (Millett et al.”
BackgroundSelf-citationBias in algorithmic filtering and personalization
Bozdag, E. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9321-6 · Cited by 649 · see citing works
“Just as any computer system, they can unfairly discriminate against certain individuals or groups of individuals in favor of others (Friedman and Nissenbaum 1996).”
BackgroundInfluentialBias and Debias in Recommender System: A Survey and Future Directions
Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3564284 · Cited by 741 · see citing works
“uired traits” [50], and the unfairness can be defined as follow: Unfairness. The system systematically and unfairly discriminates against certain individuals or groups of individuals in favor others [51]. Unfairness issue has been an obstacle to making recommender systems more entrenched within our society. In particul…”
BackgroundKuziemski, M., & Misuraca, G. (2020) · Telecommunications Policy · doi:10.1016/j.telpol.2020.101976 · Cited by 610 · see citing works
Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535 · see citing works
“In these cases, biases are said to ‘ sneak in ’ (University of Bath, 2017), ‘ whether on purpose or by accident ’ (Barocas & Selbst, 2016, p. 674), or in ways that only emerge over time (Friedman & Nissenbaum, 1996).”
Background“Participant” Perceptions of Twitter Research Ethics
Fiesler, C. L., & Proferes, N. (2018) · Social Media + Society · doi:10.1177/2056305118763366 · Cited by 545 · see citing works
“However, both of these beliefs may be misguided, as re-identification is possible in large data sets (Zimmer, 2010a), and computer algorithms can be biased in both design and application (Friedman & Nissenbaum, 1996).”
BackgroundA Framework of Potential Sources of Harm Throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 576 · see citing works
Algorithmic Transparency in the News Media
Diakopoulos, N., & Koliska, M. (2017) · Digital Journalism · doi:10.1080/21670811.2016.1208053 · Cited by 515 · see citing works
“One of the main concerns regarding computational systems is the existence of embedded biases, which researchers have long criticized (Friedman and Nissenbaum 1996).”
BackgroundEthical and Bias Considerations in Artificial Intelligence/Machine Learning
Hanna, M. G., Pantanowitz, L., Jackson, B., Palmer, O., Visweswaran, S., Pantanowitz, J., Deebajah, M., & Rashidi, H. H. (2025) · Modern Pathology · doi:10.1016/j.modpat.2024.100686 · Cited by 535 · see citing works
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · see citing works
“Both algorithms and interfaces, however, can be biased in how they represent the world [9,34].”
BackgroundKöchling, A., & Wehner, M. C. (2020) · Business Research · doi:10.1007/s40685-020-00134-w · Cited by 511 · see citing works
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · see citing works
“Corresponding with [14] this outcome systematically disadvantages individuals of one gender by preferring the other at the top-k positions.”
BackgroundHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · see citing works
Algorithmic Accountability and Public Reason
Binns, R. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0263-5 · Cited by 457 · see citing works
“In the case of knowledge-based systems, the knowledge that is fed into the system, and assumptions that are involved in modelling it, may reflect biases of the system designers and data collection process (Wiener, 1960; Weizenbaum, 1972; Dutton & Kraemer, 1980; Friedman & Nissenbaum, 1996).”
BackgroundMadaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · see citing works
“Although HCI researchers have developed methods for designing systems with human values in mind (e.g., [38,39,91,93]), they have not yet become best practices or even propagated into professional training for UX researchers.”
BackgroundMethodHow to Design AI for Social Good: Seven Essential Factors
Floridi, L., Cowls, J., King, T. C., & Taddeo, M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00213-5 · Cited by 430 · see citing works
“These decisions may be based on factors of ethical importance (e.g., ethnic, gender, or religious grounds) and irrelevant to the decision-making at hand, or they may be relevant but legally protected as a nondiscriminatory characteristic (Friedman and Nissenbaum 1996).”
BackgroundTransparency in Algorithmic and Human Decision-Making: Is There a Double Standard?
Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019) · Philosophy & Technology · doi:10.1007/s13347-018-0330-6 · Cited by 420 · see citing works
“For example, what Friedman and Nissenbaum (1996) call B technical ^ bias arises from the inherent constraints imposed by the technology itself.”
BackgroundGreene, D., Hoffmann, A. L., & Stark, L. (2019) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2019.258 · Cited by 386 · see citing works
“longstanding work in computing ethics [11], [36], [37].”
BackgroundA Survey on the Fairness of Recommender Systems
Wang, Y., Ma, W., Zhang, M., Liu, Y., & Ma, S. (2022) · ACM Transactions on Information Systems · doi:10.1145/3547333 · Cited by 377 · see citing works
“Although fairness has been studied in computer science for decades [27], and there is a lot of related work in machine learning [100, 101], fairness in recommendation has its unique problems.”
BackgroundFairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330691 · Cited by 382 · see citing works
“, see [19, 22, 43] and the references therein).”
Value Sensitive Design: Theory and Methods
Friedman, B., Kahn, P. H., & Borning, A. H. (2002) · Cited by 375 · see citing works
Self-citationEmbodying Values in Technology: Theory and Practice
Flanagan, M., Howe, D. C., & Nissenbaum, H. F. (2008) · Cambridge University Press eBooks · doi:10.1017/cbo9780511498725.017 · Cited by 335 · see citing works
Self-citationRakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329 · see citing works
“While the academic discussion of algorithmic bias has an over 20-year long history [12], we have now reached a transitional phase in which this debate has taken a practical turn.”
BackgroundOut of the laboratory and into the classroom: the future of artificial intelligence in education
Schiff, D. (2021) · AI & SOCIETY · doi:10.1007/s00146-020-01033-8 · Cited by 334 · see citing works
“In contrast, an emerging fear in artificial intelligence is that of algorithmic bias, which may favor or disfavor certain subgroups by gender, sex, race, ethnicity, socioeconomic grouping, and other more subtle factors (Friedman and Nissenbaum 1996; Howard and Borenstein 2017; Bolukbasi et al. 2016; Buolamwini and Geb…”
BackgroundValues as lived experience: evolving value sensitive design in support of value discovery
Dantec, C. A. L., Poole, E., & Wyche, S. (2009) · International Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518875 · Cited by 386 · see citing works
“During the past decade, the HCI community has become increasingly interested in accounting for human values in the design of computer systems [3, 16, 18, 19, 23, 40].”
BackgroundOn the apparent conflict between individual and group fairness
Binns, R. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372864 · Cited by 312 · see citing works
“Research on bias, fairness and discrimination in socio-technical systems has a long history which significantly predates specific work on fairness in machine learning (see e.g. [23, 24]).”
BackgroundKopalle, P. K., Gangwar, M., Kaplan, A., Ramachandran, D., Reinartz, W., & Rindfleisch, A. (2021) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2021.11.002 · Cited by 323 · see citing works
Digital ageism: challenges and opportunities in artificial intelligence for older adults
Chu, C. H., Nyrup, R., Leslie, K., Shi, J., Bianchi, A., Lyn, A., McNicholl, M., Khan, S., Rahimi, S., & Grenier, A. (2022) · The Gerontologist · doi:10.1093/geront/gnab167 · Cited by 290 · see citing works
“A system discriminates unfairly if it denies an opportunity or a good or if it assigns an undesirable outcome to an individual or group of individuals on grounds that are unreasonable or inappropriate" (Friedman & Nissenbaum, 1996, p. 332).”
DeVito, M. A. (2016) · Digital Journalism · doi:10.1080/21670811.2016.1178592 · Cited by 295 · see citing works
Human Values, Ethics, and Design
Batya Friedman and Peter H. Kahn, Jr. (2002) · CRC Press eBooks · doi:10.1201/9781410606723-48 · Cited by 348 · see citing works
Should we welcome robot teachers?
SHARKEY, A. J. C. (2016) · Ethics and Information Technology · doi:10.1007/s10676-016-9387-z · Cited by 250 · see citing works
“Forms of technological bias were already being discussed nearly two decades ago (Friedman and Nissenbaum 1996).”
BackgroundSociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279 · see citing works
“Computing research has traced how marginalized communities — referring to communities that face structural forms of social exclusion [130] — disproportionately experience sociotechnical harms from algorithmic systems [90, 96].”
BackgroundPerceived risk and the consumer buying process: internet airline reservations
Cunningham, L. F., Gerlach, J. H., Harper, M. D., & Young, C. E. (2005) · International Journal of Service Industry Management · doi:10.1108/09564230510614004 · Cited by 246 · see citing works
“Initial research focused on bias in reservation systems (Friedman and Nussenbaum, 1996) that favored certain carriers and the effectiveness of the electronic marketplaces to increase price competition and reduce seller monopoly power (Bakos, 1997).”
BackgroundMapping value sensitive design onto AI for social good principles
Umbrello, S., & van de Poel, I. (2021) · AI and Ethics · doi:10.1007/s43681-021-00038-3 · Cited by 251 · see citing works
Values Levers: Building Ethics into Design
Shilton, K. (2013) · Science, Technology, & Human Values · doi:10.1177/0162243912436985 · Cited by 240 · see citing works
Algorithmic bias in machine learning-based marketing models
Akter, S., Dwivedi, Y. K., Sajib, S., Biswas, K., Bandara, R. J., & Michael, K. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.083 · Cited by 242 · see citing works
Problem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283 · see citing works
“into the system during its development and implementation [17].”
BackgroundA Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 190 · see citing works
van Giffen, B., Herhausen, D., & Fahse, T. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.076 · Cited by 230 · see citing works
“First, a monitoring plan accounts for changes in the algorithm when the context evolves (Corbett-Davies & Goel, 2018; Friedman & Nissenbaum, 1996).”
BackgroundMethodFairness in recommender systems: research landscape and future directions
Deldjoo, Y., Jannach, D., Bellogín, A., Difonzo, A., & Zanzonelli, D. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09364-z · Cited by 258 · see citing works
“In Friedman and Nissenbaum (1996), for example, bias is used to “ refer to computer systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others ”.”
Passi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225 · see citing works
“professional values, but also from the emergent sites of algorithmic re-use as opposed to their contexts of development [29,43,51].”
BackgroundHouston, L., Jackson, S. J., Rosner, D. K., Ahmed, S. I., Young, M., & Kang, L. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858470 · Cited by 235 · see citing works
“In the early 1990s, Friedman, Nissenbaum and others opened calls for value-based approaches that were oriented to producing technological systems that were “good” to use in a moral as well as a functional sense [18].”
BackgroundRoles for computing in social change
Abebe, R., Barocas, S., Kleinberg, J. M., Levy, K. E. C., Raghavan, M., & Robinson, D. G. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372871 · Cited by 319 · see citing works
“Some values may be harder to incorporate into formal models than others, perhaps because they are difficult to quantify [43].”
BackgroundA clarification of the nuances in the fairness metrics landscape
Castelnovo, A., Crupi, R., Greco, G., Regoli, D., Penco, I., & Cosentini, A. (2021) · Scientific Reports · doi:10.1038/s41598-022-07939-1 · Cited by 283 · see citing works
“It is out of the scope of this manuscript to list and discuss all the possible shades of biases that can hide in the data (Friedman and Nissenbaum, 1996; Crawford, 2013; Mehrabi et al., 2021; BaezaYates, 2018; Hardt, 2014), we want nevertheless to say something about the main cases that can be…”
BackgroundMachine Learning for Chemical Reactivity: The Importance of Failed Experiments
Strieth‐Kalthoff, F., Sandfort, F., Kühnemund, M., Schäfer, F. R., Kuchen, H., & Glorius, F. (2022) · Angewandte Chemie International Edition · doi:10.1002/anie.202204647 · Cited by 219 · see citing works
PI3K and MEK inhibitor combinations: examining the evidence in selected tumor types
Britten, C. D. (2013) · Cancer Chemotherapy and Pharmacology · doi:10.1007/s00280-013-2121-1 · Cited by 216 · see citing works
Roselli, D., Matthews, J., & Talagala, N. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317590 · Cited by 237 · see citing works
“Other researchers have proposed taxonomies of bias that serve as an excellent starting point towards managing that bias [38, 39, 40] Cramer et.”
BackgroundDemographic Bias in Biometrics: A Survey on an Emerging Challenge
Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., & Busch, C. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.2992344 · Cited by 263 · see citing works
“A good discussion of the topic of bias was provided by Danks and London [19], as well as Friedman and Nissenbaum [20], both of which explored various sources and types of bias in the context of computer systems.”
Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359283 · Cited by 217 · see citing works
“Fairness is an important value in governing algorithms as algorithms can perpetuate unfair treatment of different populations or stakeholders [27, 35, 85].”
BackgroundWillson, M. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1200645 · Cited by 205 · see citing works
Motivations of contributors to Wikipedia
Kuznetsov, S. (2006) · ACM SIGCAS Computers and Society · doi:10.1145/1215942.1215943 · Cited by 205 · see citing works
Zajko, M. (2022) · Sociology Compass · doi:10.1111/soc4.12962 · Cited by 237 · see citing works
Bias of AI-generated content: an examination of news produced by large language models
Fang, X., Che, S., Mao, M., Zhang, H., Zhao, M., & Zhao, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-55686-2 · Cited by 194 · see citing works
“At the sentence level, the bias of AIGC is evaluated as the difference between sentences related to each investigated population group in AIGC and their counterparts in the news articles produced by the selected news agencies, in terms of their expressed sentiments and toxicities.”
BackgroundExclusion by design: intersections of social, digital and data exclusion
Park, S., & Humphry, J. (2019) · Information Communication & Society · doi:10.1080/1369118x.2019.1606266 · Cited by 195 · see citing works
“Our approach adopts the view established in Science and Technology Studies (STS) that technologies are embedded within larger social systems and processes, inscribed with the rules, values and interests of typically dominant groups (Bijker & Law, 1992; Friedman & Nissenbaum, 1996).”
MethodDeep reinforcement learning in recommender systems: A survey and new perspectives
Chen, X., Yao, L., McAuley, J., Zhou, G., & Wang, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110335 · Cited by 261 · see citing works
Democratizing Algorithmic Fairness
Wong, P.-H. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00355-w · Cited by 182 · see citing works
“” (Friedman & Nissenbaum 1996, 332), and the recognition of bias in computer systems has inspired numerous approaches to detect, scrutinize and prevent biases in computer systems.”
BackgroundMiller, J. K., Friedman, B., & Jancke, G. (2007) · doi:10.1145/1316624.1316668 · Cited by 181 · see citing works
“, informed consent, fairness, security, trust, privacy) with respect to technology [5, 6, 13, 16].”
BackgroundSelf-citationTechnologies of memory: Key issues and critical perspectives
House, N. V., & Churchill, E. F. (2008) · Memory Studies · doi:10.1177/1750698008093795 · Cited by 180 · see citing works
“Like other work (Friedman and Nissenbaum, 1996) showing how power relations and values are instantiated in technologies, we see that explicit and tacit models of social and personal memory are ‘baked into ‘ the design of these technologies.”
BackgroundThe combine will tell the truth: On precision agriculture and algorithmic rationality
Miles, C. (2019) · Big Data & Society · doi:10.1177/2053951719849444 · Cited by 173 · see citing works
Gender Bias in Machine Translation
Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., & Turchi, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00401 · Cited by 189 · see citing works
“Hereby, drawing on the human-centered approach of Value Sensitive Design (Friedman and Hendry, 2019), we consider as biased an MT model that systematically and unfairly discriminates against certain individuals or groups in favour of others (Friedman and Nissenbaum, 1996).”
MethodGender bias perpetuation and mitigation in AI technologies: challenges and opportunities
O’Connor, S., & Liu, H. K. (2023) · AI & Society · doi:10.1007/s00146-023-01675-4 · Cited by 216 · see citing works
Designing Fair Ranking Schemes
Asudeh, A., Jagadish, H. V., Stoyanovich, J., & Das, G. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3300079 · Cited by 159 · see citing works
“Interpreting the definition of Friedman and Nissenbaum [21] for rankings, a biased outcome occurs when a ranking decision is based fully or partially on a protected feature.”
BackgroundHow to Design AI for Social Good: Seven Essential Factors
Floridi, L., Cowls, J., King, T. C., & Taddeo, M. (2021) · Philosophical studies series · doi:10.1007/978-3-030-81907-1_9 · Cited by 171 · see citing works
Reducing disparate exposure in ranking: A learning to rank approach
Zehlike, M., & Castillo, C. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366424.3380048 · Cited by 159 · see citing works
“Researchers have become increasingly concerned with various systematic biases [10] against sociallysalient groups, caused by historic and current discriminatory patternsmaking their way into data-drivenmodels.”
BackgroundDecentering technology in discourse on discrimination
Gangadharan, S. P., & Niklas, J. (2019) · Information Communication & Society · doi:10.1080/1369118x.2019.1593484 · Cited by 161 · see citing works
Critiquing the Reasons for Making Artificial Moral Agents
van Wynsberghe, A., & Robbins, S. (2019) · Science and Engineering Ethics · doi:10.1007/s11948-018-0030-8 · Cited by 161 · see citing works
Knobel, C. P., & Bowker, G. C. (2011) · Communications of the ACM · doi:10.1145/1965724.1965735 · Cited by 158 · see citing works
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · see citing works
Fairness-Aware Tensor-Based Recommendation
Zhu, Z., Hu, X., & Caverlee, J. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3271795 · Cited by 156 · see citing works
“Friedman [7] defined that a computer system is biased “if it systematically and unfairly discriminates against certain individuals or groups of individuals in favor of others.”
BackgroundDiscrimination, artificial intelligence, and algorithmic decision-making
Borgesius, F. Z. (2025) · arXiv · doi:10.48550/arxiv.2510.13465 · Cited by 152 · see citing works
“See also Bryson 2017; Friedman and Nissenbaum 1996; Hacker 2018, p. 1149; Kim 2017, p. 884; Vetzo, Gerards, and Nehmelman 2018, p. 145.”
Machine morality: bottom-up and top-down approaches for modelling human moral faculties
Wallach, W., Allen, C., & Smit, I. (2008) · AI & SOCIETY · doi:10.1007/s00146-007-0099-0 · Cited by 152 · see citing works
“Generally these are the explicit values of the institutions who use the systems and the engineers, who design the systems, although many systems also substantiate often unrecognized implicit values (Friedman and Nissenbaum 1996).”
BackgroundFlanagan, M., Howe, D. C., & Nissenbaum, H. F. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055076 · Cited by 149 · see citing works
“Research has focused, for example, on the value of privacy [1,2], community [13], freedom from bias [23], universal usability [46], autonomy [48], informed consent [19], and trust [16,22,38].”
BackgroundSelf-citationManning, L., Brewer, S., Craigon, P. J., Frey, J. G., Gutiérrez, A., Jacobs, N. J., Kanza, S., Munday, S., Sacks, J., & Pearson, S. (2022) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2022.04.025 · Cited by 147 · see citing works
Ethical and legal challenges of AI in marketing: an exploration of solutions
Kumar, D., & Suthar, N. (2024) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-05-2023-0068 · Cited by 180 · see citing works
Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond
Nasir, S., Khan, R. A., & Bai, S. (2024) · IEEE Access · doi:10.1109/access.2024.3369912 · Cited by 152 · see citing works
“For example, as societal values evolve, the preferences or priorities of patients and healthcare providers may change, leading to biases in algorithmic decision-making or recommendations [79].”
BackgroundIntegrating ethics in design through the value-sensitive design approach
Cummings, M. L. (2006) · Science and Engineering Ethics · doi:10.1007/s11948-006-0065-0 · Cited by 144 · see citing works
AI-driven healthcare: A review on ensuring fairness and mitigating bias
Chinta, S. V., Wang, Z., Palikhe, A., Zhang, X., Kashif, A., Smith, M. A., Liu, J., & Zhang, W. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000864 · Cited by 241 · see citing works
“Implicit bias , on the other hand, often stems from unintentional or unconscious decisions that lead to skewed datasets, frequently influenced by systemic inequities [68].”
InfluentialA Comprehensive Study on Face Recognition Biases Beyond Demographics
Terhörst, P., Kolf, J. N., Huber, M., Kirchbuchner, F., Damer, N., Morales, A., Fiérrez, J., & Kuijper, A. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3111823 · Cited by 146 · see citing works
Informed consent in the Mozilla browser: implementing value-sensitive design
Friedman, B., Howe, D. C., & Felten, E. W. (2003) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2002.994366 · Cited by 140 · see citing works
Self-citationDigital Library Use: Social Practice in Design and Evaluation
Bishop, A. P., VanHouse, N. A., & Buttenfield, B. P. (2003) · CU Scholar (University of Colorado Boulder) · Cited by 139 · see citing works
Addressing age-related bias in sentiment analysis,
Diaz, M., Johnson, I., Lazar, A., Piper, A. M., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173986 · Cited by 138 · see citing works
“Much work in critical algorithm studies examines algorithmic bias, which can be defined as “systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others” [27].”
BackgroundGoverning Algorithms: A Provocation PiecePreprint
Barocas, S., Hood, S., & Ziewitz, M. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2245322 · Cited by 133 · see citing works
Introna, L. D. (2007) · Ethics and Information Technology · doi:10.1007/s10676-006-9133-z · Cited by 133 · see citing works
“This awareness often only emerges over longer historical epochs, as suggested by Heidegger (1977). The practical and the coconstitutive play themselves out in different horizons of intelligibility.”
BackgroundForlano, L., & Mathew, A. (2014) · Journal of Urban Technology · doi:10.1080/10630732.2014.971525 · Cited by 134 · see citing works
“Value-sensitive design has been applied in a range of academic communities including human-computer interaction, computer-supported collaborative work, and information science (Friedman, 1996; Friedman and Nissenbaum, 1996; Nissenbaum, 2001; Le Dantec et al., 2009; Friedman et al., 2013).”
BackgroundTechnology as empowerment: a capability approach to computer ethics
Johnstone, J. (2007) · Ethics and Information Technology · doi:10.1007/s10676-006-9127-x · Cited by 134 · see citing works
“More recent developments that Bynum identifies include ‘valuesensitive design (e.g. Friedman and Nissenbaum 1996; Introna and Nissenbaum 2000) and Brey s ‘disclosive methodology (Brey 2001).”
BackgroundBuilding and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133 · see citing works
“Since at least the mid-1990s, critical scholars have been raising alarms about the potential for computer systems to embed, entrench, and compound social biases [24].”
BackgroundHankerson, D., Marshall, A., Booker, J., Elmimouni, H., Walker, I., & Rode, J. A. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2892578 · Cited by 137 · see citing works
- Authors
- Friedman, B., & Nissenbaum, H.
- Affiliations
- Colby College (US)
- Princeton University (US)
- Venue
- ACM Transactions on Information Systems
- Published
- 1996-07
- Volume
- 14
- Issue
- 3
- Pages
- 330-347
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2006447892
- Cited by
- 1,203
- References
- 13
- Citations by year
- 2026: 102 · 2025: 149 · 2024: 151 · 2023: 121 · 2022: 126 · 2021: 133 · 2020: 79 · 2019: 70 · 2018: 50 · 2017: 21 · 2016: 29 · 2015: 18 · 2014: 18 · 2013: 16 · 2012: 8
- FWCI
- 17.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Keywords
- technical bias, social institutions, technological constraints, societal impact of technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review