Re-imagining Algorithmic Fairness in India and Beyond

In the paper

“of the 138 papers published in 2019 and 2020, only a handful of papers even mention non-West countries, and only one of them-Marda's paper on New Delhi's predictive policing system [127]-substantially engages with a non-Western context.”

What the sources say

• Not checked yetcites [127]

In the paper

“• Occupation: traditional occupations may correspond to caste, gender, or religion; e.g., manual scavenging or butchery.”

In the paper

“Many respondents were concerned that the safety apps were populated by middle-class users and tended to mark Dalit, Muslim, and slum areas as unsafe, potentially leading to hyper-patrolling in these areas.”

In the paper

“Double standards and distance by ML makers: Indian users are perceived as 'bottom billion' data subjects, petri dishes for intrusive models, and given poor recourse-thus, effectively limiting their agency.”

In the paper

“• Skin tone: may indicate caste, ethnicity, and class; however, unlike correlations between race and skin tone, correspondences to Indian sub-groups is weaker.Dark skin tones can be discriminated against in India [60].”

What the sources say

• Not checked yetcites [60]

In the paper

“While Indians are part of the AI workforce, a majority work in services, and engineers do not entirely represent marginalities, limiting re-mediation of distances.”

In the paper

“AI is readily adopted in high-stakes domains, often too early.”

In the paper

“Sub-groups like caste (endogamous, ranked social identities, assigned at birth [17,191]), 1 gender, and religion require different fairness implementations; but AI systems in India are under-analyzed for biases, mirroring the limited mainstream public discourse on oppression.”

• Also stated in: § Gender Identity & Sexual Orientation (No official · plain

  “AI systems in India remain under-analysed for biases, mirroring the limited public discourse on oppression.”

What the sources say

• Not checked yetcites [17]
• Not checked yetcites [191]

In the paper

“The AI imaginary is aspirational in the Indian state, media, and legislation.”

In the paper

“• Zip codes: can correspond to caste, religion, ethnicity, or class. Many Indian zip codes are heterogeneous, with slums and posh houses abutting in the same area [20,36], unlike the rather homogeneous zip codes in the West from the segregated past [50].”

What the sources say

• Not checked yetcites [20]
• Not checked yetcites [36]
• Not checked yetcites [50]

In the paper

“Some respondents reported on how the state and industry apparati collected and retained valuable, large-scale data, but the datasets were not always made publicly available due to infrastructure and non-transparency issues.”

In the paper

“• Expenditure: on dietary and lifestyle items may be proxies for religion, caste, or ethnicity; e.g., expenses on pork or beef.”

In the paper

“Models are over-fitted for digitally-rich profiles-typically middle-class men-further excluding the 50% without Internet access.”

In the paper

“Many respondents observed a bias towards upper-middle class problems, data, and deployments due to easier data access and revenue, as P8, CS/IS (computer science/information sciences researcher) put it, “rich people problems like cardiac disease and cancer, not poor people's Tuberculosis, prioritised in AI [in India]”, exacerbating inequities among those who benefit from AI and those who do not.”

In the paper

“Many respondents described how datasets under-collected darker skin tones and measurement scales like Fitzpatrick scale are not calibrated on diverse Indian skin tones.”

In the paper

“• Names: are revelatory proxies for caste, religion, gender, and ethnicity, and have contributed to discrimination in India [15,202], e.g., Banerjee or Khan connote to caste or religion location.”

What the sources say

• Not checked yetcites [15]
• Not checked yetcites [202]

In the paper

“Some respondents pointed to how Dalit and Muslim bodies were used as test subjects for AI surveillance, e.g., pointing to how human efficiency trackers were increasingly deployed among Dalit sanitation workers in cities like Panchkula and Nagpur.”

In the paper

“Several respondents described how strong aspiration for AI for socio-economic upliftment was accompanied by high trust in automation, limited transparency, and the lack of an empowered Fair-ML ecosystem in India.”

In the paper

“In contrast to how AI industry and research were moderately responsive to user bias reports in the West, recourse and redress for Indians were perceived to be non-existent.”

In the paper

“Many described how social audits and working with civil society created a better understanding and accountability.8”

In the paper

“Many respondents described how lending models imagined middle-class men as end-users-even with many microfinance studies showing that women have high loan repayment rates [69,198].”

What the sources say

• Not checked yetcites [69]
• Not checked yetcites [198] source

In the paper

“Several respondents pointed to how automation solutions had fervent rhetoric; whereas in practice, accuracy and performance of systems were low.”

In the paper

“Such interventions were criticised for placing power in the hands of dominant-caste supervisors.”

In the paper

“Many discussed how low quality designs, algorithms, and support were deployed for Indians, attributing to weak tech policy and enforcement of accountability in India.”

In the paper

“• Language: Language can correspond to religion, class, ethnicity, and caste.Many AI systems serve in English, which only 10% of Indians understand [174].”

What the sources say

• Not checked yetcites [174] source

In the paper

“Many respondents described how the Indian technology sector claimed to be 'merit-based', open to anyone highly gifted in the technical sciences; but many have pointed to how merit is a function of caste privilege [196,207].”

What the sources say

• Not checked yetcites [196] source
• Not checked yetcites [207]

In the paper

“Several respondents discussed how Indian AI engineers were largely”

In the paper

“• Documentation: Several AI applications require state-issued documentation like Aadhaar or birth certificates, e.g., in finance.The economically poor are also reported to be document-poor [173].”

What the sources say

• Not checked yetcites [173]

In the paper

“• Mobility: can correlate to gender and disability.Indian women travel shorter distances and depend on public transport, for safety and cost [138].Persons with disabilities often have lower mobility in India, due to a lack of ramps and accessible infrastructure [188].”

What the sources say

• Not checked yetcites [138]
• Not checked yetcites [188]

In the paper

“Many criticised how neo-liberal AI (including those from the state) tended to treat Indians as 'bottom billion data subjects' in the periphery [211]-being subject to intrusive models, non-consensual automation, poor tech policies, inadequate user research, low-cost or free products that are low standard, and considered 'unsaturated markets'.”

What the sources say

• Not checked yetcites [211]

In the paper

“In some cases, those with 'poor' data profiles subverted model predictions-as in P23's (STS researcher) research on financial lending, where women overwhelmingly availed and paid back loans in the names of male relatives to avoid perceived gender bias in credit scoring.”

In the paper

“Even when feedback mechanisms were included, respondents shared that they were not always localised for India, and incidents were not always recognised unless an activist contacted the company staff directly.”

In the paper

“• Devices and infrastructure: Internet access corresponds to gender, caste, and class, with 67% Internet users being males [102].”

What the sources say

• Not checked yetcites [102]

In the paper

“For instance, P11, tech policy researcher, illustrated how lending apps instantly determined creditworthiness through alternative credit histories built based on the user's SMS messages, calls, and social networks (due to limited credit or banking history).”

In the paper

“Some talked about a colonial mindset of tight control in decision-making on automation laws, leading to reticent and monoscopic views by the judiciary and state.”

In the paper

“e.g., P29 (LGBTQ+ activist) discussed how AI was justified in the public domain, e.g., surveillance for smart cities, 11 as women's safety measures, but tended to invisibilise transgender members or increase monitoring of Dalit and Muslim areas, e.g., a FR was deployed outside women's restrooms to detect intrusion by non-female entrants, potentially leading to violence against transgender members.”

In the paper

“Many respondents expressed concern over AI advances in detecting sexuality, criminality, or terrorism (e.g., [187,212]) potentially being exported to India and harming minorities.”

What the sources say

• Not checked yetcites [187]
• Not checked yetcites [212] source

In the paper

“Critiques were raised in our study on how neo-liberal AI followed a broader pattern of extraction from the 'bottom billion' data subjects and labourers.”

In the paper

“Many described how disproportionate confidence in high-tech solutions, combined with limited technology policy engagement among decision-makers, appeared to lead to sub-par high-stakes solutions, e.g., the FR service used by Delhi Police was reported to have very low accuracy and failed to distinguish between boy and girl children [5]. 10”

What the sources say

• Not checked yetcites [5]

In the paper

“P15 (journalist) pointed out that journalists may face disapproval for questioning certain narratives.”

In the paper

“Our study discusses how caste, religion, and tribe are eluded even within the Indian technology discourse and policy.”

In the paper

“We found that data was not always reliable due to socioeconomic factors, ML products for Indian users sufffer from double standards, and AI was seen with unquestioning aspiration.”

In the paper

“Additionally, respondents pointed to significant scope creep in high-stakes AI, e.g., a few mentioned how the tender for an FR system was initially motivated by detecting criminals, later missing children, to then arresting protestors [5].”

What the sources say

• Not checked yetcites [5]

In the paper

“Some respondents mentioned how a few automation solutions were announced following public sentiment, but turned into surveillance e.g., how predictive policing in Delhi and FR in train stations was announced after Nirbhaya's gruesome gangrape in 2012 and women's safety incidents [28,33].”

What the sources say

• Not checked yetcites [28]
• Not checked yetcites [33]

In the paper

“Many respondents described how limited debate and analysis of AI in India led to a weak implementation of”

In the paper

“Fairness in India was reported to suffer from a lack of access to contributing datasets, APIs, and documentation, with several respondents describing how challenging it was for researchers and civil society to assess the high-stakes AI systems.”

In the paper

“We presented a qualitative study and discourse analysis of algorithmic power in India, and found that algorithmic fairness assumptions are challenged in the Indian context.”

In the paper

“Indeed, notions of algorithmic fairness seem to fit within a broad arc of enlightenment and post-enlightenment thinking, including in actuarial risk assessment [147].”

What the sources say

• Not checked yetcites [147] source

In the paper

“Similarly, a study of children in three different cultures found that notions of distributive justice are not universal: “children from a partially hunter-gatherer, egalitarian African culture distributed the spoils more equally than did the other two cultures, with merit playing only a limited role” [185].”

What the sources say

• Not checked yetcites [185] source

In the paper

“Summarising decades worth of research, [120] says that the more collectivist and hierarchical societies in the East differs from the more individualistic and egalitarian cultures of the West in how different forms of justicedistributive, procedural, and retributive-are conceptualised and achieved.”

What the sources say

• Not checked yetcites [120] source

In the paper

“Conventional western AI fairness is becoming a universal ethical framework for AI; consider the AI strategies from India [1], Tunisia [205], Mexico [129], and Uruguay [13] that espouse fairness and transparency, but pay less attention to what is fair in local contexts.”

What the sources say

• Not checked yetcites [1]
• Not checked yetcites [205]
• Not checked yetcites [129] source

In the paper

“Indeed, researchers since the late 1960s have tried to translate US antidiscrimination law into statistical metrics [90].”

What the sources say

• Not checked yetcites [90]

In the paper

“While the concern for justice has a long history in the West (e.g., Aristotle, Rawls) and the East (e.g., Confucius, Chanakya), they show that the majority of empirical work on social justice has been situated in the US and Western Europe, grounding the understanding of justice in the Western cultural context.”

What the sources say

• Not checked yetcites [120] source

In the paper

“Without engagement with the conditions, values, politics, and histories of the non-West, AI fairness can be a tokenism, at best-pernicious, at worst-for communities.”

In the paper

“For instance, [40] compared the acquisition of fairness behaviour in seven different societies: Canada, India, Mexico, Peru, Senegal, Uganda, and the USA, and found that while children from all cultures developed aversion towards disadvantageous inequity (avoid receiving less than a peer), advantageous inequity aversion (avoid receiving more than a peer) was more prevalent in the West.”

What the sources say

• Not checked yetcites [40] source

In the paper

“Researchers have documented societal biases in predictive policing in New Delhi [127], caste [202] and ethnicity [15] biases in job applications, call-center job callbacks [27], castebased wage-gaps [124], caste discrimination in agricultural loans decisions [116], and even in online matrimonial ads [157].”

What the sources say

• Not checked yetcites [127]
• Not checked yetcites [202]
• Not checked yetcites [15]

In the paper

“We recruited respondents via a combination of reaching out directly and personal contacts, using purposeful sampling [151]-i.e., identifying and selecting experts with relevant experience-iterative until saturation.”

What the sources say

• Not checked yetcites [151] source

In the paper

“Accountability is fundamentally about answerability for actions [114], and central to these are three phases by which an actor is made answerable to a forum: information-sharing, deliberation and discussion, and the imposition of consequences [214].”

What the sources say

• Not checked yetcites [114] source
• Not checked yetcites [214]

In the paper

“Lack of an ecosystem of tools, policies, and stakeholders like journalists, researchers, and activists to interrogate high-stakes AI inhibits meaningful fairness in India.”

In the paper

“Representational gaps in data is documented as one of the major challenges in achieving responsible AI from a global perspective [21,190].”

What the sources say

• Not checked yetcites [21]
• Not checked yetcites [190] source

In the paper

“Half the Indian population lacks access to the Internet-the excluded half is primarily women, rural communities, and Adivasis [96,106,153,168,189].”

What the sources say

• Not checked yetcites [96]
• Not checked yetcites [106]
• Not checked yetcites [153]
• Not checked yetcites [168]
• Not checked yetcites [189]

In the paper

“Recently, the IEEE Standards Association highlighted the monopoly of Western ethical traditions in AI ethics, and inquired how incorporating Buddhist, Ubuntu, and Shinto-inspired ethical traditions might change the processes of responsible AI [93].”

What the sources say

• Not checked yetcites [93]

In the paper

“Accountability is fundamentally about answerability for actions [114], and central to these are three phases by which an actor is made answerable to a forum: information-sharing, deliberation and discussion, and the imposition of consequences [214].”

What the sources say

• Not checked yetcites [214]

In the paper

“We took great care to create a research ethics protocol to protect respondent privacy and safety, especially due to the sensitive nature of our inquiry.”

In the paper

“conventional gender roles in Indian society lead to men having better access to devices, documentation, and mobility (see [64,178]), women often borrowed phones.”

What the sources say

• Not checked yetcites [64] source
• Not checked yetcites [178]

In the paper

“In a societal context where the distance between models and dis-empowered communities is large-via technical distance, social distance, ethical distance, temporal distance, and physical distance-a myopic focus on localising fair model outputs alone can backfire.”

In the paper

“Nissenbaum distinguishes four barriers to responsibility in computer systems: (1) the problem of many hands, (2) bugs, (3) blaming the computer, and (4) ownership without liability [146].”

What the sources say

• Not checked yetcites [146]

In the paper

“Transcripts were coded and analyzed for patterns using an inductive approach [201].”

What the sources say

• Not checked yetcites [201]

In the paper

“Many, though not all, graduates of Indian Institutes of Technology, founders of pioneering Indian software companies and nearly all Nobel prize winners of Indian origin have come from privileged castes and class backgrounds [71,196].”

What the sources say

• Not checked yetcites [71]
• Not checked yetcites [196] source

In the paper

“Researchers have pointed to the aspirational role played by technology in India, signifying symbolic meanings of modernity and progress via technocracy [149,150,176].”

What the sources say

• Not checked yetcites [149] source
• Not checked yetcites [150]
• Not checked yetcites [176]

In the paper

“Reservations assigns quotas for marginalised communities at the national and state levels.7”

What the sources say

• Not checked yetcites [164] source

In the paper

“reported to be attractive for improving model robustness and training data [14].”

What the sources say

• Not checked yetcites [14]

In the paper

“many have pointed to how merit is a function of caste privilege [196,207].”

What the sources say

• Not checked yetcites [196] source
• Not checked yetcites [207]

In the paper

“AI for societal benefit is a pivotal development thrust in India, with a focus on healthcare, agriculture, education, smart cities, and mobility [1]-influencing citizen imaginaries of AI.”

What the sources say

• Not checked yetcites [1]

In the paper

“Depending on the policy, reservations can allocate quotas from 30% to 80%.”

In the paper

“Equipped with microphones, GPS, cameras, and a SIM, the trackers allowed detailed surveillance of movement and work, leading to some women workers avoiding restrooms for fear of camera capture, avoiding sensitive conversations for fear of snooping, and waiting for the tracker to die before going home [111].”

What the sources say

• Not checked yetcites [111]

In the paper

“Indian film stars are overwhelmingly fair-skinned, conventionally attractive, and able-bodied [108], not fully representative of the larger society.”

What the sources say

• Not checked yetcites [108] source

In the paper

“Originally designed for Dalits and Adivasis, reservations have expanded to include other backward castes, women, persons with disabilities, and religious minorities.”

What the sources say

• Not checked yetcites [164] source

In the paper

“While engineers and researchers are mostly privileged everywhere, the stark socio-economic disparities between Indian engineers and the marginalised communities may further amplify the distances.”

In the paper

“One of the restorative justice measures to repair centuries of subordination-reservations are a type of affirmative action enshrined in the Indian constitution [164].”

What the sources say

• Not checked yetcites [164] source

In the paper

“Contrast with the West, where a large, active stakeholder ecosystem (of civil society, journalists, and law makers) is AI-literate and has access to open APIs and data.”

In the paper

“Reservations in India have been described as one of the world's most radical policies [25] (see [164] for more).”

What the sources say

• Not checked yetcites [25] source
• Not checked yetcites [164] source

In the paper

“AI is even more aspirational and consequential in non-Western nations, where it performs an 'efficiency' function in distributing scarce socio-economic resources and differentiating economies.”

In the paper

“AI is increasingly perceived as a masculine, hype-filled, technoutopian enterprise, with nations turning into AI superpowers [170].”

What the sources say

• Not checked yetcites [170]

In the paper

“The considerations we identified are certainly not limited to India; likewise, we call for inclusively evolving global approaches to Fair-ML.”

In the paper

“Low costs, large and diverse populations, and policy infirmities have been cited as reasons for following double standards in India, e.g., in non-consensual human trials and waste dumping [123] (also see [137]).”

What the sources say

• Not checked yetcites [123]
• Not checked yetcites [137]

In the paper

“For instance, personal names act as a signifier for various socio-demographic attributes in India, however there are no large datasets of Indian names (like the US Census data, or the SSA data) that are readily available for fairness evaluations.”

In the paper

“e.g., according to the National Crime Records Bureau (NCRB) in 2015, 34% of undertrials were Dalits and Adivasis (25% of the population); 20% were Muslims (14% of population); and 70% were non-literate (26% of the population) [203].”

What the sources say

• Not checked yetcites [203]

In the paper

“Recontextualising data and models can only go so far without participatorily empowering communities in identifying problems, specifying fairness expectations, and designing systems.”

In the paper

“The quota system gives rise to the problem of matching under distributional constraints, a known problem in economics [23,79,105], but has not received attention within the FAccT community (although [51] is related).”

What the sources say

• Not checked yetcites [23] source
• Not checked yetcites [79]
• Not checked yetcites [105]

In the paper

“Missing Indic factors and values in data and models, compounded by double standards and disjuncture in ML, deployed in an environment of unquestioning AI aspiration, face the risk of reinforcing injustices and harms.”