CoChat
Sign Up
Cited by 76Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)

Conceptualising fairness: three pillars for medical algorithms and health equity

Open access
Sikstrom, Laura, Maslej, Marta M, Hui, Katrina, Findlay, Zoe, Buchman, Daniel Z, Hill, Sean L·2022·BMJ Health & Care Informatics
DOI 10.1136/bmjhci-2021-100459PMC PMC8753410
Open this paper in CoChat — free accountOpen PDF

Abstract

Objectives: Fairness is a core concept meant to grapple with different forms of discrimination and bias that emerge with advances in Artificial Intelligence (eg, machine learning, ML). Yet, claims to fairness in ML discourses are often vague and contradictory. The response to these issues within the scientific community has been technocratic. Studies either measure (mathematically) competing definitions of fairness, and/or recommend a range of governance tools (eg, fairness checklists or guiding principles). To advance efforts to operationalise fairness in medicine, we synthesised a broad range of literature. Methods: We conducted an environmental scan of English language literature on fairness from 1960-July 31, 2021. Electronic databases Medline, PubMed and Google Scholar were searched, supplemented by additional hand searches. Data from 213 selected publications were analysed using rapid framework analysis. Search and analysis were completed in two rounds: to explore previously identified issues (a priori), as well as those emerging from the analysis (de novo). Results: Our synthesis identified ‘Three Pillars for Fairness’: transparency, impartiality and inclusion. We draw on these insights to propose a multidimensional conceptual framework to guide empirical research on the operationalisation of fairness in healthcare. Discussion: We apply the conceptual framework generated by our synthesis to risk assessment in psychiatry as a case study. We argue that any claim to fairness must reflect critical assessment and ongoing social and political deliberation around these three pillars with a range of stakeholders, including patients. Conclusion: We conclude by outlining areas for further research that would bolster ongoing commitments to fairness and health equity in healthcare.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjhci-2021-100459

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Cognitive bias is introduced during data annotation or input processes and often reflects subjective judgments of healthcare providers, such as manual data entry errors or inconsistent diagnoses.

    “Cognitive bias also poses a major challenge (n = 5), introduced during data annotation or input processes. Kudina et al. and Sikstrom et al., [49,51] acknowledge that this bias often reflects the subjective judgments of healthcare providers, such as manual data entry errors or inconsistent diagnoses.”

    From Frameworks encompassing intersectional perspective of artificial intelligence in healthcare. Scoping review (2026) · cites this paper as [51]

    The Results > Impartiality section says medical practices can reflect arbitrary assumptions about patients and that human actors help gather and curate datasets. The Discussion also gives misdiagnosis of schizophrenia as an example of a feature that could bias a model. The paper does not specifically describe cognitive bias arising during annotation or input, manual data-entry errors, or inconsistent diagnoses as a general mechanism; page numbers were not available in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Models validated on biased datasets frequently exhibit reduced accuracy for underserved populations, such as rural communities or individuals with unique healthcare needs.

    “For instance, Sikstrom et al. and Haseeb et al., [46,51] reveal that models validated on biased datasets frequently exhibit reduced accuracy for underserved populations, such as rural communities or individuals with unique healthcare needs.”

    From Frameworks encompassing intersectional perspective of artificial intelligence in healthcare. Scoping review (2026) · cites this paper as [51]

    The Introduction says models trained on biased datasets can amplify inequities, and Results > Inclusion says small samples of marginalised groups can bias outputs toward the majority group. Those passages support the general concern, but they do not establish that models validated on biased datasets frequently have reduced accuracy for rural communities or people with unique healthcare needs. No page numbers are available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 146

  1. [1]Neff G. Why Big Data Won't Cure Us. Big Data 2013; 1: 117– 23. 10.1089/big.2013.0029 25161827 PMC4114418 · DOI
  2. [2]Singh JP, Grann M, Fazel S. A comparative study of violence risk assessment tools: a systematic review and metaregression analysis of 68 studies involving 25,980 participants. Clin Psychol Rev 2011; 31: 499– 513. 10.1016/j.cpr.2010.11.009 21255891 · DOI
  3. [4]Chen IY, Joshi S, Ghassemi M. Treating health disparities with artificial intelligence. Nat Med 2020; 26: 16– 17. 10.1038/s41591-019-0649-2 31932779 · DOI · Paper page
  4. [6]Friedler SA, Scheidegger C, Venkatasubramanian S. On the (im) possibility of fairness. arXiv preprint arXiv:1609 07236, 2016. Available: https://arxiv.org/abs/1609.07236 · arXiv
  5. [8]Green B, Hu L. The myth in the methodology: Towards a recontextualization of fairness in machine learning. In: Proceedings of the machine learning: the debates workshop, 2018. https://econcs.seas.harvard.edu/files/econcs/files/green_icml18.pdf
  6. [9]Barocas S, Hardt M, Narayanan A. Fairness in machine learning. Nips tutorial 2017; 1: 2017.
  7. [10]Darlington RB. Another look at "cultural fairness"1. J Educ Meas 1971; 8: 71– 82. 10.1111/j.1745-3984.1971.tb00908.x · DOI · Paper page
  8. [11]Cleary TA. Test bias: prediction of grades of Negro and white students in integrated colleges. J Educ Meas 1968; 5: 115– 24. 10.1111/j.1745-3984.1968.tb00613.x · DOI · Paper page
  9. [12]Dwork C, Hardt M, Pitassi T. Fairness through awareness. Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, New York, NY, USA, 2012: 214– 26.
  10. [16]Srivastava M, Heidari H, Krause A. Human perception of fairness: a descriptive approach to fairness for machine learning. in association for computing machinery. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, New York, NY, USA, 2019: 2459– 68.
  11. [17]Zou J, Schiebinger L. Design AI so that it’s fair. Nature 2018; 559: 324– 6. 30018439
  12. [19]boyd danah, Crawford K. Critical questions for big data. Information, Communication & Society 2012; 15: 662– 79. 10.1080/1369118X.2012.678878 · DOI · Paper page
  13. [20]Rawls J. Fairness to Goodness. Philos Rev 1975; 84: 536– 54. 10.2307/2183853 · DOI
  14. [21]Powers M. Assistant Professor of Philosophy Madison Powers. In: Social justice: the moral foundations of public health and health policy. Oxford University Press, 2006.
  15. [22]Nuhrat Y. Fair to swear? gendered formulations of fairness in football in turkey. Journal of Middle East Women’s Studies 2017; 13: 25– 46.
  16. [23]Berliner D, Lambek M, Shweder R, et al. Anthropology and the study of contradictions. HAU: Journal of Ethnographic Theory 2016; 6: 1– 27. 10.14318/hau6.1.002 · DOI
  17. [24]Meegan D. America the fair: using brain science to create a more just nation. Cornell University Press, 2019.
  18. [25]Wolsink M. Wind power implementation: The nature of public attitudes: Equity and fairness instead of ‘backyard motives’. Renewable and Sustainable Energy Reviews 2007; 11: 1188– 207. 10.1016/j.rser.2005.10.005 · DOI
  19. [26]Butler R. Children making sense of economic insecurity: Facework, fairness and belonging. Journal of Sociology 2017; 53: 94– 109. 10.1177/1440783316630113 · DOI
  20. [27]Deeming C. Social democracy and social policy in neoliberal times. J Sociol 2014; 50: 577– 600. 10.1177/1440783313492240 PMC4246053 25473376 · DOI
  21. [28]Benjamin R. Race after technology: Abolitionist tools for the new jim code. Soc Forces 2020; 98: 1– 3. 10.1093/sf/soz162 · DOI
  22. [29]Perez CC. Invisible women: exposing data bias in a world designed for men random house, 2019. PMC7194768 32354824
  23. [30]Crawford K. Atlas of AI. Yale University Press, 2021.
  24. [34]Rajkomar A, Hardt M, Howell MD, et al. Ensuring fairness in machine learning to advance health equity. Ann Intern Med 2018; 169: 866– 72. 10.7326/M18-1990 30508424 PMC6594166 · DOI · Paper page
  25. [35]Panch T, Mattie H, Celi LA. The "inconvenient truth" about AI in healthcare. NPJ Digit Med 2019; 2: 77. 10.1038/s41746-019-0155-4 31453372 PMC6697674 · DOI · Paper page
  26. [37]Boag W, Kovaleva O, McCoy TH, et al. Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes. Transl Psychiatry 2021; 11: 32. 10.1038/s41398-020-01104-w 33431794 PMC7801508 · DOI
  27. [38]Suchting R, Green CE, Glazier SM, et al. A data science approach to predicting patient aggressive events in a psychiatric hospital. Psychiatry Res 2018; 268: 217– 22. 10.1016/j.psychres.2018.07.004 30064068 · DOI
  28. [39]Viljoen JL, Cochrane DM, Jonnson MR. Do risk assessment tools help manage and reduce risk of violence and reoffending? A systematic review. Law Hum Behav 2018; 42: 181– 214. 10.1037/lhb0000280 29648841 · DOI
  29. [40]Connor M, Armbruster M, Hurley K, et al. Diagnostic sensitivity of the dynamic appraisal of situational aggression to predict violence and aggression by behavioral health patients in the emergency department. J Emerg Nurs 2020; 46: 302– 9. 10.1016/j.jen.2019.12.006 32063386 · DOI
  30. [41]Rawls J. Justice as fairness. Philos Rev 1958; 67: 164– 94. 10.2307/2182612 · DOI
  31. [42]Carr CL. On fairness. Canadian centre for diversity and inclusion (January 2018. Routledge, 2017.
  32. [43]Fairness sWJ. Respect and the egalitarian ethos revisited. J Ethics 2010; 14: 335– 50.
  33. [44]Silverstein M. The indeterminacy of contextualization: when is enough enough. The contextualization of language 1992; 22: 55– 76.
  34. [47]Rowel R, Moore ND, Nowrojee S, et al. The utility of the environmental scan for public health practice: lessons from an urban program to increase cancer screening. J Natl Med Assoc 2005; 97: 527– 34. 15868772 PMC2568702
  35. [48]Charlton P, Kean T, Liu RH, et al. Use of environmental scans in health services delivery research: a scoping review. BMJ Open 2021; 11: e050284. 10.1136/bmjopen-2021-050284 PMC8587593 34758992 · DOI · Paper page
  36. [49]Krishnan M. Against Interpretability: a critical examination of the Interpretability problem in machine learning. Philos Technol 2020; 33: 487– 502. 10.1007/s13347-019-00372-9 · DOI
  37. [51]Zarsky T. The trouble with algorithmic decisions: an analytic road map to examine efficiency and fairness in automated and opaque decision making. Sci Technol Human Values 2016; 41: 118– 32.
  38. [55]Sandvig C, Hamilton K, Karahalios K. Auditing algorithms: research methods for detecting discrimination on Internet platforms. Data and discrimination: converting critical concerns into productive inquiry 2014; 22: 4349– 57.
  39. [57]Lepri B, Oliver N, Letouzé E, et al. Fair, transparent, and accountable algorithmic decision-making processes. Philos Technol 2018; 31: 611– 27. 10.1007/s13347-017-0279-x · DOI · Paper page
  40. [58]Char DS, Abràmoff MD, Feudtner C. Identifying ethical considerations for machine learning healthcare applications. Am J Bioeth 2020; 20: 7– 17. 10.1080/15265161.2020.1819469 PMC7737650 33103967 · DOI · Paper page
  41. [59]Brundage M, Avin S, Wang J. Toward trustworthy AI development: mechanisms for supporting verifiable claims. arXiv, 2020. Available: https://arxiv.org/abs/2004.07213 · arXiv
  42. [60]Krompaß D, Esteban C, Tresp V, et al. Exploiting latent Embeddings of nominal clinical data for predicting Hospital readmission. KI - Künstliche Intelligenz 2015; 29: 153– 9. 10.1007/s13218-014-0344-x · DOI
  43. [61]Geiger RS. Beyond opening up the black box: investigating the role of algorithmic systems in Wikipedian organizational culture. Big Data Soc 2017; 4: 205395171773073. 10.1177/2053951717730735 · DOI
  44. [63]et al Xu Z, Liu J, Cheng D. Assessing the Fairness of Classifiers with Collider Bias. arXiv [cs.LG], 2020. Available: http://arxiv.org/abs/2010.03933 · arXiv
  45. [64]Darlow L, Jastrzębski S, Storkey A. Latent Adversarial Debiasing: Mitigating Collider Bias in Deep Neural Networks. arXiv [cs.LG], 2020. Available: http://arxiv.org/abs/2011.11486 · arXiv
  46. [65]Prosperi M, Guo Y, Sperrin M, et al. Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nat Mach Intell 2020; 2: 369– 75. 10.1038/s42256-020-0197-y · DOI · Paper page
  47. [66]et al Chou Y-L, Moreira C, Bruza P. Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications. arXiv [cs.AI], 2021. Available: http://arxiv.org/abs/2103.04244 · arXiv
  48. [67]Saravanakumar KK. The Impossibility Theorem of Machine Fairness - A Causal Perspective. arXiv [cs.LG], 2020. Available: http://arxiv.org/abs/2007.06024 · arXiv
  49. [68]Kleinberg J. Inherent trade-offs in algorithmic fairness. Abstracts of the 2018 ACM International Conference 2018 https://dl.acm.org/doi/abs/10.1145/3219617.3219634 · DOI · Paper page
  50. [69]Wang C, Han B, Patel B. In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction. arXiv [stat.ML], 2020. Available: http://arxiv.org/abs/2005.04176 · arXiv
  51. [70]Slack D, Hilgard S, Jia E. How can we fool lime and SHAP?Adversarial Attacks on Post hoc Explanation Methods, 2019. Available: https://openreview.net/forum?id=nTHOa8__v0B [Accessed 19 Jul 2021].
  52. [72]Watson DS, Floridi L. The explanation game: a formal framework for interpretable machine learning. Synthese 2021; 198: 9211– 42. 10.1007/s11229-020-02629-9 · DOI
  53. [73]Samek W, Wiegand T, Müller K-R. Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models. arXiv [cs.AI], 2017. Available: http://arxiv.org/abs/1708.08296 · arXiv
  54. [74]Gunning D, Aha D. DARPA’s explainable artificial intelligence (XAI) program. AI Mag 2019; 40: 44– 58.
  55. [75]Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 2019; 1: 206– 15. 10.1038/s42256-019-0048-x PMC9122117 35603010 · DOI · Paper page
  56. [76]Babic B, Gerke S, Evgeniou T, et al. Beware explanations from AI in health care. Science 2021; 373: 284– 6. 10.1126/science.abg1834 34437144 · DOI · Paper page
  57. [79]McCradden MD. When is accuracy off-target? Transl Psychiatry 2021; 11: 369. 10.1038/s41398-021-01479-4 34226492 PMC8257647 · DOI
  58. [80]Kaler A. Health interventions and the persistence of rumour: the circulation of sterility stories in African public health campaigns. Soc Sci Med 2009; 68: 1711– 9. 10.1016/j.socscimed.2009.01.038 19282080 · DOI
  59. [81]Nations MK, Monte CM. "I'm not dog, no!": cries of resistance against cholera control campaigns. Soc Sci Med 1996; 43: 1007– 24. 10.1016/0277-9536(96)00083-4 8888470 · DOI
  60. [82]Fairhead J. Understanding social resistance to the Ebola response in the forest region of the Republic of guinea: an anthropological perspective. Afr Stud Rev 2016; 59: 7– 31. 10.1017/asr.2016.87 · DOI
  61. [83]Chandler C, Fairhead J, Kelly A, et al. Ebola: limitations of correcting misinformation. Lancet 2015; 385: 1275– 7. 10.1016/S0140-6736(14)62382-5 25534188 · DOI
  62. [84]Saha D, Schumann C, Mcelfresh D. Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics. In: Iii HD, Singh A, eds. Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020: 8377– 87.
  63. [85]Kelliher A, Barry B. Designing Therapeutic Care Experiences with AI in Mind. In: 2018 AAAI Spring Symposium Series. aaai.org, 2018. Available: https://www.aaai.org/ocs/index.php/SSS/SSS18/paper/viewPaper/17585
  64. [86]et al Anderson A, Dodge J, Sadarangani A. Explaining Reinforcement Learning to Mere Mortals: An Empirical Study. arXiv [cs.HC], 2019. Available: http://arxiv.org/abs/1903.09708 · arXiv
  65. [88]Ho A. Are we ready for artificial intelligence health monitoring in elder care? BMC Geriatr 2020; 20: 358. 10.1186/s12877-020-01764-9 32957946 PMC7504871 · DOI
  66. [89]Hengstler M, Enkel E, Duelli S. Applied artificial intelligence and trust—The case of autonomous vehicles and medical assistance devices. Technol Forecast Soc Change 2016; 105: 105– 20. 10.1016/j.techfore.2015.12.014 · DOI · Paper page
  67. [90]Peck EM, Ayuso SE, El-Etr O. Data is personal: attitudes and perceptions of data visualization in rural Pennsylvania. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2019: 1– 12.
  68. [92]Carr S. 'AI gone mental': engagement and ethics in data-driven technology for mental health. J Ment Health 2020; 29: 125– 30. 10.1080/09638237.2020.1714011 32000544 · DOI
  69. [94]Reddy S, Allan S, Coghlan S, et al. A governance model for the application of AI in health care. J Am Med Inform Assoc 2020; 27: 491– 7. 10.1093/jamia/ocz192 31682262 PMC7647243 · DOI · Paper page
  70. [95]Baker DB, Kaye J, Terry SF. Governance through privacy, fairness, and respect for individuals. EGEMS 2016; 4: 1207. 10.13063/2327-9214.1207 27141520 PMC4827784 · DOI
  71. [98]Noble SU. Algorithms of Oppression: how search Engines reinforce racism. NYU Press, 2018. 34709921
  72. [99]Pierson E, Cutler DM, Leskovec J, et al. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nat Med 2021; 27: 136– 40. 10.1038/s41591-020-01192-7 33442014 · DOI · Paper page
  73. [100]Higashi RT, Tillack A, Steinman MA, et al. The 'worthy' patient: rethinking the 'hidden curriculum' in medical education. Anthropol Med 2013; 20: 13– 23. 10.1080/13648470.2012.747595 23581400 · DOI
  74. [104]Kleinberg J, Ludwig J, Mullainathan S, et al. Algorithms as discrimination detectors. Proc Natl Acad Sci U S A 2020; 117: 30096– 100. 10.1073/pnas.1912790117 32723823 PMC7720101 · DOI
  75. [105]Dourish P. What we talk about when we talk about context. Pers Ubiquitous Comput 2004; 8: 19– 30. 10.1007/s00779-003-0253-8 · DOI · Paper page
  76. [106]de Paula R, Holanda M, Gomes LSA, et al. Provenance in bioinformatics workflows. BMC Bioinformatics 2013; 14 Suppl 11: S6. 10.1186/1471-2105-14-S11-S6 PMC3816297 24564294 · DOI
  77. [107]Mayernik MS. Open data: accountability and transparency. Big Data Soc 2017; 4: 205395171771885. 10.1177/2053951717718853 · DOI · Paper page
  78. [108]Wercelens P, da Silva W, Castro K. Data Provenance Management of Bioinformatics Workflows in Federated Clouds. In: 2019 IEEE International Conference on bioinformatics and biomedicine, 2019: 750– 4.
  79. [109]Goble C, Cohen-Boulakia S, Soiland-Reyes S, et al. Fair computational Workflows. Data Intelligence 2020; 2: 108– 21. 10.1162/dint_a_00033 · DOI
  80. [110]Leavy S, Siapera E, O’Sullivan B. Ethical Data Curation forAI: An Approach based on Feminist Epistemology and CriticalTheories of Race. aies-conference.com. Available: https://www.aies-conference.com/2021/wp-content/posters/171_%20Ethical%20Data%20Curation%20for%20AI_%20An%20Approach%20based%20on%20Feminist%20Epistemology%20and%20Critical%20Theories%20of%20Race.pdf
  81. [111]ES J, Gebru T. Lessons from archives: strategies for collecting sociocultural data in machine learning. arXiv [cs.LG] 2019.
  82. [112]Seaver N. Algorithms as culture: some tactics for the ethnography of algorithmic systems. Big Data & Society 2017. 10.1177/2053951717738104 · DOI · Paper page
  83. [113]Zajko M. Conservative AI and social inequality: conceptualizing alternatives to bias through social theory. AI Soc 2007 https://arxiv.org/ftp/arxiv/papers/2007/2007.08666.pdf
  84. [114]Neff G, Tanweer A, Fiore-Gartland B, et al. Critique and contribute: a practice-based framework for improving critical data studies and data science. Big Data 2017; 5: 85– 97. 10.1089/big.2016.0050 28632445 PMC5515123 · DOI · Paper page
  85. [115]Mayernik MS. Metadata accounts: achieving data and evidence in scientific research. Soc Stud Sci 2019; 49: 732– 57. 10.1177/0306312719863494 31354073 PMC7323761 · DOI
  86. [116]Selbst AD, Boyd D, Friedler SA. Fairness and abstraction in Sociotechnical systems. in association for computing machinery. Proceedings of the Conference on Fairness, Accountability, and Transparency, New York, NY, USA, 2019: 59– 68.
  87. [117]Cury M, Whitworth E, Barfort S. Hybrid methodology: combining ethnography, cognitive science, and machine learning to inform the development of Context-Aware personal computing and assistive technology. Ethnographic Praxis in Industry Conference Proceedings, 2019: 254– 81.
  88. [118]Mitchell S, Potash E, Barocas S, et al. Algorithmic fairness: choices, assumptions, and definitions. Annu Rev Stat Appl 2021; 8: 141– 63. 10.1146/annurev-statistics-042720-125902 · DOI · Paper page
  89. [119]Trewin S, Basson S, Muller M, et al. Considerations for AI fairness for people with disabilities. AI Matters 2019; 5: 40– 63. 10.1145/3362077.3362086 · DOI · Paper page
  90. [121]Goldenstein J, Poschmann P. Analyzing meaning in big data: performing a MAP analysis using grammatical parsing and topic modeling. Sociol Methodol 2019; 49: 83– 131. 10.1177/0081175019852762 · DOI
  91. [122]Arnold T, Fuller HJA. In search of the user’s language: Natural language processing, computational ethnography, and error-tolerant interface design. In: Advances in Usability, User Experience and Assistive Technology. Cham: Springer International Publishing, 2019: 36– 43.
  92. [123]Abramson CM, Joslyn J, Rendle KA, et al. The promises of computational ethnography: improving transparency, replicability, and validity for realist approaches to ethnographic analysis. Ethnography 2018; 19: 254– 84. 10.1177/1466138117725340 · DOI
  93. [124]Moore RJ, Smith R, Liu Q. Using computational ethnography to enhance the curation of real-world data (RWD) for chronic pain and invisible disability use cases. SIGACCESS Access. Comput. 2020: 1– 7. 10.1145/3412836.3412840 · DOI
  94. [125]Leahey E. Overseeing research practice: the case of data editing. Sci Technol Human Values 2008; 33: 605– 30.
  95. [128]Blomberg SN, Christensen HC, Lippert F, et al. Effect of machine learning on Dispatcher recognition of out-of-hospital cardiac arrest during calls to emergency medical services: a randomized clinical trial. JAMA Netw Open 2021; 4: e2032320. 10.1001/jamanetworkopen.2020.32320 33404620 PMC7788469 · DOI
  96. [130]Emanuel EJ, Wachter RM. Artificial intelligence in health care: will the value match the hype? JAMA 2019; 321: 2281– 2. 10.1001/jama.2019.4914 31107500 · DOI · Paper page
  97. [131]Tomlinson A, Furukawa TA, Efthimiou O, et al. Personalise antidepressant treatment for unipolar depression combining individual choices, risks and big data (PETRUSHKA): rationale and protocol. Evid Based Ment Health 2020; 23: 52– 6. 10.1136/ebmental-2019-300118 31645364 PMC7229905 · DOI · Paper page
  98. [132]Thieme A, Belgrave D, Doherty G. Machine learning in mental health: a systematic review of the HCI literature to support the development of effective and Implementable ml systems. ACM Trans Comput-Hum Interact 2020; 27: 1– 53.
  99. [133]Bauer M, Monteith S, Geddes J, et al. Automation to optimise physician treatment of individual patients: examples in psychiatry. Lancet Psychiatry 2019; 6: 338– 49. 10.1016/S2215-0366(19)30041-0 30904127 · DOI · Paper page
  100. [135]Kemp J, Zhang T, Inglis F, et al. Delivery of compassionate mental health care in a digital Technology-Driven age: Scoping review. J Med Internet Res 2020; 22: e16263. 10.2196/16263 32141833 PMC7084292 · DOI
  101. [136]Chin-Yee B, Upshur R. Clinical judgement in the era of big data and predictive analytics. J Eval Clin Pract 2018; 24: 638– 45. 10.1111/jep.12852 29237237 · DOI · Paper page
  102. [137]Hunt LM, Bell HS, Baker AM, et al. Electronic health records and the disappearing patient. Med Anthropol Q 2017; 31: 403– 21. 10.1111/maq.12375 28370246 PMC6104392 · DOI
  103. [138]Topol EJ. High-Performance medicine: the convergence of human and artificial intelligence. Nat Med 2019; 25: 44– 56. 10.1038/s41591-018-0300-7 30617339 · DOI · Paper page
  104. [139]Glasziou P, Moynihan R, Richards T, et al. Too much medicine; too little care. BMJ 2013; 347: f4247. 10.1136/bmj.f4247 23820022 · DOI
  105. [140]McCradden MD, Stephenson EA, Anderson JA. Clinical research underlies ethical integration of healthcare artificial intelligence. Nat Med 2020; 26: 1325– 6. 10.1038/s41591-020-1035-9 32908273 · DOI · Paper page
  106. [141]All of Us Research Program Investigators, Denny JC, Rutter JL, et al. The "All of Us" Research Program. N Engl J Med 2019; 381: 668– 76. 10.1056/NEJMsr1809937 31412182 PMC8291101 · DOI · Paper page
  107. [142]Biruk C. Seeing like a research project: producing "high-quality data" in AIDS research in Malawi. Med Anthropol 2012; 31: 347– 66. 10.1080/01459740.2011.631960 22746683 · DOI
  108. [143]Ghassemi M, Naumann T, Schulam P, et al. Practical guidance on artificial intelligence for health-care data. Lancet Digit Health 2019; 1: e157– 9. 10.1016/S2589-7500(19)30084-6 33323184 · DOI
  109. [144]Fox K. The Illusion of Inclusion: Large Scale Genomic Data Sovereignty and Indigenous Populations. In:Association for Computing Machinery. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, New York, NY, USA, 2020: 3591.
  110. [146]Kukutai T, Taylor J O. Data sovereignty for Indigenous peoples: current practice and future needs. ANU Press, 2016.
  111. [147]Zakaria C, Balan R, Lee Y. StressMon: scalable detection of perceived stress and depression using passive sensing of changes in work Routines and group interactions. Proc ACM Hum-Comput Interact 2019; 3: 1– 29.34322658
  112. [148]Vincent N, Li H, Tilly N. Data Leverage: A Framework for Empowering the Public in its Relationship with Technology Companies. In:Association for Computing Machinery. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, 2021: 215– 27.
  113. [149]Pinto AD, Glattstein-Young G, Mohamed A, et al. Building a Foundation to reduce health inequities: routine collection of sociodemographic data in primary care. J Am Board Fam Med 2016; 29: 348– 55. 10.3122/jabfm.2016.03.150280 27170792 · DOI
  114. [150]Mc Kenzie K, Antwi BAM, Tuck A. The case for diversity, 2016. Available: https://pdfs.semanticscholar.org/ebcd/d11cd033afc93b389e8a28c331155869d0a7.pdf [Accessed 23 Mar 2021].
  115. [151]Kiran T, Sandhu P, Aratangy T, et al. Patient perspectives on routinely being asked about their race and ethnicity: qualitative study in primary care. Can Fam Physician 2019; 65: e363– 9. 31413042 PMC6693598
  116. [153]Cristina Mora G. Making Hispanics. University of Chicago Press, 2021.
  117. [154]Holmes JH, Beinlich J, Boland MR, et al. Why is the electronic health record so challenging for research and clinical care? Methods Inf Med 2021; 60: 032– 48. 10.1055/s-0041-1731784 PMC9295893 34282602 · DOI
  118. [155]Thompson D. The schematic state. Cambridge University Press, 2016.
  119. [156]Saini A. Superior: the return of race science. Beacon Press, 2019.
  120. [157]Treviranus J. The Value of Being Different. In: Proceedings of the 16th International Web for All Conference. New York, NY, USA:: Association for Computing Machinery 2019: 1– 7.
  121. [159]He H, Garcia EA. Learning from imbalanced data. IEEE Trans Knowl Data Eng 2009; 21: 1263– 84.
  122. [160]Willen SS. How is health-related "deservingness" reckoned? Perspectives from unauthorized im/migrants in Tel Aviv. Soc Sci Med 2012; 74: 812– 21. 10.1016/j.socscimed.2011.06.033 21821324 · DOI
  123. [161]Tomasev N, McKee KR, Kay J. Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities. arXiv [cs.CY], 2021. Available: http://arxiv.org/abs/2102.04257 · arXiv
  124. [164]Seng JS, Lopez WD, Sperlich M, et al. Marginalized identities, discrimination burden, and mental health: empirical exploration of an interpersonal-level approach to modeling intersectionality. Soc Sci Med 2012; 75: 2437– 45. 10.1016/j.socscimed.2012.09.023 23089613 PMC3962770 · DOI
  125. [165]Rosenfield S. Triple jeopardy? mental health at the intersection of gender, race, and class. Soc Sci Med 2012; 74: 1791– 801. 10.1016/j.socscimed.2011.11.010 22326306 · DOI
  126. [166]Buolamwini J, Gebru T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In: Proceedings of the 1st conference on fairness, accountability and transparency. New York, NY, USA: PMLR, 2018: 77– 91.
  127. [168]Koppe G, Meyer-Lindenberg A, Durstewitz D. Deep learning for small and big data in psychiatry. Neuropsychopharmacology 2021; 46: 176– 90. 10.1038/s41386-020-0767-z 32668442 PMC7689428 · DOI · Paper page
  128. [170]Kleinman A. From illness as culture to caregiving as moral experience. N Engl J Med 2013; 368: 1376– 7. 10.1056/NEJMp1300678 23574117 · DOI
  129. [173]Topol E. Others. The Topol Review. In: Preparing the healthcare workforce to deliver the digital future, 2019: 1– 48.
  130. [174]Cuperfain AB, Hui K, Berkhout SG, et al. Patient, family and provider views of measurement-based care in an early-psychosis intervention programme. BJPsych Open 2021; 7. 10.1192/bjo.2021.1005 · DOI
  131. [175]Dack C, Ross J, Papadopoulos C, et al. A review and meta-analysis of the patient factors associated with psychiatric in-patient aggression. Acta Psychiatr Scand 2013; 127: 255– 68. 10.1111/acps.12053 23289890 · DOI
  132. [176]Menger V, Spruit M, van Est R, et al. Machine learning approach to inpatient violence risk assessment using routinely collected clinical notes in electronic health records. JAMA Netw Open 2019; 2: e196709. 10.1001/jamanetworkopen.2019.6709 31268542 PMC6613290 · DOI
  133. [177]Abebe R, Barocas S, Kleinberg J. Roles for computing in social change. arXiv [cs.CY] 2019.
  134. [178]Khan SS, Ye B, Taati B, et al. Detecting agitation and aggression in people with dementia using sensors-A systematic review. Alzheimers Dement 2018; 14: 824– 32. 10.1016/j.jalz.2018.02.004 29571749 · DOI · Paper page
  135. [179]Appelbaum PS, Robbins PC, Monahan J. Violence and delusions: data from the MacArthur violence risk assessment study. Am J Psychiatry 2000; 157: 566– 72. 10.1176/appi.ajp.157.4.566 10739415 · DOI
  136. [180]Meerai S, Abdillahi I, Poole J. An introduction to anti-black Sanism Intersectionalities: a global Journal of social work analysis, research, Polity, and practice 2016; 5: 18– 35.
  137. [181]Olbert CM, Nagendra A, Buck B. Meta-Analysis of black vs. white racial disparity in schizophrenia diagnosis in the United States: do structured assessments attenuate racial disparities? J Abnorm Psychol 2018; 127: 104– 15. 10.1037/abn0000309 29094963 · DOI
  138. [182]Kohler-Hausmann I. Eddie Murphy and the dangers of counterfactual causal thinking about detecting racial discrimination. Nw UL Rev 2018; 113: 1163.
  139. [183]Papadopoulos C, Ross J, Stewart D, et al. The antecedents of violence and aggression within psychiatric in-patient settings. Acta Psychiatr Scand 2012; 125: 425– 39. 10.1111/j.1600-0447.2012.01827.x 22268678 · DOI
  140. [184]Matthews M, Bell E. Assessment of risk of violent offending for adults with intellectual disability and/or autism spectrum disorder. In: The Wiley Handbook of what works in violence risk management, 2020: 349– 66.
  141. [185]Gerson R, Malas N, Mroczkowski MM. Crisis in the emergency department: the evaluation and management of acute agitation in children and adolescents. Child Adolesc Psychiatr Clin N Am 2018; 27: 367– 86. 10.1016/j.chc.2018.02.002 29933788 · DOI
  142. [186]Fernandes NA, Sawyer A, Zaheer J, et al. Adults with intellectual and developmental disabilities presenting to a psychiatric emergency department: a descriptive analysis and predictors of admission. J Ment Health Res Intellect Disabil 2020; 13: 384– 95. 10.1080/19315864.2020.1822963 · DOI
  143. [187]Valenti E, Giacco D, Katasakou C, et al. Which values are important for patients during involuntary treatment? A qualitative study with psychiatric inpatients. J Med Ethics 2014; 40: 832– 6. 10.1136/medethics-2011-100370 24129367 · DOI · Paper page
  144. [188]McGuinness D, Murphy K, Bainbridge E, et al. Individuals' experiences of involuntary admissions and preserving control: qualitative study. BJPsych Open 2018; 4: 501– 9. 10.1192/bjo.2018.59 30564446 PMC6293449 · DOI
  145. [189]Hui K, Cooper RB, Zaheer J. Engaging patients and families in the ethics of involuntary psychiatric care. Am J Bioeth 2020; 20: 82– 4. 10.1080/15265161.2020.1754511 32441608 · DOI
  146. [190]Care HKR. closeness, and becoming ‘better’: Transformation and therapeutic process in American adolescent psychiatric custody. Ethos 2016; 44: 313– 32.

Cited by

Sorted by citations

Authors
Sikstrom, L., Maslej, M. M., Hui, K., Findlay, Z., Buchman, D. Z., & Hill, S. L.
Affiliations
  • Centre for Addiction and Mental Health (CA)
  • University of Toronto (CA)
Corresponding author
Sikstrom, Laura
ORCID
Venue
BMJ Health & Care Informatics
Publisher
BMJ
Published
2022-01
Volume
29
Issue
1
Pages
e100459
ISSN
2632-1009 (ISSN-L)
DOI
10.1136/bmjhci-2021-100459
PMC
PMC8753410
OpenAlex
W4206398356
Cited by
76
References
25
Citations by year
2026: 16 · 2025: 24 · 2024: 13 · 2023: 20 · 2022: 3
FWCI
2.86
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Health & Care Informatics (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Social Sciences and Humanities Research Council of Canada (430-2021-0116)
  • Dalla Lana School of Public Health Interdisciplinary Data Science Seed Funding
  • Canadian Institutes of Health Research
  • AMS Fellowship in Compassion and Artificial Intelligence
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
fairness, algorithmic fairness, health equity, clinical algorithms, artificial intelligence in healthcare, transparency, impartiality, inclusion, conceptual framework, risk assessment, algorithmic discrimination, stakeholder engagement
MeSH terms
Machine Learning, Health Equity, Artificial Intelligence, Delivery of Health Care, Humans, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review