Seeing through health information technology: The need for transparency in software, algorithms, data privacy, and regulation

Abstract

Big data and artificial intelligence (AI) algorithmic decision-making are becoming more common in healthcare as they are in other sectors. Improvements in both data protection and access, as well as in algorithms, for purposes as different as predictive policing, assessing government performance, democratic deliberation, scientific research, news reporting, and healthcare provision are hoped for through increased data collection and access. At the same time, concerns abound about how data are generated, collected, processed, and used; and how algorithms are trained, how and if they work, and whether they actually improve care. Calls for transparency in algorithms, data, and data use have been increasing, in part, in the hopes of increasing effectiveness, accountability, and privacy. Opaqueness can thwart the promise technologies can help achieve. Rather than considering transparency an unmitigated good, or opaqueness as necessarily bad, circumstances surrounding the particulars should be taken into account so that wise decisions may be made regarding how much transparency is advisable for what purpose, and how best to achieve it. This paper sketches transparency issues in the context of health information technologies such as AI, robotics, telemedicine (including telehealth, and smart phone apps), electronic health records (EHRs), and health-related data. Concerns that surround healthcare, health-related commercial information technologies, and social media that collect user-generated data can illuminate more general information technology transparency considerations. I start with software, where general calls for algorithmic transparency are common. These healthcare information technologies are based on complex software systems comprised of numerous algorithms that are not well understood. I echo those calls for transparency, as well as indicate additional software-related transparency issues, such as those stemming from intellectual property (IP) and liability. Concerns surrounding algorithmic transparency in general, or rather its lack, apply, then, to healthcare software. Then I move to privacy and health data, emphasizing areas where privacy is compromised through lack of transparency. Throughout, I allude to contractual and regulatory issues that contribute to the lack of transparency and point toward the need for change. This survey is unusual in bringing together different aspects of the need for transparency from multiple disciplines, and for using health information technologies as the lens to do so. It draws on literature in law, medical informatics, ethics, and other domains. It suggests more generally the need for transparency in how information technologies and data are used and some ways toward achieving that. It takes an interdisciplinary approach that can lead to multifaceted recommendations for addressing these issues more effectively. Much software is protected as IP. Trade secrets or vague patents make it hard to know what exactly it does, how it does it, what data were used to create and test it, and the quality of that data as well as of software validation procedures and of output results. Even if system architecture, software, procedures, and data were disclosed, an algorithm in the software could be so complex that it is difficult even for experienced software gurus to comprehend, and nearly impossible to be assured it functions correctly. In addition to software directly coded by people, algorithms that are themselves created by software, developed through machine learning or genetic programming or other AI approaches result in software impossible to predict, and even if revealed, nearly impossible to understand. If an algorithm is ‘trained’ by humans, the nature and quality of that training, and trainers, also may be unknown. Further, it also can be very difficult to assess the underlying data.1 There are many ways, then, that it is difficult to know how and why an algorithm gives the results it does, resulting in a black box that leads to informational asymmetries so that how it was developed, trained, or works, or gives the results it does is not known to those using or affected by it. That makes assessing the algorithm and associated ethical considerations, risks, and responsibilities challenging.2 Transparency, explainability, interpretability, testability, understandability, identifiability of inherent biases or outright errors, correctability, accountability all are compromised, yet all are considered fundamental to accepting algorithmic recommendations and to human rights (e.g., the EU’s General Data Protection Regulation (GDPR).3). Lack of transparency affects not only AI algorithms, but also complex systems, such as EHR systems for ordering, communicating, storing, and retrieving clinical information. For example, clinical record data entry and record keeping includes ordering a patient’s medications. This can involve automated dosage calculations based, in part, on the p