Learning from the Failure of Autonomous and Intelligent Systems: Accidents, Safety, and Sociotechnical Sources of Risk
Open accessAbstract
Efforts to develop autonomous and intelligent systems (AIS) have exploded across a range of settings in recent years, from self‐driving cars to medical diagnostic chatbots. These have the potential to bring enormous benefits to society but also have the potential to introduce new—or amplify existing—risks. As these emerging technologies become more widespread, one of the most critical risk management challenges is to ensure that failures of AIS can be rigorously analyzed and understood so that the safety of these systems can be effectively governed and improved. AIS are necessarily developed and deployed within complex human, social, and organizational systems, but to date there has been little systematic examination of the sociotechnical sources of risk and failure in AIS. Accordingly, this article develops a conceptual framework that characterizes key sociotechnical sources of risk in AIS by reanalyzing one of the most publicly reported failures to date: the 2018 fatal crash of Uber's self‐driving car. Publicly available investigative reports were systematically analyzed using constant comparative analysis to identify key sources and patterns of sociotechnical risk. Five fundamental domains of sociotechnical risk were conceptualized—structural, organizational, technological, epistemic, and cultural—each indicated by particular patterns of sociotechnical failure. The resulting SOTEC framework of sociotechnical risk in AIS extends existing theories of risk in complex systems and highlights important practical and theoretical implications for managing risk and developing infrastructures of learning in AIS.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In 2020, an AI-powered self-driving car was involved in a fatal accident because it was unable to understand the context of the situation and collided with a pedestrian.
“In 2020, an AI-powered selfdriving car was involved in a fatal accident.The car was unable to understand the context of the situation and it collided with a pedestrian [100].”
From Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond (2024) · cites this paper as [100]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1998) · Sage eBooks · Cited by 38,631 · see citing works
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789 · see citing works
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
Building Theories from Case Study Research
Eisenhardt, K. M. (1989) · The Academy of Management Review · doi:10.2307/258557 · Cited by 17,455 · see citing works
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · see citing works
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · see citing works
Reason, J. (1990) · doi:10.1017/cbo9781139062367 · Cited by 5,108 · see citing works
Managing the Risks of Organizational Accidents
REASON, J. T. (2016) · Routledge eBooks · doi:10.4324/9781315543543 · Cited by 4,461 · see citing works
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 2× · see citing works
“…social dynamics that promote the differential distribution of ignorance—such as professional boundaries or organizational silos—or that purposefully produce secrecy, such as the withholding of confidential information (Burrell, 2016; Costas & Grey, 2016; McGoey, 2019; Turner, 1978; Vaughan, 1996).”
BackgroundRisk management in a dynamic society: a modelling problem
Rasmussen, J. (1997) · Safety Science · doi:10.1016/s0925-7535(97)00052-0 · Cited by 2,994 · see citing works
Normal Accidents: Living with High-Risk Technologies
Cummings, L. L., & Perrow, C. (1984) · Administrative Science Quarterly · doi:10.2307/2392945 · Cited by 2,861 · see citing works
Organizational Silence: A Barrier to Change and Development in a Pluralistic World
Morrison, E. W., & Milliken, F. J. (2000) · The Academy of Management Review · doi:10.2307/259200 · Cited by 2,776 · see citing works
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639 · see citing works
A new accident model for engineering safer systems
Leveson, N. G. (2003) · Safety Science · doi:10.1016/s0925-7535(03)00047-x · Cited by 2,262 · see citing works
Grounded Theory in Management Research
Locke, K. (2003) · SAGE Publications Ltd eBooks · doi:10.4135/9780857024428 · Cited by 2,238 · see citing works
Bainbridge, L. (1983) · Automatica · doi:10.1016/0005-1098(83)90046-8 · Cited by 2,205 · see citing works
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167 · see citing works
(2013) · Cited by 1,951 · see citing works
Managing the Unexpected: Assuring High Performance in an Age of Complexity
(2002) · International Journal of Productivity and Performance Management · doi:10.1108/ws.2002.07951dae.003 · Cited by 1,927 · see citing works
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710 · see citing works
Improving Pregnancy Outcome during Imprisonment: A Model Residential Care Program
Siefert, K., & Pimlott, S. (2001) · Social Work · doi:10.1093/sw/46.2.125 · Cited by 1,647 · see citing works
Engineering a Safer World: Systems Thinking Applied to Safety
(2012) · Choice Reviews Online · doi:10.5860/choice.49-6305 · Cited by 1,566 · see citing works
What Theory is Not, Theorizing Is
Weick, K. E. (1995) · Administrative Science Quarterly · doi:10.2307/2393789 · Cited by 1,365 · see citing works
Organizational Culture as a Source of High Reliability
Weick, K. E. (1987) · California Management Review · doi:10.2307/41165243 · Cited by 1,358 · see citing works
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314 · see citing works
An investigation of the Therac-25 accidents
Leveson, N. G., & Turner, C. S. (1993) · Computer · doi:10.1109/mc.1993.274940 · Cited by 1,245 · see citing works
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · cited 1× · see citing works
“However, the focus of much work to date has primarily been on defining broad ethical principles (Jobin, Ienca, & Vayena, 2019; IEEE, 2019), exploring issues of accountability and liability (Morley et al., 2020; Pöllänen, Read, Lane, Thompson, & Salmon, 2020), and addressing technical aspects of assurance (Brundage, Av…”
BackgroundA clinically applicable approach to continuous prediction of future acute kidney injury
Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., et al. (2019) · Nature · doi:10.1038/s41586-019-1390-1 · Cited by 1,086 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024 · see citing works
The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA.
Sagan, S. D., & Vaughan, D. (1997) · Administrative Science Quarterly · doi:10.2307/2393926 · Cited by 937 · see citing works
The Organizational and Interorganizational Development of Disasters
Turner, B. A. (1976) · Administrative Science Quarterly · doi:10.2307/2391850 · Cited by 876 · cited 3× · see citing works
“…to explain the social, organizational, and technological processes involved in the failure of complex systems, encompassing organizational structure (Perrow, 1984), human activity (Reason, 1990), technological control (Leveson, 2011), and cultural incubation (Turner, 1976; Weick & Sutcliffe, 2003).”
BackgroundMateriality and change: Challenges to building better theory about technology and organizing
Leonardi, P. M., & Barley, S. R. (2008) · Information and Organization · doi:10.1016/j.infoandorg.2008.03.001 · Cited by 776 · see citing works
(2019) · Cited by 771 · see citing works
Glaser, B. G. (2015) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 749 · see citing works
Leonardi, P. M., & Barley, S. R. (2010) · Academy of Management Annals · doi:10.5465/19416521003654160 · Cited by 733 · see citing works
Resilience Engineering in Practice a Guidebook
Hollnagel, E., Pariès, J., Woods David, D., & Wreathall, J. (2010) · HAL · Cited by 671 · see citing works
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662 · see citing works
NORMAL ACCIDENTS: LIVING WITH HIGH-RISK TECHNOLOGIES
Perrow, C. (1984) · International Linear Collider · Cited by 604 · see citing works
Scoping review on the use of socially assistive robot technology in elderly care
Abdi, J., Al-Hindawi, A., Ng, T., & Vizcaychipi, M. P. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-018815 · Cited by 583 · see citing works
There is a blind spot in AI research
Crawford, K., & Calo, R. (2016) · Nature · doi:10.1038/538311a · Cited by 580 · see citing works
How the Machine 'Thinks:' Understanding Opacity in Machine Learning AlgorithmsPreprint
Burrell, J. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2660674 · Cited by 511 · see citing works
Ethical governance is essential to building trust in robotics and artificial intelligence systems
Winfield, A. F. T., & Jirotka, M. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0085 · Cited by 493 · see citing works
Smithson, M. J. (1989) · Cognitive science · doi:10.1007/978-1-4612-3628-3 · Cited by 478 · see citing works
Ignorance and Uncertainty: Emerging Paradigms
Smithson, M. J. (2012) · Medical Entomology and Zoology · Cited by 476 · see citing works
Man-made disasters: why technology and organizations (sometimes) fail
Pidgeon, N., & O'Leary, M. (2000) · Safety Science · doi:10.1016/s0925-7535(00)00004-7 · Cited by 473 · see citing works
The Negotiated Order of Organizational Reliability
Schulman, P. R. (1993) · Administration & Society · doi:10.1177/009539979302500305 · Cited by 446 · see citing works
Turner, B. A. (1981) · Quality & Quantity · doi:10.1007/bf00164639 · Cited by 445 · see citing works
Handbook of human factors and ergonomics
Stanton, N. A. (2021) · Ergonomics · doi:10.1080/00140139.2021.2005272 · Cited by 426 · see citing works
Drift into Failure: From Hunting Broken Components to Understanding Complex Systems
Dekker, S. W. A. (2011) · Cited by 414 · see citing works
The problem with incident reporting
Macrae, C. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004732 · Cited by 389 · see citing works
Self-citationMachine learning, social learning and the governance of self-driving cars
Stilgoe, J. (2018) · Social Studies of Science · doi:10.1177/0306312717741687 · Cited by 375 · see citing works
The next catastrophe: reducing our vulnerabilities to natural, industrial, and terrorist disasters
(2007) · Choice Reviews Online · doi:10.5860/choice.45-2108 · Cited by 369 · see citing works
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000375277 · Cited by 360 · see citing works
Friendly fire: the accidental shootdown of U.S. Black Hawks over Northern Iraq
(2000) · Choice Reviews Online · doi:10.5860/choice.38-1151 · Cited by 323 · see citing works
Moral crumple zones: cautionary tales in human–robot interaction
Elish, M. C. (2019) · Engaging Science, Technology, and Society · doi:10.17351/ests2019.260 · Cited by 321 · cited 1× · see citing works
“There has been little systematic consideration of the complex sociotechnical sources of risk and failure in AIS, or the organizational infrastructures required to learn from those failures (Bryson & Winfield, 2017; Elish, 2019; Stilgoe, 2018; Winfield & Jirotka, 2017).”
Backgroundon behalf of the RiH-team, Wiig, S., Aase, K., Billett, S. R., Canfield, C., Røise, O., Njå, O., Guise, V., Haraldseid-Driftland, C., Ree, E., et al. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-05224-3 · Cited by 308 · see citing works
Self-citationToward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · see citing works
Standardizing Ethical Design for Artificial Intelligence and Autonomous Systems
Bryson, J. J., & Winfield, A. F. T. (2017) · Computer · doi:10.1109/mc.2017.154 · Cited by 298 · see citing works
Hospitals as Cultures of Entrapment: A Re-Analysis of the Bristol Royal Infirmary
Weick, K. E., & Sutcliffe, K. M. (2003) · California Management Review · doi:10.2307/41166166 · Cited by 287 · see citing works
Snook, S. A. (2000) · Princeton University Press eBooks · doi:10.1515/9781400840977 · Cited by 280 · see citing works
Human error and the problem of causality in analysis of accidents
Rasmussen, J. (1990) · Philosophical transactions of the Royal Society of London. Series B, Biological sciences · doi:10.1098/rstb.1990.0088 · Cited by 259 · see citing works
(2019) · doi:10.5040/9781350225725 · Cited by 257 · see citing works
Safety paradoxes and safety culture
REASON, J. T. (2000) · Injury Control and Safety Promotion · doi:10.1076/1566-0974(200003)7:1;1-v;ft003 · Cited by 255 · see citing works
THE USE OF GROUNDED THEORY FOR THE QUALITATIVE ANALYSIS OF ORGANIZATIONAL BEHAVIOUR
Turner, B. A. (1983) · Journal of Management Studies · doi:10.1111/j.1467-6486.1983.tb00211.x · Cited by 232 · see citing works
The Seven Deadly Myths of "Autonomous Systems"
Bradshaw, J. M., Hoffman, R. R., Johnson, M., & Woods, D. D. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.70 · Cited by 218 · see citing works
Beyond Asimov: The Three Laws of Responsible Robotics
Murphy, R., & Woods, D. D. (2009) · IEEE Intelligent Systems · doi:10.1109/mis.2009.69 · Cited by 193 · see citing works
Roe, E. M., & Schulman, P. R. (2008) · Stanford University Press eBooks · doi:10.1515/9781503627123 · Cited by 190 · see citing works
Safety of patient-facing digital symptom checkers
Fraser, H., Coiera, E. W., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 182 · see citing works
Accident and Design: Contemporary Debates in Risk Management
Adams, J., Hood, C. P., & Jones, D. K. C. (1997) · Geographical Journal · doi:10.2307/3059768 · Cited by 172 · see citing works
A Berkeley View of Systems Challenges for AI
Stoica, I., Song, D., Popa, R. A., Patterson, D., Mahoney, M. W., Katz, R., Joseph, A. D., Jordan, M., Hellerstein, J. M., Gonzalez, J. E., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05855 · Cited by 171 · see citing works
Imposing limits on autonomous systems
Hancock, P. A. (2017) · Ergonomics · doi:10.1080/00140139.2016.1190035 · Cited by 162 · see citing works
Preventing repeated real world AI failures by cataloging incidents: the AI incident database
McGregor, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17817 · Cited by 156 · see citing works
What Difference Does a Robot Make? The Material Enactment of Distributed Coordination
Beane, M., & Orlikowski, W. J. (2015) · Organization Science · doi:10.1287/orsc.2015.1004 · Cited by 144 · see citing works
Causes of Disaster: Sloppy Management
Turner, B. A. (1994) · British Journal of Management · doi:10.1111/j.1467-8551.1994.tb00172.x · Cited by 143 · see citing works
Stanton, N. A., Salmon, P. M., Walker, G. H., et al. (2019) · Safety Science · doi:10.1016/j.ssci.2019.06.008 · Cited by 143 · see citing works
Drift into Failure, From Hunting Broken Components to Understanding Complex Systems
McCarthy, G. W. (2012) · Aviation Space and Environmental Medicine · doi:10.3357/asem.3152.2012 · Cited by 135 · see citing works
Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)Preprint
Hawkins, R., Paterson, C., Picardi, C., Jia, Y., Calinescu, R., & Habli, I. (2021) · arXiv.org · doi:10.48550/arxiv.2102.01564 · Cited by 134 · see citing works
‘Remixing Rasmussen’: The evolution of Accimaps within systemic accident analysis
Waterson, P. E., Jenkins, D. P., Salmon, P. M., & Underwood, P. (2016) · Applied Ergonomics · doi:10.1016/j.apergo.2016.09.004 · Cited by 133 · see citing works
Kroll, J. A. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0084 · Cited by 132 · see citing works
Autonomous buses: Intentions to use, passenger experiences, and suggestions for improvement
Mouratidis, K., & Serrano, V. C. (2021) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2020.12.007 · Cited by 129 · see citing works
Governing the safety of artificial intelligence in healthcare
Macrae, C. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2019-009484 · Cited by 123 · see citing works
Self-citationPerrow, C. B. (2011) · Princeton University Press eBooks · doi:10.1515/9781400838516 · Cited by 115 · see citing works
Svenson, O. (1991) · Risk Analysis · doi:10.1111/j.1539-6924.1991.tb00635.x · Cited by 111 · see citing works
The Coming of Age of Risk Governance
Klinke, A., & Renn, O. (2019) · Risk Analysis · doi:10.1111/risa.13383 · Cited by 107 · see citing works
Secrecy at Work: The Hidden Architecture of Organizational Life
Grey, C., & Costas, J. (2016) · Cited by 103 · see citing works
The Case for an Ethical Black Box
Winfield, A. F. T., & Jirotka, M. D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-64107-2_21 · Cited by 95 · see citing works
Early warnings, weak signals and learning from healthcare disasters
Macrae, C. (2014) · BMJ Quality & Safety · doi:10.1136/bmjqs-2013-002685 · Cited by 93 · see citing works
Self-citationWinfield, A. F. T. (2012) · Oxford University Press eBooks · doi:10.1093/actrade/9780199695980.001.0001 · Cited by 92 · see citing works
Dekker, S. W. A. (2016) · CRC Press eBooks · doi:10.4324/9781315251271 · Cited by 90 · see citing works
“737-Cabriolet”: The Limits of Knowledge and the Sociology of Inevitable Failure
Downer, J. (2011) · American Journal of Sociology · doi:10.1086/662383 · Cited by 89 · see citing works
Autonowashing: The Greenwashing of Vehicle Automation
Dixon, L. (2020) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2020.100113 · Cited by 86 · see citing works
McCray, L. E., Oye, K., & Petersen, A. (2010) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2009.12.001 · Cited by 84 · see citing works
Learning from failure: the need for independent safety investigation in healthcare
Macrae, C., & Vincent, C. A. (2014) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814555939 · Cited by 81 · see citing works
Self-citationWho’s Driving Innovation?: New Technologies and the Collaborative State
Stilgoe, J. (2020) · doi:10.1007/978-3-030-32320-2 · Cited by 80 · see citing works
Barley, S. R. (2020) · doi:10.1093/oso/9780198795209.001.0001 · Cited by 78 · see citing works
Normal Accident Theory As Frame, Link, and Provocation
Weick, K. E. (2004) · Organization & Environment · doi:10.1177/1086026603262031 · Cited by 76 · see citing works
Rethinking the Maturity of Artificial Intelligence in Safety‐Critical Settings
Cummings, M. L. (2021) · AI Magazine · doi:10.1002/j.2371-9621.2021.tb00005.x · Cited by 76 · see citing works
Adaptive governance for the Internet of Things: Coping with emerging security risks
Brass, I., & Sowell, J. H. (2021) · Regulation & Governance · doi:10.1111/rego.12343 · Cited by 71 · see citing works
Managing Major Hazards: The Lessons of the Moura Mine Disaster
Tucker, E., & Hopkins, A. (2000) · Industrial and Labor Relations Review · doi:10.2307/2696147 · Cited by 70 · see citing works
Safety, Culture and Risk The Organisational Causes of Disasters
Hopkins, A. (2008) · Cited by 70 · see citing works
Macrae, C. (2014) · Palgrave Macmillan UK eBooks · doi:10.1057/9781137376121 · Cited by 65 · see citing works
Self-citationHopkins, A. (1999) · Safety Science · doi:10.1016/s0925-7535(99)00015-6 · Cited by 57 · see citing works
Pöllänen, E., Read, G. J. M., Lane, B. R., Thompson, J., & Salmon, P. M. (2020) · Ergonomics · doi:10.1080/00140139.2020.1744064 · Cited by 57 · see citing works
Direct-to-consumer medical machine learning and artificial intelligence applications
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00331-0 · Cited by 57 · see citing works
Safety, Culture and Risk. the Organisational Causes of Disasters
Hocking, B. (2007) · Health Sociology Review · Cited by 55 · see citing works
Moments of Resilience: Time, Space and the Organisation of Safety in Complex Sociotechnical Systems
Macrae, C. (2018) · SpringerBriefs in applied sciences and technology · doi:10.1007/978-3-030-03189-3_3 · Cited by 45 · see citing works
Self-citationIntroduction: Toward a Multidisciplinary Science of Human-Computer Interaction
Carroll, J. M. (2003) · Elsevier eBooks · doi:10.1016/b978-155860808-5/50001-0 · Cited by 43 · see citing works
Robot Accident Investigation: A Case Study in Responsible Robotics
Winfield, A. F. T., Winkle, K., Webb, H., Lyngs, U., Jirotka, M. D., & Macrae, C. (2020) · Springer eBooks · doi:10.1007/978-3-030-66494-7_6 · Cited by 40 · see citing works
Self-citationRobot Transparency: Improving Understanding of Intelligent Behaviour for Designers and Users
Wortham, R. H., Theodorou, A. A., & Bryson, J. J. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-64107-2_22 · Cited by 37 · see citing works
Salmon, P. M., Carden, T., & Hancock, P. A. (2020) · Human Factors and Ergonomics in Manufacturing & Service Industries · doi:10.1002/hfm.20883 · Cited by 36 · see citing works
Making risks visible: Identifying and interpreting threats to airline flight safety
Macrae, C. (2008) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317908x314045 · Cited by 35 · see citing works
Self-citationGephart, R. P. (2004) · Organization & Environment · doi:10.1177/1086026603262030 · Cited by 34 · see citing works
Architects of intelligence the truth about AI from the people building It
Ford, M. (2018) · Cited by 33 · see citing works
When failure is an option: redundancy, reliability and regulation in complex technical systems
Downer, J. (2009) · London School of Economics and Political Science Theses Online (London School of Economics and Political Science) · Cited by 32 · see citing works
Hulme, A., Stanton, N. A., Walker, G. H., Waterson, P. E., & Salmon, P. M. (2020) · Ergonomics · doi:10.1080/00140139.2020.1869321 · Cited by 31 · see citing works
The Limits of Safety: Organizations, Accidents, and Nuclear Weapons
Sagan, S. D. 1. (1993) · Project Muse (Johns Hopkins University) · Cited by 30 · see citing works
Michaelides-Mateou, S., & Mateou, A. (2010) · Cited by 27 · see citing works
THE SOCIAL AETIOLOGY OF DISASTERS
Turner, B. A. (1979) · Disasters · doi:10.1111/j.1467-7717.1979.tb00198.x · Cited by 27 · see citing works
Smithson, M. J. (1990) · International Journal of Mass Emergencies & Disasters · doi:10.1177/028072709000800302 · Cited by 27 · see citing works
Who's Driving Innovation?: New Technologies and the Collaborative State
Stilgoe, J. (2019) · UCL Discovery (University College London) · Cited by 27 · see citing works
Organizational Risk: “Muddling Through” 40 Years of Research
Gould, K. P. (2020) · Risk Analysis · doi:10.1111/risa.13460 · Cited by 26 · see citing works
The Limits of Safety: Organizations, Accidents, and Nuclear Weapons.
Rousseau, D. M., & Sagan, S. D. (1996) · Administrative Science Quarterly · doi:10.2307/2394001 · Cited by 25 · see citing works
Row over Babylon’s chatbot shows lack of regulation
Iacobucci, G. (2020) · BMJ · doi:10.1136/bmj.m815 · Cited by 21 · see citing works
Erensal, Y. C., & Albayrak, Y. E. (2006) · Human Factors and Ergonomics in Manufacturing & Service Industries · doi:10.1002/hfm.20063 · Cited by 16 · see citing works
A new national safety investigator for healthcare: the road ahead
Macrae, C., & Vincent, C. A. (2017) · Journal of the Royal Society of Medicine · doi:10.1177/0141076817694577 · Cited by 16 · see citing works
Self-citationOn Ignorance and Apocalypse: A Brief Introduction to ‘Epistemic Accidents’
Downer, J. (2019) · CRC Press eBooks · doi:10.4324/9781351190237-5 · Cited by 15 · see citing works
Secrecy at work the hidden architecture of organizational life
Costas, J. 1., & Grey, C. 1. (n.d.) · Cited by 14 · see citing works
Autonomous vehicles and human factors/ergonomics – A challenge but not a threat
Waterson, P. E. (2019) · Ergonomics · doi:10.1080/00140139.2019.1563335 · Cited by 9 · see citing works
Waterson, P. E. (2019) · Applied Ergonomics · doi:10.1016/j.apergo.2019.103011 · Cited by 7 · see citing works
Prologue: The “Brave New World” of Social Sciences in Interdisciplinary Risk Research
Klinke, A., Renn, O., Goble, R. L., et al. (2021) · Risk Analysis · doi:10.1111/risa.13716 · Cited by 5 · see citing works
Investigating for Improvement: Building a National Safety Investigator for Healthcare
Macrae, C., & Vincent, C. A. (2017) · Cited by 4 · see citing works
Self-citationDriven the race to create the autonomous car
Davies, A. (n.d.) · Cited by 2 · see citing works
Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident DatabasePreprint
McGregor, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.08512 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 12
- Braithwaite, J. (2011). The essence of responsive regulation. UBC Law Review, 44(3), 475-520.
- Edmondson, A. C. (2018). The fearless organisation: Creating psy- chologiical safety in the worksplace for learning, innovation and growth. London, UK: Wiley.
- EU. (2010). Regulation No 996/2010 of the European Parliament and of the Council of 20 October 2010 on the investigation and prevention of accidents and incidents in civil aviation and re- pealing Directive 94/56/EC. Official Journal of the European Union, 12.11.2010.
- Hart, C., Dombroff, M. A., & Tochen, D. K. (2018). Independent review of the safety culture of Uber Technologies, Inc's Ad- vanced Technologies Group. Final report. Richmond, Virginia:
- Kletz, T. (1994). Learning from accidents. Oxford, UK: Butterworth-Heinneman.
- Lehr, D., & Ohm, P. (2017). Playing with the data: What legal scholars should learn about machine learning. UC Davis Law Review, 51, 653-717.
- Macrae, C. (2014a). Close calls: managing risk and resilience in air- lline flight safety. London, UK: Palgrave Macmillan.
- O'Leary, M., Macrae, C., & Pidgeon, N. (2002). Safety data collec- tion in British Airways flight operations. In C. Johnson, (Ed.), Proceedings of a workshop on the investigation and reporting of incidents and accidents (pp. 89-98). Glasgow, UK: University of Glasgow.
- Reason, J. (1990). Human error. Cambridge, UK: Cambridge Uni- versity Press.
- Turner, B., & Pidgeon, N. (1997). Man-made disasters (2nd ed). Oxford, UK: Butterworth-Heinemann.
- Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (1999). Organiz- ing for high reliability: Processes of collective mindfulness. In R.S. Sutton & B.M. Shaw, (Eds.), Research in organizational be- haviour (Vol. 1, pp. 81-123). Stanford, CA: JAI Press.
- Winfield, A.F.T. (2012). Robotics: A very short introduction. Ox- ford, UK: Oxford University Press.
Cited by
Sorted by citations
Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond
Nasir, S., Khan, R. A., & Bai, S. (2024) · IEEE Access · doi:10.1109/access.2024.3369912 · Cited by 152 · see citing works
Psychological consequences of legal responsibility misattribution associated with automated vehicles
Liu, P., Du, M., & Li, T. (2021) · Ethics and Information Technology · doi:10.1007/s10676-021-09613-y · Cited by 20 · see citing works
Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents
La Malfa, G., Meegahapola, L., Bogucka, E., Zhang, J. M., Luck, M., Black, E., & Quercia, D. (2026) · arXiv · doi:10.48550/arxiv.2608.08601
- Authors
- Macrae, C.
- Affiliations
- University of Nottingham (GB)
- Corresponding author
- Macrae, Carl
- ORCID
- Macrae, Carl
- Venue
- Risk Analysis
- Publisher
- Wiley
- Published
- 2021-11-23
- Volume
- 42
- Issue
- 9
- Pages
- 1999-2025
- ISSN
- 0272-4332 (ISSN-L)
- 1539-6924
- OpenAlex
- W3215206150
- Cited by
- 76
- References
- 135
- Citations by year
- 2026: 23 · 2025: 18 · 2024: 20 · 2023: 10 · 2022: 3 · 2021: 2
- FWCI
- 6.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Risk Analysis (publishedVersion, cc-by)
- SSRN Electronic Journal (publishedVersion)
- PubMed (publishedVersion)
- Repository@Nottingham (University of Nottingham) (publishedVersion, other-oa)
- Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by)
- Funders
- Wellcome Trust (213632/Z/18/Z)
- Topics
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Risk and Safety Analysis
- Keywords
- autonomous vehicles, risk management, organizational risk, epistemic risk, cultural risk, technological risk, structural risk, complex systems, learning from failure, safety governance
- MeSH terms
- Accidents, Humans, Causality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
