Machine learning, social learning and the governance of self-driving cars
Open accessThe DOI matches its Crossref record; 116 of the 116 references it cites were found in a registry or an index; none of the 99 sources its claims cite is checked for support yet.
Abstract
Self-driving cars, a quintessentially ‘smart’ technology, are not born smart. The algorithms that control their movements are learning as the technology emerges. Self-driving cars represent a high-stakes test of the powers of machine learning, as well as a test case for social learning in technology governance. Society is learning about the technology while the technology learns about society. Understanding and governing the politics of this technology means asking ‘Who is learning, what are they learning and how are they learning?’ Focusing on the successes and failures of social learning around the much-publicized crash of a Tesla Model S in 2016, I argue that trajectories and rhetorics of machine learning in transport pose a substantial governance challenge. ‘Self-driving’ or ‘autonomous’ cars are misnamed. As with other technologies, they are shaped by assumptions about social needs, solvable problems, and economic opportunities. Governing these technologies in the public interest means improving social learning by constructively engaging with the contingencies of machine learning.
Overview
CoChat Summary
TL;DR
The paper argues that self-driving cars are misnamed and that their governance is a high-stakes test of social learning, because machine learning systems remain opaque, contingent, and shaped by social assumptions. Using the May 2016 Tesla Model S fatal crash, it shows how industry and regulators often focus on human error and aggregate outcomes rather than the processes and purposes of innovation, and it proposes data sharing and democratized learning as governance responses.
Key findings
- “Self-driving” and “autonomous” are misnomers: cars are not self-contained, self-sufficient, or self-taught; Tesla’s “fleet learning” means each car learns from the accumulated experience of thousands of sister vehicles, and autonomy only happens when tasks and contexts are sufficiently constrained.
- In the May 2016 Tesla crash, the car was in Autopilot and hit a crossing truck at 74 mph in a 65 mph zone; NTSB data showed Joshua Brown’s 40-minute journey included 37.5 minutes on Autopilot, his hands were off the wheel for 37 minutes, he touched the wheel eight times, the longest gap was six minutes, and neither he nor Autopilot applied the brakes. The NTSB calculated he would have had at least ten seconds to react had he seen the truck, and NHTSA noted automatic emergency braking was not designed for crossing-path collisions.
- Governance responses often privatized learning: Tesla called Autopilot a “public beta,” Musk framed acceptability through aggregate math (over 1M auto deaths per year worldwide, approximately half a million saved if Autopilot were universally available), and NHTSA’s report largely directed recommendations at drivers. California DMV disengagement reports showed Tesla’s four test vehicles covered only 550 miles in 2016 and disengaged on average every three miles, while Tesla claimed customers had covered more than a billion miles in Autopilot mode by the end of 2016.
- The paper identifies constructive alternatives: Mobileye’s “semantic abstraction” versus Nvidia’s “end-to-end” learning, recognition that handoffs from automation can take as long as 40 seconds, and governance proposals for data sharing, event data recorders, and “ethical black boxes.”
Methods
- Design: qualitative STS/conceptual analysis and single-case study of public debate around self-driving car innovation, centered on the May 2016 Tesla Model S fatal crash and its aftermath.
- Data: official investigations (NTSB, NHTSA, Florida police), Tesla blog posts and statements, regulatory guidance (NHTSA, California DMV), media reports, and informal online exchanges among Tesla users.
- Analysis: interpretive tracing of competing problem definitions, promises, and learning practices; contrasts machine learning opacity with social learning and governance proposals.
Limitations
- The paper states no formal limitations section, but it acknowledges evidentiary opacity: the NTSB could determine that detection of the truck did not occur but not why, and Autopilot’s decision-making remained largely off-limits to investigators.
- Evident from the text: the analysis is a single-case study focused on Tesla and the US regulatory context, limiting generalizability to other self-driving car governance settings.
- Evident from the text: it relies on public/official documents, media, and online exchanges rather than systematic primary data, and it is normative/conceptual rather than a quantitative test of safety or governance outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 85 Not checked · 73 Own results and interpretations
In the paper
“The car's brain remained largely off-limits to investigators.”
What the sources say
- Not checked yetcites NTSB, 2017c
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The first official report of the May 2016 crash, from the Florida police, put the blame squarely on the truck driver for failing to yield the right of way (Traffic Crash Records, 2016).”
What the sources say
- Not checked yetcites Traffic Crash Records, 2016
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Their final word on the probable cause of the Tesla crash added a concern with Autopilot's 'operational design, which permitted [the driver's] prolonged disengagement from the driving task and his use of the automation in ways inconsistent with guidance and warnings from the manufacturer' (NTSB, 2017c).”
What the sources say
- Not checked yetcites NTSB, 2017c
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“'Self-driving' or 'autonomous' cars are misnamed.”
- Also stated in: § 'Autonomous vehicles' and 'self-driving cars' as misnomers · plain
“Just as self-driving cars cannot be self-driven, so autonomous vehicles can never be truly autonomous (Stayton, 2015).”
What the sources say
- Not checked yetcites Stayton, 2015
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 'Autonomous vehicles' and 'self-driving cars' as misnomers · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Focusing on the successes and failures of social learning around the much-publicized crash of a Tesla Model S in 2016, I argue that trajectories and rhetorics of machine learning in transport pose a substantial governance challenge.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
51 more claims
Create a free account to see all 160 claims and to verify the 99 references still open.
- It may state two separate claims as one. 48
- Probably not a substantial claim. 76
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“According to the recent literature, factors contributing to the overall lack of algorithmic transparency include the cognitive impossibility for humans to interpret massive algorithmic models and datasets; a lack of appropriate tools to visualise and track large volumes of code and data; code and data that are so poorly structured that they are impossible to read; and ongoing updates and human”
From The ethics of algorithms: key problems and solutions (2022) · cites this paper as Stilgoe 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Democratizing learning
“(GHSA, 2017; Policy Network 2016; Waymo 2017)”
§ References
“GHSA (Governors Highway Safety Association) (2016) Autonomous vehicles meet human drivers: Traffic safety issues for states.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Constructing human deficiency
“Musk claimed that these updates would 'minimize the possibility of people doing crazy things with it' (quoted in Charton, 2015).”
§ References
“Charton A (2015) Tesla owners need to stop doing crazy things with Autopilot, says Elon Musk. International Business Times . 4 November.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The gamification of driving
“(Lipson and Kurman, 2015: 219).”
§ References
“Lipson H and Kurman, M (2016) Driverless: Intelligent Cars and the Road Ahead . Cambridge: MIT Press.”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“As of 2016, these neural networks’ greatest feats have been in digital image recognition, where it is claimed their abilities have surpassed those of humans (Krizhevsky et al., 2012), and voice recognition (LeCun et al., 2015).”
BackgroundLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“…and modes of machine learning (for example, reinforcement learning, which adopts a trial, error and reward approach to optimization, described in one leading paper [Mnih et al., 2015] as ‘deeply rooted in psychological and neuroscientific perspectives on animal behaviour’) is inescapably social.”
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Policy paradigms, social learning, and the state: The case of economic policymaking in Britain
Hall, P. A. (1993) · Comparative Politics · doi:10.2307/422246 · Cited by 7,369 · cited 1×
“These frameworks have an analytical basis in theories of politics as a form of learning (Hall, 1993), with policymaking seen not just as the playing out of stakeholder interests, but as ‘puzzling together’ (Hoppe, 2011), in the tradition of Dewey (1916).”
BackgroundThe ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567 · cited 1×
“As it has become clear that citizens’ lives are increasingly shaped and cajoled by the influence of algorithms and artificial intelligence, social scientists, legal scholars and philosophers have developed critical approaches (Mittelstadt et al., 2016).”
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · cited 2×
“Their emphasis is on ‘end-to-end’ machine learning, in which the system works out the rules that need to be prioritized (detecting the edges of roads, for example) for the solution of its task (Bojarski et al., 2016a).”
BackgroundHow the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 2×
“The tension between rule-fol-lowing and rule-making in machine learning has started to be explored by literature on the politics of algorithms (Burrell, 2016).”
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · cited 1×
“Frameworks of responsible innovation also aim to engage upstream with the processes and purposes of innovation, recognizing that, in addition to questions of risk, public concerns also relate to how and why innovation happens (Stilgoe et al., 2013).”
BackgroundSelf-citationGummett, P. (1988) · Physics Bulletin · doi:10.1088/0031-9112/39/8/019 · Cited by 2,974
Bijker, W. E., Hughes, T. P., & Pinch, T. (1987) · Cited by 2,957
Winner, L. (2017) · Computer Ethics · doi:10.4324/9781315259697-21 · Cited by 2,317 · cited 1×
“Work in science and technology studies (STS) has gone some way towards explaining the social constitution of technologies such as nuclear power, genetically modified foods, nanotechnology and geoengineering (Kearnes et al., 2006; Szerszynski et al., 2013; Winner, 1980).”
BackgroundEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“This has led some to call for a ‘right to explanation’ (Goodman and Flaxman, 2016) in cases where the ‘production of prediction’ (Mackenzie, 2015) has profound consequences for people’s lives.”
The sociology of expectations in science and technology
Borup, M., Brown, N., Konrad, K., & Van Lente, H. (2006) · Technology Analysis & Strategic Management · doi:10.1080/09537320600777002 · Cited by 1,902 · cited 2×
“One attempt (though flawed [Borup et al., 2006]) to capture and challenge the levels of promise surrounding emerging technologies has been developed by Gartner, a consultancy firm.”
Ananny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · cited 3×
“When Google’s algorithms were misidentifying images of people as dogs and, with predictably greater controversy, black people as gorillas, the company’s own engineers could not pinpoint where the problem lay (Ananny and Crawford, 2016).”
BackgroundMethodInfluentialToward a history of epistemic things: synthesizing proteins in the test tube
(1998) · Choice Reviews Online · doi:10.5860/choice.35-3290 · Cited by 1,752 · cited 1×
“If experiments are understood, following Rheinberger (1997), as systems for the organized production of surprises, then an approach to technology as a social experiment should prompt us to ask what sorts of uncertainties are considered pertinent by different actors and how they respond (Gross,…”
BackgroundReed, M. S., Evely, A. C., Cundill, G., Fazey, I., Glass, J., Laing, A., Newig, J., Parrish, B., Prell, C., Raymond, C., et al. (2010) · Ecology and Society · doi:10.5751/es-03564-1504r01 · Cited by 1,554
The social dilemma of autonomous vehicles
Bonnefon, J.-F., Shariff, A., & Rahwan, I. (2016) · Science · doi:10.1126/science.aaf2654 · Cited by 1,527 · cited 1×
“One early governance concern with self-driving cars has related to the ethics of algorithms making life or death decisions in the event of crashes and the responsibilities that this might place upon designers and manufacturers (Bonnefon et al., 2016; Crawford and Calo, 2016).”
BackgroundCitizens, Experts, and the Environment: The Politics of Local Knowledge
Fischer, F. (2000) · Duke University Press eBooks · doi:10.1215/9780822380283 · Cited by 1,518
The Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · International Data Privacy Law · doi:10.1093/idpl/ipx005 · Cited by 1,308
Kalra, N., & Paddock, S. M. (2016) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2016.09.010 · Cited by 1,221
Technologies of Humility: Citizen Participation in Governing Science
Jasanoff, S. (2003) · Minerva · doi:10.1023/a:1025557512320 · Cited by 1,004
Policy without polity? Policy analysis and the institutional void
Hajer, M. (2003) · Policy Sciences · doi:10.1023/a:1024834510939 · Cited by 841 · cited 1×
“The conception of governance as social learning is particularly pertinent to new technologies, which typically emerge in what Hajer (2003) calls an ‘institutional void’, surrounded by uncertainties not just about the effects of technology (Collingridge, 1980) but also the object of governance…”
BackgroundTechnologies of Humility: Citizen Participation in Governing Science
Jasanoff, S. (2005) · Wozu Experten? · doi:10.1007/978-3-322-80692-5_17 · Cited by 768 · cited 2×
“Given the uncertainties of innovation and an institutionalized tendency towards hubris (Jasanoff, 2003), a focus on social learning offers a way to understand and democratize the means and ends of the social experiment of technological change.”
BackgroundLate lessons from early warnings: the precautionary principle 1896-2000
MacGarvin, M., Lambert, B. E., Infante, P. F., Greenberg, M., Gee, D., Koppe, J. G., Keys, J. G., Farman, J. C., Ibarreta, D., Swan, S. H., et al. (2001) · Figshare · Cited by 754
Understanding ‘anticipatory governance’
Guston, D. H. (2014) · Social Studies of Science · doi:10.1177/0306312713508669 · Cited by 749 · cited 1×
“The ends enabled by technological innovation are typically hard to anticipate and hard to challenge, in part because of innovators’ lack of reflexivity about uncertainties and contingencies (Guston, 2014).”
BackgroundInventing accuracy: an historical sociology of nuclear missile guidance
(1991) · Choice Reviews Online · doi:10.5860/choice.29-0924 · Cited by 745
Worthington, R. (1982) · American Political Science Review · doi:10.2307/1960465 · Cited by 714
Autonomous Vehicle Technology: A Guide for Policymakers
Anderson, J., Kalra, N., Stanley, K., Sorensen, P., Samaras, C., & Oluwatola, O. (2016) · doi:10.7249/rr443-2 · Cited by 653
Risk and Sociocultural Theory: New Directions and Perspectives
(1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511520778 · Cited by 635 · cited 1×
“In legitimizing only the probabilistic risk quantity, he is ignoring concerns about risk quality, such as those revealed through social science relating to control, trust, fairness, catastrophic potential, novelty and uncertainty (Irwin et al., 1999; Lupton, 1999; Renn, 1998).”
BackgroundOrganisational Applications of Social Cognitive Theory
Bandura, A. (1988) · Australian Journal of Management · doi:10.1177/031289628801300210 · Cited by 631 · cited 1×
“The first, originating in educational psychology (e.g., Bandura, 1988), starts from the observation that people can and do learn in groups – with and from others.”
BackgroundWeinberg, A. M. (1972) · Minerva · doi:10.1007/bf01682418 · Cited by 624
The language instinct : the new science of language and mind
Pinker, S. A. (1995) · Penguin Books · Cited by 593
There is a blind spot in AI research
Crawford, K., & Calo, R. (2016) · Nature · doi:10.1038/538311a · Cited by 580 · cited 1×
“One early governance concern with self-driving cars has related to the ethics of algorithms making life or death decisions in the event of crashes and the responsibilities that this might place upon designers and manufacturers (Bonnefon et al., 2016; Crawford and Calo, 2016).”
BackgroundThe Social Control of Technology
Mayne, A. J. (1982) · Journal of the Operational Research Society · doi:10.2307/2634327 · Cited by 577
Public participation in impact assessment: A social learning perspective
Webler, T., Kastenholz, H., & Renn, O. (1995) · Environmental Impact Assessment Review · doi:10.1016/0195-9255(95)00043-e · Cited by 551 · cited 1×
“Recent efforts to expand practices of public deliberation on new technologies can be seen as a form of learning with and about the social context of new technologies, with the additional aim of galvanizing collective action (Fischer, 2000; Webler et al., 1995).”
BackgroundLatour, B., & Venn, C. (2002) · Theory, Culture & Society · doi:10.1177/026327602761899246 · Cited by 543
Berkes, F., & Turner, N. J. (2006) · Human Ecology · doi:10.1007/s10745-006-9008-2 · Cited by 530 · cited 1×
“Systems that are good at social learning are seen as more resilient, interpreting shocks and crises as opportunities (Berkes and Turner, 2006).”
BackgroundEdwards, L., & Veale, M. (2017) · doi:10.31228/osf.io/97upg · Cited by 524
Weinberg, A. M. (1972) · Science · doi:10.1126/science.177.4045.211 · Cited by 485
Dewey, J. (1916) · University of Chicago Press eBooks · doi:10.1037/13833-000 · Cited by 461
Public uptake of science: a case for institutional reflexivity
Wynne, B. (1993) · Public Understanding of Science · doi:10.1088/0963-6625/2/4/003 · Cited by 395 · cited 1×
“Institutions often lack the reflexive capacity to take on board and respond either to the views of others (Wynne, 1993) or to early warnings of danger (European Environment Agency [EEA], 2001).”
BackgroundToward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness
Ananny, M. (2016) · Science, Technology, & Human Values · doi:10.1177/0162243915606523 · Cited by 395
The role of risk perception for risk management
Renn, O. (1998) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(97)00119-1 · Cited by 353 · cited 1×
“In legitimizing only the probabilistic risk quantity, he is ignoring concerns about risk quality, such as those revealed through social science relating to control, trust, fairness, catastrophic potential, novelty and uncertainty (Irwin et al., 1999; Lupton, 1999; Renn, 1998).”
BackgroundBeyond ‘substantial equivalence’
Millstone, E., Brunner, E., & Mayer, S. (1999) · Nature · doi:10.1038/44006 · Cited by 335 · cited 1×
“Those advocating a more precautionary approach to regulation have argued that a product-based approach, by emphasizing the ‘substantial equivalence’ between an innovation and its predecessors, overlooks uncertainties that may emerge as problematic (Millstone et al., 1999).”
BackgroundVirtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.11 · Cited by 326 · cited 1×
“Reinforcement learning in simulated environments, including games such as Grand Theft Auto, is seen as one way of accelerating self-driving car training (Filipowicz et al., 2017; Pan et al., 2017).”
MethodHow Fair Is Safe Enough? The Cultural Approach to Societal Technology Choice1
Rayner, S., & Cantor, R. (1987) · Risk Analysis · doi:10.1111/j.1539-6924.1987.tb00963.x · Cited by 326 · cited 1×
“Discussions that start with ‘risk’ end up being about the purposes of a particular technology and its alternatives (Rayner and Cantor, 1987).”
BackgroundVirtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · arXiv · doi:10.48550/arxiv.1704.03952 · Cited by 326
Kelly, K. (2010) · Cited by 322
Unruly Technology: Practical Rules, Impractical Discourses and Public Understanding
Wynne, B. (1988) · Social Studies of Science · doi:10.1177/030631288018001006 · Cited by 319 · cited 4×
“Technological accidents can wrest control of the social experiment away from the tech-nologists, laying bare the rules and assumptions that shape black boxes and exposing the uncertainties that are so easy to ignore when things work well (Wynne, 1988).”
BackgroundInfluentialKalra, N., & Paddock, S. (2016) · RAND Corporation eBooks · doi:10.7249/rr1478 · Cited by 305
The governance of problems: Puzzling, powering and participation
Hoppe, R. (2010) · doi:10.1332/policypress/9781847429629.001.0001 · Cited by 286
What Algorithms Want: Imagination in the Age of Computing
Finn, E. (2017) · The MIT Press eBooks · doi:10.7551/mitpress/9780262035927.001.0001 · Cited by 267
The production of prediction: What does machine learning want?
Mackenzie, A. (2015) · European Journal of Cultural Studies · doi:10.1177/1367549415577384 · Cited by 259 · cited 1×
“This has led some to call for a ‘right to explanation’ (Goodman and Flaxman, 2016) in cases where the ‘production of prediction’ (Mackenzie, 2015) has profound consequences for people’s lives.”
BackgroundThe Ethics of Invention: Technology and the Human Future
Lee, M. (2017) · Law, Innovation and Technology · doi:10.1080/17579961.2017.1306967 · Cited by 254 · cited 1×
“This exacerbates the tendency to blame the things that come to be labelled ‘unintended consequences’ (Jasanoff, 2016), on ‘human error’ (Shorrock, 2013) or ‘user error’ (Wynne, 1988).”
Background(2012) · Cited by 253
Vigilance Decrement and Passive Fatigue Caused by Monotony in Automated Driving
Körber, M., Cingel, A., Zimmermann, M., & Bengler, K. (2015) · Procedia Manufacturing · doi:10.1016/j.promfg.2015.07.499 · Cited by 247 · cited 1×
“This is a well-known feature of automated systems, bolstered by research in self-driving car simulators (Körber et al., 2015), in which the ‘vigilance decrement’ of humans becomes more problematic as automation improves.”
MethodSlave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244
Ignorance and surprise: science, society, and ecological design
(2011) · Choice Reviews Online · doi:10.5860/choice.48-4791 · Cited by 219
Ignorance and Surprise: Science, Society, and Ecological Design
Gross, M. (2010) · The MIT Press eBooks · doi:10.7551/mitpress/9780262013482.001.0001 · Cited by 218 · cited 1×
“…understood, following Rheinberger (1997), as systems for the organized production of surprises, then an approach to technology as a social experiment should prompt us to ask what sorts of uncertainties are considered pertinent by different actors and how they respond (Gross, 2010; Stilgoe, 2016).”
BackgroundThe Seven Deadly Myths of "Autonomous Systems"
Bradshaw, J. M., Hoffman, R. R., Johnson, M., & Woods, D. D. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.70 · Cited by 216 · cited 2×
“As Bradshaw et al. (2013) point out, ‘autonomous system’ is a misnomer, if not an oxymoron.”
BackgroundWhat Algorithms Want: Imagination in the Age of Computing
Finn, E. (2017) · Project Muse (Johns Hopkins University) · Cited by 199
The Trouble with Autopilots: Assisted and Autonomous Driving on the Social Road
Brown, B., & Laurier, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025462 · Cited by 186 · cited 1×
“YouTube is replete with hands-free Tesla driving displays and other haphazard experimentation (Brown and Laurier, 2017).”
BackgroundCohen, T., & Cavoli, C. (2019) · Transport Reviews · doi:10.1080/01441647.2018.1524401 · Cited by 180
Pilkington, A. (1996) · Language and Literature: International Journal of Stylistics · doi:10.1177/096394709600500107 · Cited by 174
Faulty Environments and Risk Reasoning: The Local Understanding of Industrial Hazards
Irwin, A., Simmons, P., & Walker, G. (1999) · Environment and Planning A: Economy and Space · doi:10.1068/a311311 · Cited by 171 · cited 1×
“In legitimizing only the probabilistic risk quantity, he is ignoring concerns about risk quality, such as those revealed through social science relating to control, trust, fairness, catastrophic potential, novelty and uncertainty (Irwin et al., 1999; Lupton, 1999; Renn, 1998).”
Driverless: Intelligent Cars and the Road Ahead
Lipson, H., & Kurman, M. (2016) · Project Muse (Johns Hopkins University) · Cited by 171
The Governance of Problems: Puzzling, Powering and Participation
Hoppe, R. (2010) · Policy Press eBooks · doi:10.46692/9781847426307 · Cited by 169
Public Experiments and Displays of Virtuosity: The Core-Set Revisited
Collins, H. M. (1988) · Social Studies of Science · doi:10.1177/030631288018004006 · Cited by 166 · cited 1×
“Unlike the DARPA challenges, which were genuine public experiments, subsequent efforts had closer control of their uncertainties (see Collins, 1988).”
ResultFrom Bio to Nano: Learning Lessons from the UK Agricultural Biotechnology Controversy
Kearnes, M., Grove‐White, R., Macnaghten, P., Wilsdon, J., & Wynne, B. (2006) · Science as Culture · doi:10.1080/09505430601022619 · Cited by 159 · cited 1×
“Work in science and technology studies (STS) has gone some way towards explaining the social constitution of technologies such as nuclear power, genetically modified foods, nanotechnology and geoengineering (Kearnes et al., 2006; Szerszynski et al., 2013; Winner, 1980).”
BackgroundControversies as Informal Technology A ssessment
Rip, A. (1986) · Knowledge · doi:10.1177/107554708600800216 · Cited by 155 · cited 1×
“1 However, while accidents may be an instructive form of ‘informal technology assess-ment’ (Rip, 1986), the interpretation of their lessons is, as with all learning, socially filtered.”
BackgroundWhy Solar Radiation Management Geoengineering and Democracy Won't Mix
Szerszynski, B., Kearnes, M., Macnaghten, P., Owen, R., & Stilgoe, J. (2013) · Environment and Planning A: Economy and Space · doi:10.1068/a45649 · Cited by 153 · cited 1×
“Work in science and technology studies (STS) has gone some way towards explaining the social constitution of technologies such as nuclear power, genetically modified foods, nanotechnology and geoengineering (Kearnes et al., 2006; Szerszynski et al., 2013; Winner, 1980).”
BackgroundSelf-citationNissenbaum, H. (1994) · Communications of the ACM · doi:10.1145/175222.175228 · Cited by 149 · cited 1×
“As Nissenbaum (1994) observed more than twenty years ago, we must ‘accept that the producers of computer systems are not, in general, fully answerable for the impacts of their products.”
BackgroundThe Ethics of Invention: Technology and the Human Future
Jasanoff, S. (2016) · Cited by 142
Product, process, or programme: three cultures and the regulation of biotechnology
Jasanoff, S. (1995) · Cambridge University Press eBooks · doi:10.1017/cbo9780511563706.016 · Cited by 136 · cited 1×
“The other approach, more common in Europe, focuses on the novelty of the process (Jasanoff, 1995; Kuzma, 2016).”
BackgroundBearing Account-able Witness to the Ethical Algorithmic System
Neyland, D. (2016) · Science, Technology, & Human Values · doi:10.1177/0162243915598056 · Cited by 126 · cited 1×
“As computer entrepreneur-turned-academic Kevin Slavin puts it: ‘We’re writing things we can no longer read’ (quoted in Neyland, 2016).”
BackgroundCooperative (rather than autonomous) vehicle-highway automation systems
Shladover, S. (2009) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2009.932716 · Cited by 122 · cited 1×
“…the stimulus of DARPA may have set self-driving in a counter-productive direction, given the very different needs of military vehicles, operating in unstructured environments, compared to conventional cars, which are, in most places, embedded in well-organized highway systems (Shladover, 2009).”
BackgroundGeoengineering as Collective Experimentation
Stilgoe, J. (2015) · Science and Engineering Ethics · doi:10.1007/s11948-015-9646-0 · Cited by 115 · cited 1×
“…understood, following Rheinberger (1997), as systems for the organized production of surprises, then an approach to technology as a social experiment should prompt us to ask what sorts of uncertainties are considered pertinent by different actors and how they respond (Gross, 2010; Stilgoe, 2016).”
BackgroundSelf-citationNational Motor Vehicle Crash Causation Survey
Hs, D. (2008) · Cited by 101
Our Bodies in the Trolley’s Path, or Why Self-driving Cars Must *Not* Be Programmed to Kill
JafariNaimi, N. (2018) · Science, Technology, & Human Values · doi:10.1177/0162243917718942 · Cited by 87 · cited 2×
“JafariNaimi (2017) argues that the self-driving car presents an opportunity for reframing transport governance.”
BackgroundChander, A. (2013) · Scholarly Commons (University of the Pacific) · Cited by 87
Real Robots, Real Learning Problems
Brooks, R. A., & Matarić, M. J. (1993) · Springer eBooks · doi:10.1007/978-1-4615-3184-5_8 · Cited by 84 · cited 1×
“Such uncontrolled encounters are more publicly useful than any number of advertisements of perfection, which, as roboticists have pointed out, are ‘doomed to succeed’ (Brooks and Mataric, 1993).”
BackgroundControlled Flight into Terrain Accidents: System-Induced Errors
Wiener, E. L. (1977) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872087701900207 · Cited by 82 · cited 1×
“Reframing errors as ‘system-induced’ (Wiener, 1977) rather than merely a function of human frailty allows for genuine learning.”
BackgroundThe Novelty Trap: Why Does Institutional Learning about New Technologies Seem So Difficult?
Rayner, S. (2004) · Industry and Higher Education · doi:10.5367/0000000042683601 · Cited by 82 · cited 5×
“They must navigate what Rayner (2004) calls the ‘novelty trap’, advertising a technology as ground-breaking and transformative while seeking to per-suade regulators that it is no more than an incremental step beyond existing approaches.”
BackgroundInfluentialHuman Factors Evaluation of Level 2 and Level 3 Automated Driving Concepts
Blanco, M., Atwood, J., Vasquez, H. M., Trimble, T. E., Fitchett, V., Radlbeck, J., Fitch, G. M., Russell, S. M., Green, C. A., Cullinane, B., et al. (2015) · doi:10.13140/rg.2.1.1874.7361 · Cited by 81
Technological Opacity, Predictability, and Self-Driving CarsPreprint
Surden, H., & Williams, M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2747491 · Cited by 79 · cited 1×
“…to travel unlabelled, so that other road users do not take advantage of their presumed generosity, there is growing recognition among engineers of the need for vehicles to actively communicate their presence, their intentions and their capabilities to other road users (Surden and Williams, 2016).”
BackgroundThe Case for an Ethical Black Box
Winfield, A. F. T., & Jirotka, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-64107-2_21 · Cited by 76 · cited 1×
“The need to improve social learning has led some to call for the inclusion of ‘ethical black boxes’ in robotic systems (Winfield and Jirotka, 2017).”
BackgroundPolicy: Reboot the debate on genetic engineering
Kuzma, J. (2016) · Nature · doi:10.1038/531165a · Cited by 71 · cited 1×
“The other approach, more common in Europe, focuses on the novelty of the process (Jasanoff, 1995; Kuzma, 2016).”
BackgroundWhy robots should be social: Enhancing machine learning through social human-robot interaction
de Greeff, J., & Belpaeme, T. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138061 · Cited by 66 · cited 1×
“Engineers have already had to engage in a form of social-ized machine learning, building on research in social robotics (e.g., de Greeff and Belpaeme, 2015).”
BackgroundSocial Learning: A Model for Policy Research
Friedmann, J. R. P., & Abonyi, G. (1976) · Environment and Planning A Economy and Space · doi:10.1068/a080927 · Cited by 62
Life in the Fast Lane: The Evolution of an Adaptive Vehicle Control System
Jochem, T. M., & Pomerleau, D. (1996) · AI Magazine · doi:10.1609/aimag.v17i2.1221 · Cited by 53
Solar Radiation Management and the Governance of Hubris
Owen, R. (2014) · Royal Society of Chemistry eBooks · doi:10.1039/9781782621225-00212 · Cited by 46 · cited 1×
“…of governance as social learning is particularly pertinent to new technologies, which typically emerge in what Hajer (2003) calls an ‘institutional void’, surrounded by uncertainties not just about the effects of technology (Collingridge, 1980) but also the object of governance itself (Owen, 2014).”
BackgroundHandover issues in autonomous driving: A literature review
Morgan, P. A., Alford, C., & Parkhurst, G. P. (2016) · UWE Research Repository (UWE Bristol) · Cited by 43
The New Geo-Governance: A Baroque Approach
Paquet, G. (2005) · University of Ottawa Press eBooks · doi:10.1353/book4421 · Cited by 41
Leonardi, P. M. (2009) · Social Studies of Science · doi:10.1177/0306312709346079 · Cited by 40 · cited 1×
“Crash-testing, for example, has been a vital part of automotive innovation, but it has been imagined as a private activity, conducted in laboratories or computer simulations (Leonardi, 2010).”
On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training
Shalev‐Shwartz, S., & Shashua, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06915 · Cited by 37
The new geo-governance: a baroque approach
(2006) · Choice Reviews Online · doi:10.5860/choice.43-4923 · Cited by 33
Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (pre-print)Preprint
Elish, M. C. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2757236 · Cited by 32
Autonomous Vehicles for Personal Transport: A Technology AssessmentPreprint
Moore, M. M., & Lu, B. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1865047 · Cited by 27 · cited 1×
“One engineering analysis (Moore and Lu, 2011) argues that the rules written by machines from available data would be less arbitrary than the legal ‘rules of the road’ (see also Both, 2014).”
BackgroundPaying Attention to the Man behind the Curtain
Cummings, M. L., & Thornburg, K. M. (2010) · IEEE Pervasive Computing · doi:10.1109/mprv.2011.7 · Cited by 26 · cited 1×
“Long periods of autonomy fundamentally change the role of the human from operator (‘in the loop’) to supervisor (‘on the loop’) (Cummings and Thornburg, 2011; Perrow, 1984).”
BackgroundAutonomous Vehicles Meet Human Drivers: Traffic Safety Issues for States
Hedlund, J. (2017) · Cited by 19
Walsh, T. (2016) · Communications of the ACM · doi:10.1145/2838729 · Cited by 14 · cited 1×
“One AI researcher has suggested the need for a ‘Turing red flag law’ (Walsh, 2016), mandating the clear labelling of all autonomous systems.”
BackgroundDriverless Dreams: Technological Narratives and the Shape of the Automated Car
Stayton, E. (2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 13
(2013) · Information Polity · doi:10.3233/ip-130311 · Cited by 13
Meyer, August or Clemens August
(2011) · Oxford University Press eBooks · doi:10.1093/benz/9780199773787.article.b00121828 · Cited by 7
The normativity of automated driving: a case study of embedding norms in technology
Eriksson, M. (2017) · Information & Communications Technology Law · doi:10.1080/13600834.2017.1269872 · Cited by 6 · cited 1×
“…of road safety, for example, Vision Zero, a strategy for reducing road deaths that began in Sweden, offers an alternative model of social learning that puts car innovation alongside infrastructure, law and social norms in redistributing responsibility for safety (Eriksson, 2017; JafariNaimi, 2017).”
BackgroundToward a History of Epistemic Things: Synthesizing Proteins in the Test Tube. Hans-Jorg Rheinberger
Fruton, J. S. (1998) · Isis · doi:10.1086/384059 · Cited by 6
(2005) · Sage eBooks · doi:10.4135/9781412983907.n1625 · Cited by 3
A Reflection: How Asperger's Syndrome has Changed my Life
Mitchell, C. (2005) · SAGE Publications Ltd eBooks · doi:10.4135/9781446212660.n16 · Cited by 2
Oakes, Col Richard, (12 March 1876–14 July 1944)
(2007) · Oxford University Press eBooks · doi:10.1093/ww/9780199540884.013.u229901 · Cited by 2
Press Conference, Hyde Park, New York, September 4, 1935, 11 A.M.
(2014) · Harvard University Press eBooks · doi:10.4159/harvard.9780674599468.c3 · Cited by 1
Burger, E. B. (2018) · Princeton University Press eBooks · doi:10.1515/9780691188881-009 · Cited by 1
(2013)
European Union regulations on algorithmic decision-making and a "right to explanation"
Goodman, B., & Flaxman, S. (2016) · arXiv · doi:10.48550/arxiv.1606.08813 · cited 1×
“This has led some to call for a ‘right to explanation’ (Goodman and Flaxman, 2016) in cases where the ‘production of prediction’ (Mackenzie, 2015) has profound consequences for people’s lives.”
Research works printed but not linked: 19
- Ackerman E (2016) After mastering Singapore's streets, NuTonomy's robo-taxies are poised to take on new cities. IEEE Spectrum . 29 December. Available at: http://spectrum.ieee.org/transportation/self-driving/after-mastering-singapores-streetsnutonomys-robotaxis-are-poised-to-take-on-new-cities (accessed 12 March 2017).
- Filipowicz A, Liu J and Kornhauser A (2017) Learning to recognize distance to stop signs using the virtual world of Grand Theft Auto 5. Paper for the Transportation Research Board.
- Frankel T (2016) Elon Musk says Tesla's Autopilot is already 'probably' better than human drivers. The Washington Post , 1 November.
- Gomes L (2016) Google self-driving car will be ready soon for some, in decades for others. IEEE Spectrum . 18 March. Available at: http://spectrum.ieee.org/cars-thatthink/transportation/self-driving/google-selfdriving-car-will-be-ready-soon-for-somein-decades-for-others (accessed 12 March 2017).
- Guizzo E and Ackerman E (2015) Gill Pratt discusses Toyota's AI plans and the future of robots and cars. IEEE Spectrum , 11 Sept.
- Kosner AW (2013) Why is machine learning (CS 229) the most popular course at Stanford? Forbes . 29 December.
- Lambert F (2016) Transcript: Elon Musk's press conference about Tesla Autopilot under v8.0 update [Part 2]. Electrek . 11 September. Available at: https://electrek.co/2016/09/11/transcript-elon-musks-press-conference-tesla-autopilotunder-v8-0-update-part-2/ (accessed 12 March 2017).
- Lane A (2016) 'Lo and Behold' and 'Mia Madre' reviews. The New Yorker. 29 August.
- Nourbakhsh IR (2013) Robot Futures. Cambridge, MA: MIT Press.
- Parson EA and Clark WC (1995) Sustainable development as social learning: Theoretical perspectives and practical challenges for the design of a research program. In Holling ES, Gunderson LH, and Light S (eds) Barriers and Bridges to the Renewal of Ecosystems. New York: Columbia University Press, 428-460.
- Perrow C (1984) Normal Accidents: Living with High Risk Systems . New York, Basic Books.
- Puzzanghera J (2016) Fatal Tesla crash exposes lack of regulation over autopilot technology. Los Angeles Times . 1 July.
- Ramsey M (2016). Mobileye ends partnership with Tesla. The Wall Street Journal . 26 July.
- Ranft F, Adler M, Diamond P, Guerrero E, Laza M (2016) Freeing the Road: Shaping the Future for Autonomous Vehicles. London: Policy Network.
- Surden H and Williams MA(2016) Technological opacity, predictability, and selfdriving cars. Cardozo Law Review 38: 121.
- Wetmore J (2003) Driving the dream: The history and motivations behind 60 years of automated highway systems in America. Automotive History Review 7: 4–19.
- Winner L (1977) Autonomous Technology: Technology-out-of-Control as a Theme in Political Thought. Cambridge, MA: MIT Press.
- Wynne B (1989) Frameworks of rationality in risk management: Towards the testing of naive sociology. In: Brown J (ed.) Environmental Threats: Social Sciences Approaches to Public Risk Perceptions. New York: Belhaven Press, pp. 33–45.
- Wynne B (1992) Risk and social learning: Reification to engagement. In: Krimsky S, Golding D (eds) Social Theories of Risk. Westport, CT: Praeger, pp. 275–300.
Cited by
The Ethics of Artificial Intelligence: Principles, Challenges, and Opportunities
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 507
In AI We Trust: Ethics, Artificial Intelligence, and Reliability
Ryan, M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00228-y · Cited by 600
“So far, this has been a fairly effective procedure, despite some controversial incidents such as the Tesla Model S fatality in 2016 (Stilgoe 2018).”
BackgroundEthical governance is essential to building trust in robotics and artificial intelligence systems
Winfield, A. F. T., & Jirotka, M. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0085 · Cited by 493
“Given that real-world trials of driverless car autopilots have already resulted in several fatal accidents [46,47], there is clearly an urgent need for transparency in order to discover how and why those accidents occurred, remedy any technical or operational faults, and establish accountability.”
BackgroundFour Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them
Santoni de Sio, F., & Mecacci, G. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00450-x · Cited by 490
“…even when this is not the intention of their authors, these proposals even run the risk of fuelling the temptation of “solutionism” (Morozov, 2013); (Stilgoe, 2017), the belief that complex socio-technical and political problems can be “solved” (or avoided) by the introduction of new techniques.”
BackgroundInfluentialAI governance: themes, knowledge gaps and future agendas
Birkstedt, T., Minkkinen, M., Tandon, A., & Mäntymäki, M. (2023) · Internet Research · doi:10.1108/intr-01-2022-0042 · Cited by 309
Metadata
- Authors
- Stilgoe, J.
- Affiliations
- University College London (GB)
- Corresponding author
- Stilgoe, Jack
- ORCID
- Stilgoe, Jack
- Venue
- Social Studies of Science
- Publisher
- SAGE Publications
- Published
- 2017-11-21
- Volume
- 48
- Issue
- 1
- Pages
- 25-56
- ISSN
- 0036-8539 (ISSN-L)
- 0306-3127
- 1460-3659
- OpenAlex
- W2607757716
- Semantic Scholar
- 95615c6bce2123f12e39c3d9eb293ebb759501aa
- Cited by
- 375
- References
- 113
- Citations by year
- 2026: 26 · 2025: 25 · 2024: 39 · 2023: 70 · 2022: 53 · 2021: 58 · 2020: 50 · 2019: 34 · 2018: 16 · 2017: 3
- FWCI
- 64.51
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Social Studies of Science (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- autonomous vehicles, machine learning, social learning, technology governance, algorithmic learning, transport policy, public interest
- MeSH terms
- Accidents, Traffic, Social Learning, Machine Learning, Humans, Technology, Motor Vehicles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.