Mediating artificial intelligence developments through negative and positive incentives
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
The field of Artificial Intelligence (AI) is going through a period of great expectations, introducing a certain level of anxiety in research, business and also policy. This anxiety is further energised by an AI race narrative that makes people believe they might be missing out. Whether real or not, a belief in this narrative may be detrimental as some stake-holders will feel obliged to cut corners on safety precautions, or ignore societal consequences just to “win”. Starting from a baseline model that describes a broad class of technology races where winners draw a significant benefit compared to others (such as AI advances, patent race, pharmaceutical technologies), we investigate here how positive (rewards) and negative (punishments) incentives may beneficially influence the outcomes. We uncover conditions in which punishment is either capable of reducing the development speed of unsafe participants or has the capacity to reduce innovation through over-regulation. Alternatively, we show that, in several scenarios, rewarding those that follow safety measures may increase the development speed while ensuring safe choices. Moreover, in the latter regimes, rewards do not suffer from the issue of over-regulation as is the case for punishment. Overall, our findings provide valuable insights into the nature and kinds of regulatory actions most suitable to improve safety compliance in the contexts of both smooth and sudden technological shifts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Altruistic punishment in humans
Fehr, E., & Gächter, S. (2002) · Nature · doi:10.1038/415137a · Cited by 5,616 · cited 2×
“Incentives such as punishment and rewards have been shown to provide important mechanisms to promote the emergence of positive behaviour (such as cooperation and fairness) in the context of social dilemmas [25–35].”
BackgroundResultEvolutionary Dynamics: Exploring the Equations of Life
NOWAK, M. A. (2006) · doi:10.2307/j.ctvjghw98 · Cited by 2,438
Coordinated Punishment of Defectors Sustains Cooperation and Can Proliferate When Rare
Boyd, R., Gintis, H., & Bowles, S. (2010) · Science · doi:10.1126/science.1183665 · Cited by 613
Social learning promotes institutions for governing the commons
Sigmund, K., De Silva, H., Traulsen, A., & Hauert, C. (2010) · Nature · doi:10.1038/nature09203 · Cited by 580 · cited 1×
“Future studies may look at more complex incentivising mechanisms [50] such as reputation and public image manipulation [64, 65], emotional motives of guilt and apology-forgiveness [60, 66], institutional and coordinated incentives [34, 46], and the subtle combination of different forms of incentive (e.”
BackgroundViewpoint: when will AI exceed human performance? evidence from AI experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11222 · Cited by 579
The Social Control of Technology
Mayne, A. J. (1982) · Journal of the Operational Research Society · doi:10.2307/2634327 · Cited by 577 · cited 1×
“This is the case for many kinds of technological races especially when data about the effect of a new technology is usually lacking and only becomes available when it has been created and used enough (see the Collingridge Dilemma [48]), as are the cases of the domain supremacy race through AI [18,19] and the race for…”
MethodSigmund, K. (2010) · doi:10.1515/9781400832255 · Cited by 570 · cited 4×
“umes, upon completion of each round, that there is a probability !that another development round is required to reach DSAI|which results in an average number W= (1 !) 1 of rounds per competition/race [12]. We thus do not make any assumption about the time required to reach DSAI in a given domain. Yet once the race end…”
BackgroundMethodResultInfluentialThe race for coronavirus vaccines: a graphical guide
Callaway, E. (2020) · Nature · doi:10.1038/d41586-020-01221-y · Cited by 492
Evolutionary cycles of cooperation and defection
Imhof, L. A., Fudenberg, D., & Nowak, M. A. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0502589102 · Cited by 381
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 2×
“Future studies may look at more complex incentivising mechanisms [47] such as reputation and public image manipulation [51,52], emotional motives of guilt and apology-forgiveness [53,54], institutional and coordinated incentives [28, 36], and the subtle combination of different forms of incentive (e.”
BackgroundWhen Will AI Exceed Human Performance? Evidence from AI Experts
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., & Evans, O. (2017) · arXiv · doi:10.48550/arxiv.1705.08807 · Cited by 285 · cited 2×
“This is the case for many kinds of technological races especially when data about the effect of a new technology is usually lacking and only becomes available when it has been created and used enough (see the Collingridge Dilemma [52]), as are the cases of the domain supremacy race through AI [21, 22] and the race for…”
BackgroundMethodA bottom-up institutional approach to cooperative governance of risky commons
Vasconcelos, V. V., Santos, F. C., & Pacheco, J. M. (2013) · Nature Climate Change · doi:10.1038/nclimate1927 · Cited by 183
Responses to catastrophic AGI risk: a survey
Sotala, K., & Yampolskiy, R. V. (2015) · Physica Scripta · doi:10.1088/0031-8949/90/1/018001 · Cited by 121
Emergence of Social Punishment and Cooperation through Prior Commitments
Han, T. A. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10120 · Cited by 33 · cited 1×
“Incentives such as punishment and rewards have been shown to provide important mechanisms to promote the emergence of positive behaviour (such as cooperation and fairness) in the context of social dilemmas [25–35].”
BackgroundSelf-citation
Cited by
Combination of institutional incentives for cooperative governance of risky commons
Sun, W., Liu, L., Chen, X., Szolnoki, A., & Vasconcelos, V. V. (2021) · iScience · doi:10.1016/j.isci.2021.102844 · Cited by 80
“In a recent work (Han et al., 2021), it is shown that peer punishment is more efficient than peer reward in promoting safety compliance and in mitigating the risk from Artificial Intelligence unsafe development.”
BackgroundEarly exclusion leads to cyclical cooperation in repeated group interactions
Liu, L., Xiao, Z., Chen, X., & Szolnoki, A. (2022) · Journal of The Royal Society Interface · doi:10.1098/rsif.2021.0755 · Cited by 50
Zhang, X., Zhou, W., Hu, W., Zhou, S., Hu, X., & Yang, L. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.102900 · Cited by 1
Metadata
- Authors
- Han, T. A., Pereira, L. M., Lenaerts, T., & Santos, F. C.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-01-26
- Volume
- 16
- Issue
- 1
- Pages
- e0244592
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W3125059163
- Cited by
- 57
- References
- 14
- FWCI
- 10.49
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.