Telling autonomous systems what to do
Open accessAbstract
Recent progress in Artificial Intelligence, sensing and network technology, robotics, and (cloud) computing has enabled the development of intelligent autonomous machine systems. Telling such autonomous systems "what to do" in a responsible way, is a non-trivial task. For intelligent autonomous machines to function in human society and collaborate with humans, we see three challenges ahead affecting meaningful control of autonomous systems. First, autonomous machines are not yet capable of handling failures and unexpected situations. Providing procedures for all possible failures and situations is unfeasible because the state-action space would explode. Machines should therefore become self-aware (self-assessment, self-management) enabling them to handle unexpected situations when they arise. This is a challenge for the computer science community. Second, in order to keep (meaningful) control, humans come into a new role of providing intelligent autonomous machines with objectives or goal functions (including rules, norms, constraints and moral values), specifying the utility of every possible outcome of actions of autonomous machines. Third, in order to be able to collaborate with humans, autonomous systems will require an understanding of (us) humans (i.e., our social, cognitive, affective and physical behaviors) and the ability to engage in partnership interactions (such as explanations of task performances, and the establishment of joint goals and work agreements). These are new challenges for the cognitive ergonomics community.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Demanding explainability, observability, or transparency may cause AI systems to constrain their potential benefit for human society to what can be understood by humans.
“In addition, according to Werkhoven et al. (2018) demanding explainability observability, or transparency (Belkom, 2019; van den Bosch et al., 2019) may cause artificial intelligent systems to constrain their potential benefit for human society, to what can be understood by humans.”
From Human- versus Artificial Intelligence (2021) · cites this paper as Werkhoven et al. (2018)
At some point, humans may want to trust that goals can be achieved against less cost and better outcomes when they accept solutions even if they may be less transparent for humans.
“At some point, humans may want to trust that goals can be achieved against less cost and better outcomes, when we accept solutions even if they may be less transparent for humans (Werkhoven et al., 2018).”
From Human- versus Artificial Intelligence (2021) · cites this paper as Werkhoven et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
The Complete Works of Aristotle the Revised Oxford Translation
Aristotle, J., Barnes, J. A., Smith, W., & Ross (1984) · Princeton University Press eBooks · Cited by 2,382 · see citing works
(1986) · North-Holland eBooks · Cited by 1,883 · see citing works
Billinghurst, M., Clark, A., & Lee, G. (2015) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000049 · Cited by 1,189 · see citing works
Exoskeletons for industrial application and their potential effects on physical work load
de Looze, M. P., Bosch, T., Krause, F., et al. (2015) · Ergonomics · doi:10.1080/00140139.2015.1081988 · Cited by 1,067 · see citing works
Cognitive Systems Engineering: New Wine in New Bottles
Hollnagel, E., & Woods, D. (1983) · Int. J. Man Mach. Stud. · doi:10.1016/s0020-7373(83)80034-0 · Cited by 846 · see citing works
(2017) · Cited by 761 · see citing works
Meaningful Human Control over Autonomous Systems: A Philosophical Account
Santoni de Sio, F., & van den Hoven, J. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2018.00015 · Cited by 566 · see citing works
Billinghurst, M., Clark, A., & Lee, G. (2015) · now publishers, Inc. eBooks · doi:10.1561/9781601989215 · Cited by 536 · see citing works
Wittgenstein, L. (1965) · The Philosophical Review · doi:10.2307/2183526 · Cited by 498 · see citing works
Gangemi, A., & Presutti, V. (2009) · Springer eBooks · doi:10.1007/978-3-540-92673-3_10 · Cited by 476 · see citing works
Coactive Design: Designing Support for Interdependence in Joint Activity
Johnson, M., Bradshaw, J. M., Feltovich, P. J., Jonker, C. M., van Riemsdijk, M. B., & Sierhuis, M. (2014) · Journal of Human-Robot Interaction · doi:10.5898/jhri.3.1.johnson · Cited by 316 · see citing works
A human factors perspective on automated driving
Kyriakidis, M., de Winter, J., Stanton, N. A., Bellet, T., van Arem, B., Brookhuis, K., Martens, M., Bengler, K. J., Andersson, J., Merat, N., et al. (2017) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2017.1293187 · Cited by 314 · see citing works
The complete works of Aristotle: The revised Oxford translation.
Baumrin, B. H. (1986) · Metaphilosophy · doi:10.1111/j.1467-9973.1986.tb00397.x · Cited by 312 · see citing works
Morewedge, C. K., Yoon, H., Scopelliti, I., Symborski, C., Korris, J., & Kassam, K. (2015) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732215600886 · Cited by 259 · see citing works
Wittgenstein, L. (2014) · Cited by 242 · see citing works
Lewis, P. M., Ackland, H. M., Lowery, A. J., & Rosenfeld, J. V. (2015) · Brain Research · doi:10.1016/j.brainres.2014.11.020 · Cited by 228 · see citing works
Englund, C., Chen, L., Ploeg, J., et al. (2016) · IEEE Wireless Communications · doi:10.1109/mwc.2016.7553038 · Cited by 139 · see citing works
Behavioral Ethics and Teaching Ethical Decision Making
Drumwright, M. E., Prentice, R. A., & Biasucci, C. (2015) · Decision Sciences Journal of Innovative Education · doi:10.1111/dsji.12071 · Cited by 103 · see citing works
Tielman, M. L., Neerincx, M. A., Bidarra, R., Kybartas, B., & Brinkman, W.-P. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0771-y · Cited by 94 · see citing works
Self-citationUsing Perceptual and Cognitive Explanations for Enhanced Human-Agent Team Performance
Neerincx, M. A., van der Waa, J., Kaptein, F., & van Diggelen, J. (2018) · Interacción · doi:10.1007/978-3-319-91122-9_18 · Cited by 88 · see citing works
Self-citationDesign and Evaluation of Explainable BDI Agents
Harbers, M., van den Bosch, K., & Meyer, J.-J. (2010) · 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology · doi:10.1109/wi-iat.2010.115 · Cited by 70 · see citing works
Situated Cognitive Engineering for Complex Task Environments
Maarten, S. J., Militello, L. G., Ormerod, T., & Raanan, L. (2017) · CRC Press eBooks · doi:10.1201/9781315597584-22 · Cited by 69 · see citing works
Kaptein, F., Broekens, J., Hindriks, K., & Neerincx, M. A. (2017) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2017.8172376 · Cited by 64 · see citing works
Self-citationShively, R. J., Lachter, J., Brandt, S. L., Matessa, M., Battiste, V., & Johnson, W. W. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-60642-2_1 · Cited by 56 · see citing works
Hiremath, S. V., Chen, W., Wang, W., Foldes, S., Yang, Y., Tyler-Kabara, E. C., Collinger, J. L., & Boninger, M. L. (2015) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2015.00040 · Cited by 53 · see citing works
Navigating virtual mazes: The benefits of audiovisual landmarks
Werkhoven, P. J., van Erp, J. B. F., & Philippi, T. G. (2014) · Displays · doi:10.1016/j.displa.2014.04.001 · Cited by 28 · see citing works
Self-citationA multi-objective approach to evolving platooning strategies in intelligent transportation systems
van Willigen, W., Haasdijk, E., & Kester, L. J. H. M. (2013) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2463372.2463534 · Cited by 26 · see citing works
Self-citationValue sensitive design of a virtual assistant for workload harmonization in teams
Harbers, M., & Neerincx, M. A. (2017) · Cognition Technology & Work · doi:10.1007/s10111-017-0408-4 · Cited by 22 · see citing works
Self-citationDynamic Task Allocation for Human-robot Teams
Giele, T. R. A., Mioch, T., Neerincx, M. A., & Meyer, J. C. (2015) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0005178001170124 · Cited by 18 · see citing works
Self-citationGonzález‐Ortega, J., Radovic, V. M., & Insua, D. R. (2017) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-319-65052-4_10 · Cited by 16 · see citing works
Hybrid Strategies Towards Safe “Self-Aware” Superintelligent Systems
Aliman, N.-M., & Kester, L. J. H. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-97676-1_1 · Cited by 14 · see citing works
Self-citationReducing Stress by Bonding with a Social Robot
Ligthart, M. E. U., Hindriks, K. V., & Neerincx, M. A. (2018) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3173386.3176904 · Cited by 13 · see citing works
Self-citationA Design Pattern for Working Agreements in Human-Autonomy Teaming
Gutzwiller, R. S., Espinosa, S. H., Kenny, C., & Lange, D. S. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-60591-3_2 · Cited by 12 · see citing works
Ontology Design for Task Allocation and Management in Urban Search and Rescue Missions
Saad, E., Hindriks, K. V., & Neerincx, M. A. (2018) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0006661106220629 · Cited by 12 · see citing works
Self-citationImproving Adaptive Human-Robot Cooperation through Work Agreements
Mioch, T., Peeters, M. M. M., & Neerincx, M. A. (2018) · IEEE RO-MAN · doi:10.1109/roman.2018.8525776 · Cited by 12 · see citing works
Self-citationGuidelines for Tree-based Collaborative Goal Setting
Peters, R., Broekens, J., & Neerincx, M. A. (2017) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3025171.3025188 · Cited by 7 · see citing works
Self-citationAn Intelligent Operator Support System for Dynamic Positioning
van Diggelen, J., van den Broek, H., Schraagen, J. M., & van der Waa, J. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-60204-2_6 · Cited by 6 · see citing works
Player-Avatar Link: Interdisciplinary Embodiment Perspectives
Rosa, N., Hürst, W., Veltkamp, R. C., & Werkhoven, P. J. (2017) · Springer eBooks · doi:10.1007/978-3-319-08234-9_110-1 · Cited by 6 · see citing works
Self-citationEthical Regulators and Super-Ethical SystemsPreprint
Ashby, M. (2019) · Preprints.org · doi:10.20944/preprints201907.0349.v1 · Cited by 6 · see citing works
Proceedings of the 16th International Conference on Agents and Artificial Intelligence
Cismaru, A. M. (2024) · doi:10.5220/0000183700003636 · Cited by 6 · see citing works
Maximising effectiveness of distributed mobile observation systems in dynamic situations
Kester, L. J. H. M., & Ditzel, M. (2014) · TNO Repository · Cited by 5 · see citing works
Self-citationBosse, T., Breebaart, L., van Diggelen, J., et al. (2017) · International Journal of Agent-Oriented Software Engineering · doi:10.1504/ijaose.2017.087656 · Cited by 5 · see citing works
Producing near-real-time intelligence: predicting the world of tomorrow
Barros, A. I., van den Broek, A. C., van Dalen, J. A., et al. (2014) · Data Archiving and Networked Services (DANS) · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- J van Diggelen, H van den Broek, JM Schraagen, and J van der Waa. 2017. An Intelligent Operator Support System for Dynamic Positioning. Int. Conference on Applied Human Factors and Ergonomics, 48-59.
- T. Miller. 2017. Explanation in artificial intelligence: Insights from the social sciences. arXiv preprint arXiv:1706.07269 · arXiv
- M.A. Neerincx, F. Kaptein, M.A. Van Bekkum, H. Krieger, B. Kiefer, R. Peters, and M. Sapelli. 2016. Ontologies for social, cognitive and affective agent-based support of child's diabetes self-management. In Proc. ECAI 16, 35-38.
- J. Van der Waa, J. Van Diggelen, and M.A. Neerincx. 2018. The design and validation of an intuitive certainty measure. Proceedings of IUI 2018 Workshop on Explainable Smart Systems.
- J. Van der Waa, M.J. Robeer, J. Van Diggelen, M.J.S. Brinkhuis, and M.A. Neerincx. 2018. Contrastive Explanations with Local Foil Trees. IJCAI 2018 Workshop on Human Interpretability in Machine Learning (WHI).
Cited by
Sorted by citations
Human- versus Artificial Intelligence
Korteling, J. E. (., van de Boer-Visschedijk, G. C., Blankendaal, R. A. M., Boonekamp, R. C., & Eikelboom, A. R. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.622364 · Cited by 619 · see citing works
“At some point, humans may want to trust that goals can be achieved against less cost and better outcomes, when we accept solutions even if they may be less transparent for humans (Werkhoven et al., 2018).”
BackgroundSix challenges for human-AI Co-learning
van den Bosch, K., Schoonderwoerd, T., Blankendaal, R., & Neerincx, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22341-0_45 · Cited by 50 · see citing works
Self-citation
- Authors
- Werkhoven, P., Kester, L., & Neerincx, M.
- Affiliations
- Netherlands Organisation for Applied Scientific Research (NL)
- Venue
- Proceedings of the 36th European Conference on Cognitive Ergonomics
- Publisher
- ACM
- Published
- 2018-09-05
- Pages
- 1-8
- OpenAlex
- W2887179446
- Cited by
- 19
- References
- 47
- Citations by year
- 2026: 2 · 2024: 1 · 2023: 4 · 2022: 1 · 2021: 3 · 2020: 2 · 2019: 6
- FWCI
- 4.92
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- European Conference on Cognitive Ergonomics (publishedVersion)
- TNO Repository (submittedVersion)
- Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- autonomous systems, human-automation collaboration, self-awareness, self-management, failure management, moral values, human understanding, cognitive ergonomics, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
