A Research Agenda for Hybrid Intelligence: Augmenting Human Intellect With Collaborative, Adaptive, Responsible, and Explainable Artificial Intelligence
Open accessAbstract
We define hybrid intelligence (HI) as the combination of human and machine intelligence, augmenting human intellect and capabilities instead of replacing them and achieving goals that were unreachable by either humans or machines. HI is an important new research focus for artificial intelligence, and we set a research agenda for HI by formulating four challenges.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Complementarity between AI and MDs advises a collaboration between human and artificial minds, fostering a hybrid intelligence that could deliver outcomes superior to those reached by each mind alone.
“Such complementarity advises a collaboration between human and artificial minds, fostering a “hybrid intelligence” that could deliver outcomes superior to those reached by each mind alone2,3.”
From Experimental evidence of effective human–AI collaboration in medical decision-making (2022) · cites this paper as [3]
What is envisaged is teaming humans together with artificial intelligence to exploit the advantages of hybrid intelligence.
“what is envisaged is teaming humans together with artificial intelligence to exploit the advantages of hybrid intelligence2,3.”
From Experimental evidence of effective human–AI collaboration in medical decision-making (2022) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Hutter, F., Kotthoff, L., & Vanschoren, J. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,552 · see citing works
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · see citing works
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · see citing works
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515 · see citing works
Handbook of Ethics, Values, and Technological Design
van den Hoven, J., Vermaas, P. E., & van de Poel, I. (2015) · Springer eBooks · doi:10.1007/978-94-007-6970-0 · Cited by 314 · see citing works
Team IHMC's Lessons Learned from the DARPA Robotics Challenge Trials
Johnson, M., Shrewsbury, B., Bertrand, S., et al. (2015) · Journal of Field Robotics · doi:10.1002/rob.21571 · Cited by 254 · see citing works
A review of explanation methods for Bayesian networks
Lacave, C., & Díez, F. J. (2002) · The Knowledge Engineering Review · doi:10.1017/s026988890200019x · Cited by 240 · see citing works
Hutter, F., Kotthoff, L., & Vanschoren, J. (2020) · Cited by 218 · see citing works
Designing a human-centered, multimodal GIS interface to support emergency management
Rauschert, I., Agrawal, P., Sharma, R., et al. (2002) · ACM International Symposium on Advances in Geographic Information Systems (ACM GIS) · doi:10.1145/585147.585172 · Cited by 151 · see citing works
Reciprocity Outperforms Conformity to Promote Cooperation
Romano, A., & Balliet, D. P. (2017) · Psychological Science · doi:10.1177/0956797617714828 · Cited by 95 · see citing works
Self-citationModelling Agent Societies: Co-ordination Frameworks and Institutions
Dignum, V., & Dignum, F. P. M. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45329-6_21 · Cited by 90 · see citing works
Self-citationCook, M. B., & Smallman, H. S. (2008) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x354183 · Cited by 86 · see citing works
How much does it help to know what she knows you know? An agent-based simulation study
de Weerd, H., Verbrugge, R., & Bart Verheij, A. (2013) · Artificial Intelligence · doi:10.1016/j.artint.2013.05.004 · Cited by 81 · see citing works
Self-citationA Boxology of Design Patterns forHybrid Learningand Reasoning Systems
van Harmelen, F., & ten Teije, A. (2019) · Journal of Web Engineering · doi:10.13052/jwe1540-9589.18133 · Cited by 69 · see citing works
Interactive Learning of Grounded Verb Semantics towards Human-Robot Communication
She, L., & Chai, J. Y. (2017) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p17-1150 · Cited by 53 · see citing works
A Child and a Robot Getting Acquainted - Interaction Design for Eliciting Self-Disclosure
Ligthart, M. E. U., Fernhout, T., Neerincx, M. A., van Bindsbergen, K. L. A., Grootenhuis, M. A., & Hindriks, K. V. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/axmx9659 · Cited by 29 · see citing works
Self-citationDependence in games and dependence games
Grossi, D., & Turrini, P. (2011) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-011-9176-3 · Cited by 17 · see citing works
Self-citationMeasuring Moral Acceptability in E-deliberation
Verdiesen, I., Dignum, V., & van den Hoven, J. (2018) · ACM Transactions on Internet Technology · doi:10.1145/3183324 · Cited by 12 · see citing works
Self-citationChallenges of Human‐Aware AI Systems
Subbarao Kambhampati, B. (2020) · AI Magazine · doi:10.1609/aimag.v41i3.5257 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- M.A. de Graaf, and B F. Malle, "How people explain action (and Autonomous Intelligent Systems should too)," in Proc. Association Advancement Artificial Intelligence Fall Symp. (AAAI), 2017, pp. 19-26.
Cited by
Sorted by citations
Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13544 · Cited by 805 · see citing works
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543 · see citing works
Leading teachers' perspective on teacher-AI collaboration in education
Kim, J. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12109-5 · Cited by 376 · see citing works
Towards hybrid <scp>human‐AI</scp> learning technologies
Molenaar, I. (2022) · European Journal of Education · doi:10.1111/ejed.12527 · Cited by 348 · see citing works
Human and artificial intelligence collaboration for socially shared regulation in learning
Järvelä, S., Nguyen, A., & Hadwin, A. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13325 · Cited by 342 · see citing works
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 235 · see citing works
Molenaar, I. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100070 · Cited by 217 · see citing works
How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129 · see citing works
Sokol, K., Fackler, J., & Vogt, J. E. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01725-9 · Cited by 105 · see citing works
Alert Fatigue in Security Operations Centres: Research Challenges and Opportunities
Tariq, S., Baruwal Chhetri, M., Nepal, S., & Paris, C. (2025) · ACM Computing Surveys · doi:10.1145/3723158 · Cited by 101 · see citing works
Lodge, J. M., Yang, S., Furze, L., & Dawson, P. (2023) · Learning: Research and Practice · doi:10.1080/23735082.2023.2261106 · Cited by 90 · see citing works
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73 · see citing works
Cabitza, F., Campagner, A., Angius, R., Natali, C., & Reverberi, C. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581095 · Cited by 51 · see citing works
Hybrid Intelligence Failure Analysis for Industry 4.0: A Literature Review and Future Prospective
Mokhtarzadeh, M., Rodríguez-Echeverría, J., Semanjski, I., & Gautama, S. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-024-02376-5 · Cited by 35 · see citing works
Sensemaking with AI: how trust influences human–AI collaboration in health and creative industries
Daly, S. J., Hearn, G., & Papageorgiou, K. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.101346 · Cited by 27 · see citing works
Toward a science of human–AI teaming for decision making: a complementarity framework
Gonzalez, C., Donahue, K., Goldstein, D. G., Heidari, H., Jalali, M. S., Schelble, B., Singh, A., & Woolley, A. W. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag030 · Cited by 33 · see citing works
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13 · see citing works
Indongo, N. N., Smith, R. C., Winschiers-Theophilus, H., Kandjengo, S., & Mbinge, U. (2026) · Proceedings of the 19th Participatory Design Conference 2026, Vol. 1: Full Papers · doi:10.1145/3796624.3796667
- Authors
- Akata, Z., Balliet, D., de Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., Hung, H., Jonker, C., Monz, C., Neerincx, M., Oliehoek, F., Prakken, H., Schlobach, S., van der Gaag, L., van Harmelen, F., van Hoof, H., van Riemsdijk, B., van Wynsberghe, A., Verbrugge, R., Verheij, B., Vossen, P., & Welling, M.
- Affiliations
- University of Tübingen (DE)
- Vrije Universiteit Amsterdam (NL)
- Artificial Intelligence, University of Amsterdam
- Utrecht University (NL)
- Umeå University (SE)
- Artificial Intelligence, Vrije Universiteit Amsterdam
- University of Groningen (NL)
- Leiden University (NL)
- Delft University of Technology (NL)
- University of Amsterdam (NL)
- Vrije Universiteit Amsterdam
- Dalle Molle Institute, Lugano, Switzerland
- University of Amsterdam
- University of Twente
- Ethics and Technology, Technology University Delft
- University of Groningen
- ORCID
- Venue
- Computer
- Published
- 2020-08
- Volume
- 53
- Issue
- 8
- Pages
- 18-28
- ISSN
- 0018-9162 (ISSN-L)
- 1558-0814
- OpenAlex
- W3046399757
- Cited by
- 415
- References
- 20
- Citations by year
- 2026: 86 · 2025: 83 · 2024: 104 · 2023: 60 · 2022: 60 · 2021: 21 · 2012: 1
- FWCI
- 38.51
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Computer (publishedVersion, cc-by)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, cc-by)
- VU Research Portal (publishedVersion, cc-by)
- University of Twente Research Information (publishedVersion, cc-by)
- Research Repository (Delft University of Technology) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Funders
- Dutch Ministry of Education Culture and Science - OCW (024.004.022)
- Topics
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- hybrid intelligence, human-machine collaboration, explainable AI, responsible AI, adaptive AI, collaborative AI, research agenda
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
