CoChat
Sign Up
Cited by 279Information Systems Theories and ImplementationHuman-Automation Interaction and SafetyUsability and User Interface Design

The Future of Human-AI Collaboration: A Taxonomy of Design Knowledge for Hybrid Intelligence Systems

Open access
Dellermann, Dominik, Calma, Adrian, Lipusch, Nikolaus, Weber, Thorsten, Weigel, Sascha, Ebel, Philipp·2019·Proceedings of the Annual Hawaii International Conference on System Sciences
DOI 10.24251/hicss.2019.034
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent technological advances, especially in the field of machine learning, provide astonishing progress on the road towards artificial general intelligence. However, tasks in current real-world business applications cannot yet be solved by machines alone. We, therefore, identify the need for developing socio-technological ensembles of humans and machines. Such systems possess the ability to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results and continuously improve by learning from each other. Thus, the need for structured design knowledge for those systems arises. Following a taxonomy development method, this article provides three main contributions: First, we present a structured overview of interdisciplinary research on the role of humans in the machine learning pipeline. Second, we envision hybrid intelligence systems and conceptualize the relevant dimensions for system design for the first time. Finally, we offer useful guidance for system developers during the implementation of such applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.24251/hicss.2019.034

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The more highly developed concepts of collaborative AI, responsible AI, or hybrid intelligence systems see a long-term role for humans in the application of AI systems.

    “However, the more highly developed concepts of”

    From Keeping the organization in the loop: a socio-technical extension of human-centered artificial intelligence (2023) · cites this paper as Dellermann et al. 2019

    Section 2.3, “Hybrid Intelligence,” argues that humans and machines will collaborate continuously and describes hybrid intelligence as a likely division of labor for years or decades. That supports a long-term human role in hybrid intelligence, but the paper does not establish the same claim for responsible AI as a distinct concept. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • To develop an interaction between 4.1 and 4.2, a complementary combination of human and AI strengths is to be pursued.

    “To develop an interaction between 4.1 and 4.2, a complementary combination of human and AI strengths is to be pursued (Dellermann et al. 2019;Jarrahi 2018).”

    From Keeping the organization in the loop: a socio-technical extension of human-centered artificial intelligence (2023) · cites this paper as Dellermann et al. 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 60

  1. B. Goertzel and C. Pennachin, Artificial general intelligence, vol. 2. Springer, 2007.
  2. L. Fridman, B. Reimer, B. Mehler, and W. T. Freeman, "Cognitive Load Estimation in the Wild," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems -CHI '18, (New York, USA), pp. 1-9, 2018.
  3. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529-533, 2015.
  4. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. Van Den Driessche, T. Graepel, and D. Hassabis, "Mastering the game of Go without human knowledge," nature, vol. 529, no. 7587, pp. 484-489, 2016.
  5. E. Kamar, "Directions in hybrid intelligence: Complementing AI systems with human intelligence," in IJCAI International Joint Conference on Artificial Intelligence, 2016.
  6. C. Müller-Schloer and S. Tomforde, Organic Computing Technical Systems for Survival in the Real World. Autonomic Systems, Cham: Springer International Publishing, 2017.
  7. A. Kurakin, I. Goodfellow, and S. Bengio, "Adversarial examples in the physical world," arXiv preprint arXiv:1607.02533, 2016. · arXiv
  8. B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman, "Building machines that learn and think like people," Behavioral and Brain Sciences, vol. 40, 2017.
  9. R. C. Nickerson, U. Varshney, and J. Muntermann, "A method for taxonomy development and its application in information systems," European Journal of Information Systems, vol. 22, no. 3, pp. 336-359, 2013.
  10. T. M. Mitchell et al., "Machine learning. 1997," Burr Ridge, IL: McGraw Hill, vol. 45, no. 37, pp. 870-877, 1997.
  11. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
  12. M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter, "Efficient and robust automated machine learning," Advances in Neural Information Processing Systems, pp. 2962-2970, 2015.
  13. R. G. Gomes, P. Welinder, A. Krause, and P. Perona, "Crowdclustering," Advances in neural information processing systems, pp. 558-566, 2011.
  14. J. Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan, "Human decisions and machine predictions," The quarterly journal of economics, vol. 133, no. 1, pp. 237-293, 2017.
  15. A. Holzinger, "Interactive machine learning for health informatics: when do we need the human-in-the-loop?," Brain Informatics, vol. 3, no. 2, pp. 119-131, 2016.
  16. P. Y. Simard, S. Amershi, D. M. Chickering, A. E. Pelton, S. Ghorashi, C. Meek, G. Ramos, J. Suh, J. Verwey, M. Wang, et al., "Machine teaching: A new paradigm for building machine learning systems," arXiv preprint arXiv:1707.06742, 2017. · arXiv
  17. J. Attenberg, P. Ipeirotis, and F. Provost, "Beat the Machine," Journal of Data and Information Quality, vol. 6, no. 1, pp. 1-17, 2015.
  18. S. Gregor, "Explanations from knowledge-based systems and cooperative problem solving: an empirical study," International Journal of Human-Computer Studies, vol. 54, no. 1, pp. 81-105, 2001.
  19. "Cobi Communitysourcing Large-Scale Conference Scheduling." http://projectcobi.com last access: 2018-06-14.
  20. J. Webster and R. T. Watson, "Analyzing the past to prepare for the future: Writing a literature review," MIS Quarterly, vol. 26, no. 2, pp. 8-23, 2002.
  21. J. vom Brocke, A. Simons, B. Niehaves, K. Riemer, R. Plattfaut, A. Cleven, J. V. Brocke, and K. Reimer, "Reconstructing the Giant: On the Importance of Rigour in Documenting the Literature Search Process," 17th European Conference on Information Systems, 2009.
  22. U. Kayande, A. De Bruyn, G. L. Lilien, A. Rangaswamy, and G. H. Van Bruggen, "How incorporating feedback mechanisms in a dss affects dss evaluations," Information Systems Research, vol. 20, no. 4, pp. 527-546, 2009.
  23. W. M. Reynolds, G. E. Miller, and I. B. Weiner, Handbook of psychology. Wiley, 2013.
  24. P. Choudhary, A. Jain, and R. Baijal, "Unravelling airbnb predicting price for new listing," arXiv preprint arXiv:1805.12101, 2018. · arXiv
  25. C. A. Gomez-Uribe and N. Hunt, "The netflix recommender system: Algorithms, business value, and innovation," ACM Transactions on Management Information Systems (TMIS), vol. 6, no. 4, p. 13, 2016.
  26. D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty, "Building Watson: An Overview of the DeepQA Project," AI Magazine, vol. 31, p. 59, jul 2010.
  27. J. Cheng, J. Teevan, S. T. Iqbal, and M. S. Bernstein, "Break It Down: A Comparison of Macro-and Microtasks," Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15, pp. 4061-4064, 2015.
  28. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. Le, and A. Kurakin, "Large-scale evolution of image classifiers," arXiv preprint arXiv:1703.01041, 2017. · arXiv
  29. E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, "Regularized evolution for image classifier architecture search," arXiv preprint arXiv:1802.01548, 2018. · arXiv
  30. S. Doroudi, E. Kamar, E. Brunskill, and E. Horvitz, "Toward a Learning Science for Complex Crowdsourcing Tasks," in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems -CHI '16, (New York, USA), pp. 2623-2634, ACM Press, 2016.
  31. S. Milli, P. Abbeel, and I. Mordatch, "Interpretable and pedagogical examples," arXiv preprint arXiv:1711.00694, 2017. · arXiv
  32. S. Carter and M. Nielsen, "Using Artificial Intelligence to Augment Human Intelligence," Distill, vol. 2, no. 12, 2017.
  33. K. P. Murphy, Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
  34. X. Zhu, "Semi-supervised learning literature survey," Computer Science, University of Wisconsin-Madison, vol. 2, no. 3, pp. 4-44, 2006.
  35. R. M. Hogarth, "On the learning of intuition," Intuition in judgment and decision making, pp. 111-126, 2011.
  36. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng, "Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks," Proceedings of the conference on empirical methods in natural language processing, pp. 254-263, 2008.
  37. V. C. Raykar, S. Yu, L. H. Zhao, G. Hermosillo Valadez, C. Florin, L. Bogoni, L. Moy, and L. M. Org, "Learning From Crowds," Journal of Machine Learning Research, vol. 11, pp. 1297-1322, 2010.
  38. B. Nushi, E. Kamar, E. Horvitz, and D. Kossmann, "On human intellect and machine failures: Troubleshooting integrative machine learning systems.," AAAI, pp. 1017-1025, 2017.
  39. K. Pei, Y. Cao, J. Yang, and S. Jana, "Towards practical verification of machine learning: The case of computer vision systems," arXiv preprint arXiv:1712.01785, 2017. · arXiv
  40. G. Li, Y. Zheng, J. Fan, J. Wang, and R. Cheng, "Crowdsourced data management: Overview and challenges," in Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1711-1716, ACM, 2017.
  41. A. Chakarov, A. Nori, S. Rajamani, S. Sen, and D. Vijaykeerthy, "Debugging machine learning tasks," arXiv preprint arXiv:1603.07292, 2016. · arXiv
  42. T. Kulesza, S. Stumpf, M. Burnett, W.-K. Wong, Y. Riche, T. Moore, I. Oberst, A. Shinsel, and K. McIntosh, "Explanatory Debugging: Supporting End-User Debugging of Machine-Learned Programs," in 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, pp. 41-48, IEEE, sep 2010.
  43. K. Patel, N. Bancroft, S. M. Drucker, J. Fogarty, A. J. Ko, and J. Landay, "Gestalt: integrated support for implementation and analysis in machine learning," Proceedings of the 23nd annual ACM symposium on User interface software and technology -UIST '10, pp. 37-46, 2010.
  44. J. C. Chang, S. Amershi, and E. Kamar, "Revolt," in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems -CHI '17, 2017.
  45. J. C. Chang, A. Kittur, and N. Hahn, "Alloy: Clustering with Crowds and Computation," Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems -CHI '16, 2016.
  46. D. Dellermann, N. Lipusch, and P. Ebel, "Developing Design Principles for a Crowd-Based Business Model Validation System," in International Conference on Design Science Research in Information Systems, pp. 163-178, 2017.
  47. D. Dellermann, N. Lipusch, P. Ebel, K. M. Popp, and J. M. Leimeister, "Finding the Unicorn : Predicting Early Stage Startup Success through a Hybrid Intelligence Method," ICIS 2017 Proceedings, pp. 1-12, 2017.
  48. A. J. Quinn and B. B. Bederson, "Human computation: a survey and taxonomy of a growing field," pp. 1403-1412, ACM, 2011.
  49. A. P. Dawid and A. M. Skene, "Maximum likelihood estimation of observer error-rates using the em algorithm," Applied statistics, vol. 28, pp. 20-28, 1979.
  50. H.-C. Kim and Z. Ghahramani, "Bayesian Classifier Combination," in Proceedings of the International Conference on Artifical Intelligence and Statistics (AISTATS), 2012.
  51. K. A. M. Kamar, Z. A. Hamid, and N. Dzulkalnine, "Industrialised Building System (IBS) construction: Measuring the perception of contractors in Malaysia," in BEIAC 2012 -2012 IEEE Business, Engineering and Industrial Applications Colloquium, 2012.
  52. M. Kosinski, Y. Bachrach, P. Kohli, D. Stillwell, and T. Graepel, "Manifestations of user personality in website choice and behaviour on online social networks," Machine learning, vol. 95, no. 3, pp. 357-380, 2014.
  53. A. Segal, K. sGal, E. Kamar, E. Horvitz, and G. Miller, "Optimizing interventions via offline policy evaluation: Studies in citizen science," AAAI 2018, 2018.
  54. L. Von Ahn and L. Dabbish, "Designing games with a purpose," Communications of the ACM, vol. 51, no. 8, pp. 58-67, 2008.
  55. J. W. Vaughan, "Making better use of the crowd: How crowdsourcing can advance machine learning research," Journal of Machine Learning Research, vol. 18, no. 193, pp. 1-46, 2018.
  56. A. Bernardo, A. Cervero, M. Esteban, E. Tuero, J. R. Casanova, and L. S. Almeida, "Freshmen Program Withdrawal: Types and Recommendations.," Frontiers in psychology, vol. 8, p. 1544, 2017.
  57. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in European conference on computer vision, pp. 740-755, 2014.
  58. V. S. Sheng, F. Provost, and P. G. Ipeirotis, "Get another label? improving data quality and data mining using multiple, noisy labelers," Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 614-622, 2008.
  59. L. Zhao, Y. Zhang, and G. Sukthankar, "An active learning approach for jointly estimating worker performance and annotation reliability with crowdsourced data," arXiv preprint arXiv:1401.3836, 2014. · arXiv
  60. G. Kou, Y. Peng, and G. Wang, "Evaluation of clustering algorithms for financial risk analysis using mcdm methods," Information Sciences, vol. 275, pp. 1-12, 2014.

Cited by

Sorted by citations

Authors
Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., & Ebel, P.
Affiliations
University of Kassel (DE)
ORCID
Venue
Proceedings of the Annual Hawaii International Conference on System Sciences
Publisher
Hawaii International Conference on System Sciences
Published
2019
ISSN
  • 1530-1605 (ISSN-L)
  • 2572-6862
DOI
10.24251/hicss.2019.034
OpenAlex
W2901418095
Cited by
279
References
7
Citations by year
2026: 43 · 2025: 53 · 2024: 57 · 2023: 43 · 2022: 31 · 2021: 28 · 2020: 20 · 2019: 4
FWCI
59.22
Type
conference paper
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Hawaii International Conference on System Sciences (HICSS) (publishedVersion, cc-by-nc-nd)
  • Alexandria (UniSG) (University of St.Gallen) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Alexandria (UniSG) (University of St.Gallen) (submittedVersion)
  • ScholarSpace (University of Hawaii at Manoa) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
human-AI collaboration, artificial general intelligence, socio-technical systems, human-in-the-loop learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review