CoChat
Sign Up
PreprintCited by 456Information Systems Theories and ImplementationHuman-Automation Interaction and SafetyUsability and User Interface Design

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems

Open access
Dellermann, Dominik, Calma, Adrian, Lipusch, Nikolaus, Weber, Thorsten, Weigel, Sascha, Ebel, Philipp·2021·arXiv
DOI 10.48550/arxiv.2105.03354arXiv 2105.03354
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent technological advances, especially in the field of machine learning, provide astonishing progress on the road towards artificial general intelligence. However, tasks in current real-world business applications cannot yet be solved by machines alone. We, therefore, identify the need for developing socio-technological ensembles of humans and machines. Such systems possess the ability to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results and continuously improve by learning from each other. Thus, the need for structured design knowledge for those systems arises. Following a taxonomy development method, this article provides three main contributions: First, we present a structured overview of interdisciplinary research on the role of humans in the machine learning pipeline. Second, we envision hybrid intelligence systems and conceptualize the relevant dimensions for system design for the first time. Finally, we offer useful guidance for system developers during the implementation of such applications.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2105.03354

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 . Figure 1. Machine learning and AI.
Figure 1 . Figure 1. Machine learning and AI.CC-BY-NC-ND · From the PDF
Figure 2 . Figure2. Taxonomy development method[13].
Figure 2 . Figure2. Taxonomy development method[13].CC-BY-NC-ND · From the PDF
Figure 3 . Figure 3. Taxonomy of hybrid intelligence design.
Figure 3 . Figure 3. Taxonomy of hybrid intelligence design.CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 66

  1. B. Goertzel and C. Pennachin, Artificial general intelligence, vol. 2. Springer, 2007.
  2. L. Fridman, B. Reimer, B. Mehler, and W. T. Freeman, "Cognitive Load Estimation in the Wild," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems -CHI '18, (New York, USA), pp. 1-9, 2018.
  3. A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, "Dermatologist-level classification of skin cancer with deep neural networks," Nature, vol. 542, no. 7639, pp. 115-118, 2017.
  4. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529-533, 2015.
  5. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. Van Den Driessche, T. Graepel, and D. Hassabis, "Mastering the game of Go without human knowledge," nature, vol. 529, no. 7587, pp. 484-489, 2016.
  6. E. Kamar, "Directions in hybrid intelligence: Complementing AI systems with human intelligence," in IJCAI International Joint Conference on Artificial Intelligence, 2016.
  7. C. Müller-Schloer and S. Tomforde, Organic Computing Technical Systems for Survival in the Real World. Autonomic Systems, Cham: Springer International Publishing, 2017.
  8. A. Kurakin, I. Goodfellow, and S. Bengio, "Adversarial examples in the physical world," arXiv preprint arXiv:1607.02533, 2016. · arXiv
  9. B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman, "Building machines that learn and think like people," Behavioral and Brain Sciences, vol. 40, 2017.
  10. F. Doshi-Velez and B. Kim, "Towards a rigorous science of interpretable machine learning," arXiv preprint arXiv:1702.08608, 2017. · arXiv
  11. S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza, "Power to the people: The role of humans in interactive machine learning," AI Magazine, vol. 35, no. 4, pp. 105-120, 2014.
  12. R. C. Nickerson, U. Varshney, and J. Muntermann, "A method for taxonomy development and its application in information systems," European Journal of Information Systems, vol. 22, no. 3, pp. 336-359, 2013.
  13. T. M. Mitchell et al., "Machine learning. 1997," Burr Ridge, IL: McGraw Hill, vol. 45, no. 37, pp. 870-877, 1997.
  14. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
  15. M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter, "Efficient and robust automated machine learning," Advances in Neural Information Processing Systems, pp. 2962-2970, 2015.
  16. R. G. Gomes, P. Welinder, A. Krause, and P. Perona, "Crowdclustering," Advances in neural information processing systems, pp. 558-566, 2011.
  17. J. Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan, "Human decisions and machine predictions," The quarterly journal of economics, vol. 133, no. 1, pp. 237-293, 2017.
  18. A. Holzinger, "Interactive machine learning for health informatics: when do we need the human-in-the-loop?," Brain Informatics, vol. 3, no. 2, pp. 119-131, 2016.
  19. P. Y. Simard, S. Amershi, D. M. Chickering, A. E. Pelton, S. Ghorashi, C. Meek, G. Ramos, J. Suh, J. Verwey, M. Wang, et al., "Machine teaching: A new paradigm for building machine learning systems," arXiv preprint arXiv:1707.06742, 2017. · arXiv
  20. J. Attenberg, P. Ipeirotis, and F. Provost, "Beat the Machine," Journal of Data and Information Quality, vol. 6, no. 1, pp. 1-17, 2015.
  21. S. Gregor, "Explanations from knowledge-based systems and cooperative problem solving: an empirical study," International Journal of Human-Computer Studies, vol. 54, no. 1, pp. 81-105, 2001.
  22. "Cobi Communitysourcing Large-Scale Conference Scheduling." http://projectcobi.com last access: 2018-06-14.
  23. J. Webster and R. T. Watson, "Analyzing the past to prepare for the future: Writing a literature review," MIS Quarterly, vol. 26, no. 2, pp. 8-23, 2002.
  24. J. vom Brocke, A. Simons, B. Niehaves, K. Riemer, R. Plattfaut, A. Cleven, J. V. Brocke, and K. Reimer, "Reconstructing the Giant: On the Importance of Rigour in Documenting the Literature Search Process," 17th European Conference on Information Systems, 2009.
  25. U. Kayande, A. De Bruyn, G. L. Lilien, A. Rangaswamy, and G. H. Van Bruggen, "How incorporating feedback mechanisms in a dss affects dss evaluations," Information Systems Research, vol. 20, no. 4, pp. 527-546, 2009.
  26. W. M. Reynolds, G. E. Miller, and I. B. Weiner, Handbook of psychology. Wiley, 2013.
  27. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
  28. P. Choudhary, A. Jain, and R. Baijal, "Unravelling airbnb predicting price for new listing," arXiv preprint arXiv:1805.12101, 2018. · arXiv
  29. C. A. Gomez-Uribe and N. Hunt, "The netflix recommender system: Algorithms, business value, and innovation," ACM Transactions on Management Information Systems (TMIS), vol. 6, no. 4, p. 13, 2016.
  30. D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty, "Building Watson: An Overview of the DeepQA Project," AI Magazine, vol. 31, p. 59, jul 2010.
  31. J. Cheng, J. Teevan, S. T. Iqbal, and M. S. Bernstein, "Break It Down: A Comparison of Macro-and Microtasks," Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15, pp. 4061-4064, 2015.
  32. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. Le, and A. Kurakin, "Large-scale evolution of image classifiers," arXiv preprint arXiv:1703.01041, 2017. · arXiv
  33. E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, "Regularized evolution for image classifier architecture search," arXiv preprint arXiv:1802.01548, 2018. · arXiv
  34. S. Doroudi, E. Kamar, E. Brunskill, and E. Horvitz, "Toward a Learning Science for Complex Crowdsourcing Tasks," in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems -CHI '16, (New York, USA), pp. 2623-2634, ACM Press, 2016.
  35. S. Milli, P. Abbeel, and I. Mordatch, "Interpretable and pedagogical examples," arXiv preprint arXiv:1711.00694, 2017. · arXiv
  36. S. Carter and M. Nielsen, "Using Artificial Intelligence to Augment Human Intelligence," Distill, vol. 2, no. 12, 2017.
  37. K. P. Murphy, Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
  38. X. Zhu, "Semi-supervised learning literature survey," Computer Science, University of Wisconsin-Madison, vol. 2, no. 3, pp. 4-44, 2006.
  39. R. M. Hogarth, "On the learning of intuition," Intuition in judgment and decision making, pp. 111-126, 2011.
  40. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng, "Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks," Proceedings of the conference on empirical methods in natural language processing, pp. 254-263, 2008.
  41. V. C. Raykar, S. Yu, L. H. Zhao, G. Hermosillo Valadez, C. Florin, L. Bogoni, L. Moy, and L. M. Org, "Learning From Crowds," Journal of Machine Learning Research, vol. 11, pp. 1297-1322, 2010.
  42. B. Nushi, E. Kamar, E. Horvitz, and D. Kossmann, "On human intellect and machine failures: Troubleshooting integrative machine learning systems.," AAAI, pp. 1017-1025, 2017.
  43. K. Pei, Y. Cao, J. Yang, and S. Jana, "Towards practical verification of machine learning: The case of computer vision systems," arXiv preprint arXiv:1712.01785, 2017. · arXiv
  44. G. Li, Y. Zheng, J. Fan, J. Wang, and R. Cheng, "Crowdsourced data management: Overview and challenges," in Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1711-1716, ACM, 2017.
  45. A. Chakarov, A. Nori, S. Rajamani, S. Sen, and D. Vijaykeerthy, "Debugging machine learning tasks," arXiv preprint arXiv:1603.07292, 2016. · arXiv
  46. T. Kulesza, S. Stumpf, M. Burnett, W.-K. Wong, Y. Riche, T. Moore, I. Oberst, A. Shinsel, and K. McIntosh, "Explanatory Debugging: Supporting End-User Debugging of Machine-Learned Programs," in 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, pp. 41-48, IEEE, sep 2010.
  47. K. Patel, N. Bancroft, S. M. Drucker, J. Fogarty, A. J. Ko, and J. Landay, "Gestalt: integrated support for implementation and analysis in machine learning," Proceedings of the 23nd annual ACM symposium on User interface software and technology -UIST '10, pp. 37-46, 2010.
  48. J. C. Chang, S. Amershi, and E. Kamar, "Revolt," in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems -CHI '17, 2017.
  49. J. C. Chang, A. Kittur, and N. Hahn, "Alloy: Clustering with Crowds and Computation," Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems -CHI '16, 2016.
  50. D. Dellermann, N. Lipusch, and P. Ebel, "Developing Design Principles for a Crowd-Based Business Model Validation System," in International Conference on Design Science Research in Information Systems, pp. 163-178, 2017.
  51. D. Dellermann, N. Lipusch, P. Ebel, K. M. Popp, and J. M. Leimeister, "Finding the Unicorn : Predicting Early Stage Startup Success through a Hybrid Intelligence Method," ICIS 2017 Proceedings, pp. 1-12, 2017.
  52. A. J. Quinn and B. B. Bederson, "Human computation: a survey and taxonomy of a growing field," pp. 1403-1412, ACM, 2011.
  53. A. P. Dawid and A. M. Skene, "Maximum likelihood estimation of observer error-rates using the em algorithm," Applied statistics, vol. 28, pp. 20-28, 1979.
  54. H.-C. Kim and Z. Ghahramani, "Bayesian Classifier Combination," in Proceedings of the International Conference on Artifical Intelligence and Statistics (AISTATS), 2012.
  55. K. A. M. Kamar, Z. A. Hamid, and N. Dzulkalnine, "Industrialised Building System (IBS) construction: Measuring the perception of contractors in Malaysia," in BEIAC 2012 -2012 IEEE Business, Engineering and Industrial Applications Colloquium, 2012.
  56. M. Kosinski, Y. Bachrach, P. Kohli, D. Stillwell, and T. Graepel, "Manifestations of user personality in website choice and behaviour on online social networks," Machine learning, vol. 95, no. 3, pp. 357-380, 2014.
  57. A. Segal, K. sGal, E. Kamar, E. Horvitz, and G. Miller, "Optimizing interventions via offline policy evaluation: Studies in citizen science," AAAI 2018, 2018.
  58. L. Von Ahn and L. Dabbish, "Designing games with a purpose," Communications of the ACM, vol. 51, no. 8, pp. 58-67, 2008.
  59. J. W. Vaughan, "Making better use of the crowd: How crowdsourcing can advance machine learning research," Journal of Machine Learning Research, vol. 18, no. 193, pp. 1-46, 2018.
  60. A. Bernardo, A. Cervero, M. Esteban, E. Tuero, J. R. Casanova, and L. S. Almeida, "Freshmen Program Withdrawal: Types and Recommendations.," Frontiers in psychology, vol. 8, p. 1544, 2017.
  61. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in European conference on computer vision, pp. 740-755, 2014.
  62. V. S. Sheng, F. Provost, and P. G. Ipeirotis, "Get another label? improving data quality and data mining using multiple, noisy labelers," Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 614-622, 2008.
  63. L. Zhao, Y. Zhang, and G. Sukthankar, "An active learning approach for jointly estimating worker performance and annotation reliability with crowdsourced data," arXiv preprint arXiv:1401.3836, 2014. · arXiv
  64. G. Kou, Y. Peng, and G. Wang, "Evaluation of clustering algorithms for financial risk analysis using mcdm methods," Information Sciences, vol. 275, pp. 1-12, 2014.
  65. A. M. Chirkin and R. König, "Concept of Interactive Machine Learning in Urban Design Problems," in Proceedings of the SEACHI 2016 on Smart Cities for Better Living with HCI and UX -SEACHI 2016, 2016.
  66. Z. C. Lipton, "The mythos of model interpretability," Queue, vol. 16, no. 3, p. 30, 2018.

Cited by

Sorted by citations

See all 456

Metadata

Authors
Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., & Ebel, P.
Affiliations
University of Kassel (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
1530-1605 (ISSN-L)
DOI
10.48550/arxiv.2105.03354
arXiv
2105.03354
OpenAlex
W2901418095
Cited by
456
References
1
Citations by year
2026: 120 · 2025: 106 · 2024: 78 · 2023: 59 · 2022: 40 · 2021: 28
FWCI
32.57
Type
conference paper
Language
en
Open access
Licence
Locations
  • Hawaii International Conference on System Sciences (HICSS) (publishedVersion, cc-by-nc-nd)
  • Alexandria (UniSG) (University of St.Gallen) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Alexandria (UniSG) (University of St.Gallen) (submittedVersion)
  • ScholarSpace (University of Hawaii at Manoa) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
human-AI collaboration, artificial general intelligence, socio-technical systems, human-in-the-loop learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems. https://app.cochat.ai/papers/10.48550/arxiv.2105.03354
The paper
Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., & Ebel, P. (2021). The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems. arXiv. https://doi.org/10.48550/arxiv.2105.03354