The Design of Reciprocal Learning Between Human and Artificial Intelligence
The DOI matches its Crossref record; 71 of the 74 references it cites were found in a registry or an index.
Abstract
The need for advanced automation and artificial intelligence (AI) in various fields, including text classification, has dramatically increased in the last decade, leaving us critically dependent on their performance and reliability. Yet, as we increasingly rely more on AI applications, their algorithms are becoming more nuanced, more complex, and less understandable precisely at a time we need to understand them better and trust them to perform as expected. Text classification in the medical and cybersecurity domains is a good example of a task where we may wish to keep the human in the loop. Human experts lack the capacity to deal with the high volume and velocity of data that needs to be classified, and ML techniques are often unexplainable and lack the ability to capture the required context needed to make the right decision and take action. We propose a new abstract configuration of Human-Machine Learning (HML) that focuses on reciprocal learning, where the human and the AI are collaborating partners. We employ design-science research (DSR) to learn and design an application of the HML configuration, which incorporates software to support combining human and artificial intelligences. We define the HML configuration by its conceptual components and their function. We then describe the development of a system called Fusion that supports human-machine reciprocal learning. Using two case studies of text classification from the cyber domain, we evaluate Fusion and the proposed HML approach, demonstrating benefits and challenges. Our results show a clear ability of domain experts to improve the ML classification performance over time, while both human and machine, collaboratively, develop their conceptualization, i.e., their knowledge of classification. We generalize our insights from the DSR process as actionable principles for researchers and designers of 'human in the learning loop' systems. We conclude the paper by discussing HML configurations and the challenge of capturing and representing knowledge gained jointly by human and machine, an area we feel has great potential.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Some systems integrate basic learning capabilities [72] through feedback loops [73,74], heuristics [75], or updated context buffers to refine behavior over time, particularly in settings like personalized recommendations or conversation flow management [7678].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [74]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“Word embedding techniques are considered context-aware, in the sense that terms are modeled as an integral part of their linguistic surrounding [18, 36, 47].”
BackgroundSMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 2×
“All the supervised models shown, including the baseline model, used 10-fold cross validation to fit the model and synthetic minority oversampling technique (SMOTE) [13] to help overcome class imbalance.”
BackgroundMethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×
“Mathematically, context can be built from the corpus being classified or from an external (enhancement) corpus, a technique called word embedding [42].”
BackgroundMethodQualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · cited 1×
“In one instance, the Lasso regression did not converge [23].”
BackgroundResearch design: qualitative, quantitative, and mixed methods approaches
Caldas, M. P. (2003) · Revista de Administração Contemporânea · doi:10.1590/s1415-65552003000100015 · Cited by 18,745 · cited 1×
“We used recommended methods of qualitative analysis [12, 16, 17] to overcome inconsistencies and biases in creating the conceptualization of classification knowledge, and we added an external expert to increase reliability and minimize bias.”
Method(2006) · Cited by 18,730
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
The Knowledge-creating company: How Japanese companies create the dynamics of innovation
(1997) · Research Policy · doi:10.1016/s0048-7333(97)80234-x · Cited by 13,156 · cited 1×
“Learning in practice has been likened to a spiral of knowledge creation that pauses to take action and resumes to upgrade itself continuously [45, 69].”
BackgroundRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“More recently, researchers began exploring explainability and explainable AI (XAI) techniques [8, 11, 20, 51, 60].”
Background
Research works printed but not linked: 10
- Ahmed Abbasi and Hsinchun Chen. 2008. CyberGate: a design framework and system for text analysis of computer-mediated communication. MIS Quarterly (2008), 811--837.
- 10.5555/1622407.1622416 · doi:10.5555/1622407.1622416
- Charles Goodwin. 2000. Practices of color classification. Mind, culture, and activity 7, 1--2 (2000), 19--36.
- 10.5555/647402.725294 · doi:10.5555/647402.725294
- Ton Jörg. 2004. A theory of reciprocal learning in dyads. Cognitive systems 6, 2 (2004), 3.
- Daniel Khashabi, Erfan Sadeqi Azer, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2020. On the Possibilities and Limitations of Multi-hop Reasoning Under Linguistic Imperfections. arXiv:1901.02522 [cs.CL] · arXiv
- 10.5555/3019323 · doi:10.5555/3019323
- Chaehan So. 2020. Understanding the Prediction Mechanism of Sentiments by XAI Visualization. arXiv preprint arXiv:2003.01425 (2020). · arXiv
- Dov Te'eni. 2001. A cognitive-affective model of organizational communication for designing IT. MIS quarterly 25, 2 (2001), 251--312.
- Efpraxia D Zamani, Anastasia Griva, Konstantina Spanaki, Paidi O'Raghallaigh, and David Sammon. 2021. Making sense of business analytics in project selection and prioritisation: insights from the start-up trenches. Information Technology & People (2021).
Cited by
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“Some systems integrate basic learning capabilities [72] through feed-back loops [73,74], heuristics [75], or updated context buffers to refine behavior over time, particularly in settings like personalized recommendations or conversation flow management [76– 78].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
“Some systems integrate basic learning capabilities [72] through feed-back loops [73,74], heuristics [75], or updated context buffers to refine behavior over time, particularly in settings like personalized recommendations or conversation flow management [76– 78].”
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Assertiveness-based Agent Communication for a Personalized Medicine on Medical Imaging Diagnosis
Calisto, F. M., Fernandes, J., Morais, M., Santiago, C., Abrantes, J. M., Nunes, N., & Nascimento, J. C. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580682 · Cited by 93
“From the HCI literature [15, 61, 124], we know that the development of trust is influenced by the positive motivational attribution between the communication entity and the user.”
Background
Metadata
- Authors
- Zagalsky, A., Te'eni, D., Yahav, I., Schwartz, D. G., Silverman, G., Cohen, D., Mann, Y., & Lewinsky, D.
- Affiliations
- Tel Aviv University (IL)
- Tel Aviv University, Tel Aviv, Israel
- Bar-Ilan University (IL)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-36
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3206213635
- Cited by
- 35
- References
- 74
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 6 · 2023: 9 · 2022: 6 · 2021: 1
- FWCI
- 1.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Funders
- Israel National Cyber Directorate (0612018391)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Keywords
- mutual learning, human-AI collaboration, text classification, cybersecurity, clinical text classification, explainable AI, design principles, fusion systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Design of Reciprocal Learning Between Human and Artificial Intelligence. https://app.cochat.ai/papers/10.1145/3479587
- The paper
- Zagalsky, A., Te'eni, D., Yahav, I., Schwartz, D. G., Silverman, G., Cohen, D., Mann, Y., & Lewinsky, D. (2021). The Design of Reciprocal Learning Between Human and Artificial Intelligence. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3479587
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.