Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Open accessThe DOI matches its Crossref record; 105 of the 109 references it cites were found in a registry or an index.
Abstract
Machine learning (ML) is the fastest growing field in computer science, and health informatics is among the greatest challenges. The goal of ML is to develop algorithms which can learn and improve over time and can be used for predictions. Most ML researchers concentrate on automatic machine learning (aML), where great advances have been made, for example, in speech recognition, recommender systems, or autonomous vehicles. Automatic approaches greatly benefit from big data with many training sets. However, in the health domain, sometimes we are confronted with a small number of data sets or rare events, where aML-approaches suffer of insufficient training samples. Here interactive machine learning (iML) may be of help, having its roots in reinforcement learning, preference learning, and active learning. The term iML is not yet well used, so we define it as "algorithms that can interact with agents and can optimize their learning behavior through these interactions, where the agents can also be human." This "human-in-the-loop" can be beneficial in solving computationally hard problems, e.g., subspace clustering, protein folding, or k-anonymization of health data, where human expertise can help to reduce an exponential search space through heuristic selection of samples. Therefore, what would otherwise be an NP-hard problem, reduces greatly in complexity through the input and the assistance of a human agent involved in the learning phase.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In such cases, rule-based automation is impossible, but managers could use an augmentation approach to explore the problem further (Holzinger, 2016).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Holzinger, 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since emotions and other subjective factors affect humans, augmentation is difficult or even impossible to replicate, which means every augmentation initiative is a new learning effort (Holzinger, 2016).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Holzinger, 2016
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Consensus and Cooperation in Networked Multi-Agent Systems
Olfati-Saber, R., Fax, J. A., & Murray, R. M. (2007) · Proceedings of the IEEE · doi:10.1109/jproc.2006.887293 · Cited by 10,522 · cited 1×
“On this basis, Olfati-Saber et al. [94] presented a theoretical framework for analysis of consensus algorithms for networked multi-agent systems with fixed or dynamic topology and directed information flow.”
BackgroundI.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 2×
“Today, machine learning (ML) is the fastest growing technical field, at the intersection of informatics and statistics, tightly connected with data science and knowledge discovery, and health is among the greatest challenges [2, 3].”
Background(2012) · Cited by 9,062
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Computing Machinery and Intelligence (1950)
Turing, A. (2004) · The Essential Turing · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,118
A law of comparative judgment.
Thurstone, L. L. (1927) · Psychological Review · doi:10.1037/h0070288 · Cited by 5,210
The Psychology of Learning and Motivation
(2021) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(21)x0002-3 · Cited by 5,027
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,044
Gigerenzer, G., & Gaissmaier, W. (2011) · Annual Review of Psychology · doi:10.1146/annurev-psych-120709-145346 · Cited by 4,037
Degroot, M. H. (1974) · Journal of the American Statistical Association · doi:10.1080/01621459.1974.10480137 · Cited by 4,004
(1992) · Lecture Notes in Computer Science · doi:10.1007/bfb0022433 · Cited by 3,955
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
L -diversity: Privacy beyond k -anonymity
Machanavajjhala, A., Kifer, D., Gehrke, J., & Venkitasubramaniam, M. (2007) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/1217299.1217302 · Cited by 3,617
t-Closeness: Privacy Beyond k-Anonymity and l-Diversity
Li, N., Li, T., & Venkatasubramanian, S. (2007) · 2007 IEEE 23rd International Conference on Data Engineering · doi:10.1109/icde.2007.367856 · Cited by 3,451 · cited 1×
“…a data representation, and treating the values of an attribute distinctly by taking into account the distribution of data values for that attribute [43]; and delta-presence , which links the quality of anonymization to the risk posed by inadequate anonymization [44]), but not with regard to the…”
BackgroundValiant, L. G. (1984) · Communications of the ACM · doi:10.1145/1968.1972 · Cited by 3,279 · cited 1×
“The underlying principle is bias learning based on probable approximately correct learning (PAC learning) [79].”
BackgroundBayes, T. (1763) · Philosophical Transactions of the Royal Society of London · doi:10.1098/rstl.1763.0053 · Cited by 2,995
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817
The Analysis of Proximities: Multidimensional Scaling with an Unknown Distance Function. I.
Shepard, R. N. (1962) · Psychometrika · doi:10.1007/bf02289630 · Cited by 2,550
Unsupervised Learning by Probabilistic Latent Semantic Analysis
Hofmann, T. (2001) · Machine Learning · doi:10.1023/a:1007617005950 · Cited by 2,456
Catastrophic forgetting in connectionist networks
French, R. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01294-2 · Cited by 2,311
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · cited 1×
“A significant part of the research on multi-agent learning concerns RL-techniques [50].”
BackgroundACHIEVING k-ANONYMITY PRIVACY PROTECTION USING GENERALIZATION AND SUPPRESSION
SWEENEY, L. (2002) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s021848850200165x · Cited by 1,976
(1996) · Lecture Notes in Computer Science · doi:10.1007/3-540-61769-8 · Cited by 1,865
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
Weiß, G. (1999) · Cited by 1,640
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,488
Regularized multi--task learning
Evgeniou, T., & Pontil, M. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014067 · Cited by 1,433
Predicting protein structures with a multiplayer online game
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., & players, F. (2010) · Nature · doi:10.1038/nature09304 · Cited by 1,368 · cited 1×
“A good example of such an approach is foldit, an experimental game which takes advantage of crowdsourcing for category discovery of new protein structures [37].”
BackgroundActive Learning with Statistical Models
Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.295 · Cited by 1,317
Settles, B. (2012) · Cited by 1,183
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Hubel, D. H. (1988) · Medical Entomology and Zoology · Cited by 1,026
Generalizing data to provide anonymity when disclosing information (abstract)
Samarati, P., & Sweeney, L. (1998) · Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems · doi:10.1145/275487.275508 · Cited by 779
A Model of Inductive Bias Learning
Baxter, J. (2000) · Cited by 730
On k -anonymity and the curse of dimensionality
Aggarwal, C. C. (2005) · Cited by 614
A mathematical model for simple learning.
Bush, R. R., & Mosteller, F. (1951) · Psychological Review · doi:10.1037/h0054388 · Cited by 586
A model of inductive bias learning
Baxter, J. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.731 · Cited by 582 · cited 1×
“Baxter [80] introduced a model of bias learning which builds on the PAC learning model which concluded that learning multiple related tasks reduces the sampling burden required for good generalization and that the bias learnt on sufficiently many training tasks is likely to be good for learning…”
BackgroundSettles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01560-1 · Cited by 502
Minimization of Boolean complexity in human concept learning
Feldman, J. (2000) · Nature · doi:10.1038/35036586 · Cited by 478
Fürnkranz, J., & Hllermeier, E. (2010) · Springer eBooks · doi:10.1007/978-3-642-14125-6 · Cited by 457
Learning to Rank for Information Retrieval
Liu, T.-Y. (2011) · Springer eBooks · doi:10.1007/978-3-642-14267-3 · Cited by 407
Content-based image retrieval in radiology: current status and future directions
Akgül, C. B., Rubin, D. L., Napel, S., Beaulieu, C. F., Greenspan, H., & Acar, B. (2011) · Journal of Digital Imaging · doi:10.1007/s10278-010-9290-9 · Cited by 388
From Theories to Queries: Active Learning in Practice
Settles, B. (2011) · Cited by 387
Concept learning: An information processing problem.
Hunt, E. B. (1962) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13135-000 · Cited by 369
Reinforcement learning improves behaviour from evaluative feedback
Littman, M. L. (2015) · Nature · doi:10.1038/nature14540 · Cited by 368
Preference learning with Gaussian processes
Chu, W., & Ghahramani, Z. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102369 · Cited by 361
Consensus theoretic classification methods
Benediktsson, J. A., & Swain, P. H. (1992) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.156582 · Cited by 357
Active Learning with Support Vector Machines in the Drug Discovery Process
Warmuth, M. K., Liao, J., Rätsch, G., Mathieson, M., Putta, S., & Lemmen, C. (2003) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci025620t · Cited by 354
Large Margin Multi-Task Metric Learning
Parameswaran, S., & Weinberger, K. Q. (2010) · neural information processing systems · Cited by 335
Collaborative Multiagent Reinforcement Learning by Payoff Propagation
Kok, J. R., & Vlassis, N. A. (2006) · Open Repository and Bibliography (University of Luxembourg) · doi:10.5555/1248547.1248612 · Cited by 277
The protein threading problem with sequence amino acid interaction preferences is NP-complete
Lathrop, R. H. (1994) · "Protein Engineering, Design and Selection" · doi:10.1093/protein/7.9.1059 · Cited by 276
Reinforcement learning design for cancer clinical trials
Zhao, Y., Kosorok, M. R., & Zeng, D. (2009) · Statistics in Medicine · doi:10.1002/sim.3720 · Cited by 271
Interactively optimizing information retrieval systems as a dueling bandits problem
Yue, Y., & Joachims, T. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553527 · Cited by 268
F�rnkranz, J., & Hllermeier, E. (2010) · Cited by 259
Holzinger, A., Dehmer, M., & Jurisica, I. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-s6-i1 · Cited by 219
Self-citationThe K-armed dueling bandits problem
Yue, Y., Broder, J., Kleinberg, R., & Joachims, T. (2012) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2011.12.028 · Cited by 212
Barnard, G. A., & Bayes, T. (1958) · Biometrika · doi:10.2307/2333180 · Cited by 203
Preference-based reinforcement learning: a formal framework and a policy iteration algorithm
Fürnkranz, J., Hüllermeier, E., Cheng, W., & Park, S.-H. (2012) · Machine Learning · doi:10.1007/s10994-012-5313-8 · Cited by 187
Crowdsourcing for bioinformatics
Good, B. M., & Su, A. I. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt333 · Cited by 144
Learning Trajectory Preferences for Manipulators via Iterative ImprovementPreprint
Jain, A., Wojcik, B., Joachims, T., & Saxena, A. (2013) · Neural Information Processing Systems · doi:10.48550/arxiv.1306.6294 · Cited by 143
Training a Robot via Human Feedback: A Case Study
Knox, W. B., Stone, P., & Breazeal, C. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-02675-6_46 · Cited by 130
Active Preference Learning with Discrete Choice Data
Brochu, E., de Freitas, N., & Ghosh, A. (2007) · Cited by 125
Effect of human guidance and state space size on Interactive Reinforcement Learning
Suay, H. B., & Chernova, S. (2011) · IEEE RO-MAN · doi:10.1109/roman.2011.6005223 · Cited by 118
Glatzeder, B., Goel, V., & Müller, A. (2010) · On thinking · doi:10.1007/978-3-642-03129-8 · Cited by 112
Trotman, A. (2005) · Information Retrieval · doi:10.1007/s10791-005-6991-7 · Cited by 111
Assessing Rater Performance without a "Gold Standard" Using Consensus Theory
Weller, S. C., & Mann, N. C. (1997) · Medical Decision Making · doi:10.1177/0272989x9701700108 · Cited by 110
Holzinger, A. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40511-2_22 · Cited by 109
Self-citationPreference Mining: A Novel Approach on Mining User Preferences for Personalized Applications
Holland, S., Ester, M., & Kießling, W. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39804-2_20 · Cited by 108
Finding meaning on YouTube: Tag recommendation and category discovery
Toderici, G. D., Aradhye, H. B., PAŞCA, M. A., Sbaiz, L., & Yagnik, J. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539985 · Cited by 107
(2012) · Cited by 98
Biomedical Text Mining: State-of-the-Art, Open Problems and Future Challenges
Holzinger, A., Schantl, J., Schroettner, M., Seifert, C., & Verspoor, K. M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-43968-5_16 · Cited by 86
Self-citationHolzinger, A. (2014) · Cited by 83
Self-citationOptimal Bayesian Recommendation Sets and Myopically Optimal Choice Query Sets
Viappiani, P., & Boutilier, C. E. (2010) · neural information processing systems · Cited by 81
Roche, B., Guégan, J., & Bousquet, F. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-435 · Cited by 76
Web Intelligence Meets Brain Informatics
Zhong, N., Liu, J., Yao, Y. Y., Wu, J., Lu, S., Qin, Y., Li, K., & Wah, B. W.-S. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-77028-2_1 · Cited by 75
Clustering with Interactive Feedback
Balcan, M.-F., & Blum, A. L. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-87987-9_27 · Cited by 71
Real-Time Interactive Reinforcement Learning for Robots
Thomaz, A. L., Hoffman, G., & Breazeal, C. (2005) · Cited by 66
Biomedical Informatics: Discovering Knowledge in Big Data
Holzinger, A. (2014) · Springer eBooks · doi:10.1007/978-3-319-04528-3 · Cited by 53
Self-citationWitnesses for the Doctor in the Loop
Kieseberg, P., Schantl, J., Frühwirt, P., Weippl, E. R., & Holzinger, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23344-4_36 · Cited by 43
Self-citationLocal algorithms for interactive clusteringPreprint
Awasthi, P., Balcan, M.-F., & Voevodski, K. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6724 · Cited by 43
Morpheus: interactive exploration of subspace clustering
Müller, E., Assent, I., Krieger, R., Jansen, T., & Seidl, T. (2008) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1401890.1402026 · Cited by 41
δ-Presence without Complete World Knowledge
Nergiz, M. E., & Clifton, C. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.125 · Cited by 39
Reprint of: Computational approaches for mining user’s opinions on the Web 2.0
Petz, G., Karpowicz, M., Fürschuß, H., Auinger, A., Stříteský, V., & Holzinger, A. (2015) · Information Processing & Management · doi:10.1016/j.ipm.2014.07.011 · Cited by 37
Self-citationGoldstone, R. L., Day, S., & Son, J. Y. (2009) · On thinking · doi:10.1007/978-3-642-03129-8_7 · Cited by 32
Introspective Subgroup Analysis for Interactive Knowledge Refinement.
Atzmüller, M., Baumeister, J., & Puppe, F. (2006) · The Florida AI Research Society · Cited by 28
Hybrid Multi-agent Systems: Integrating Swarming and BDI Agents
Parunak, H. V. D., Nielsen, P. E., Brueckner, S., & Alonso, R. D. H. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-69868-5_1 · Cited by 28
Budge, E. J., Tsoti, S. M., Howgate, D. J., Sivakumar, S., & Jalali, M. (2015) · Annals of Medicine · doi:10.3109/07853890.2015.1091945 · Cited by 26
Interactive and Iterative Annotation for Biomedical Entity Recognition
Yimam, S. M., Biemann, C., Majnarić, L. T., Šabanović, Š., & Holzinger, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23344-4_34 · Cited by 18
Self-citationDiscovering Medical Knowledge Using Visual Analytics
Sturm, W., Schreck, T., Holzinger, A., & Ullrich, T. (2015) · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · doi:10.2312/vcbm.20151210 · Cited by 15
Self-citationAnalysis of Patient Groups and Immunization Results Based on Subspace Clustering
Hund, M., Sturm, W., Schreck, T., Ullrich, T., Keim, D. A., Majnarić, L. T., & Holzinger, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23344-4_35 · Cited by 14
Self-citationSpinrad, N. (2014) · Nature · doi:10.1038/514528a · Cited by 13
Yang, M., Yang, Y., Wang, W., Ding, H., & Chen, J. (2014) · Mathematical Problems in Engineering · doi:10.1155/2014/951367 · Cited by 12
Wensveen, J. M., & Wick, B. (1995) · Optometry and Vision Science · doi:10.1002/j.1538-9235.1995.tb03973.x · Cited by 10
A Hybrid System for the Recognition of Hand-Written Characters
Freund, R., Neubauer, M., Summerer, M., Gruber, S., Schaffer, J., & Swoboda, R. (2000) · Lecture notes in computer science · doi:10.1007/3-540-44522-6_7 · Cited by 2
Valiant, L. G. (2026) · Foundations of Computation and Machine Learning · doi:10.1145/3749740.3749746
Research works printed but not linked: 5
- Mitchell TM (1997) Machine learning. McGraw Hill, New York
- Fahlman SE, Hinton GE, Sejnowski TJ (1983) Massively parallel architectures for Al: NETL, Thistle, and Boltzmann machines. In: Genesereth MR (ed) AAAI-83. AAAI, Washington, DC, pp 109–113
- Wilson AG, Dann C, Lucas, Xing EP (2015) The human kernel. arXiv preprint arXiv:1510.07389 · arXiv
- Agarwal S, Cortes C, Herbrich R (2005) Learning to rank. In: Weiss Y, Schlkopf B, Platt JC (eds) NIPS Workshop
- Lynch NA (1996) Distributed algorithms. Morgan Kaufmann, San Francisco
Cited by
Artificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885
“In medical decision support we are confronted with uncertainty, with probabilistic, unknown, incomplete, imbalanced, heterogeneous, noisy, dirty, erroneous, inaccurate and missing data sets in arbitrarily high-dimensional spaces (Holzinger, Dehmer, & Jurisica, 2014), (Lee & Holzinger, 2016).”
BackgroundInfluentialSelf-citationShin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Human-in-the-loop machine learning: a state of the art
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10246-w · Cited by 1,025
Metadata
- Authors
- Holzinger, A.
- Affiliations
- University of Graz (AT)
- Corresponding author
- Holzinger, Andreas
- ORCID
- Holzinger, Andreas
- Venue
- Brain Informatics
- Published
- 2016-03-02
- Volume
- 3
- Issue
- 2
- Pages
- 119-131
- ISSN
- 2198-4018 (ISSN-L)
- 2198-4026
- PMC
- PMC4883171
- OpenAlex
- W2300445845
- Cited by
- 906
- References
- 109
- Citations by year
- 2026: 93 · 2025: 85 · 2024: 87 · 2023: 74 · 2022: 111 · 2021: 132 · 2020: 110 · 2019: 77 · 2018: 53 · 2017: 46 · 2016: 38
- FWCI
- 58.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Brain Informatics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- interactive machine learning, human-in-the-loop, health informatics, active learning (machine learning), preference learning, reinforcement learning, limited training data, rare events, human-agent interaction, NP-hard problems, k-anonymization, protein folding, subspace clustering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.