CoChat

Open-world Machine Learning: Applications, Challenges, and Opportunities

Open access
Parmar, Jitendra, Chouhan, Satyendra, Raychoudhury, Vaskar, Rathore, Santosh · 2023 · ACM Computing Surveys
Cited by 154

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

Abstract

Traditional machine learning, mainly supervised learning, follows the assumptions of closed-world learning, i.e., for each testing class, a training class is available. However, such machine learning models fail to identify the classes, which were not available during training time. These classes can be referred to as unseen classes. Open-world Machine Learning (OWML) is a novel technique, which deals with unseen classes. Although OWML is around for a few years and many significant research works have been carried out in this domain, there is no comprehensive survey of the characteristics, applications, and impact of OWML on the major research areas. In this article, we aimed to capture the different dimensions of OWML with respect to other traditional machine learning models. We have thoroughly analyzed the existing literature and provided a novel taxonomy of OWML considering its two major application domains: Computer Vision and Natural Language Processing. We listed the available software packages and open datasets in OWML for future researchers. Finally, the article concludes with a set of research gaps, open challenges, and future directions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3561381

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 37 references

Research works printed but not linked: 119

  1. A. S. Albahri, A. A. Zaidan, O. S. Albahri, B. B. Zaidan, A. H. Alamoodi, Ali H. Shareef, Jwan K. Alwan, Rula A. Hamid, M. T. Aljbory, Ali Najm Jasim, et al. 2021. Development of IoT-based mhealth framework for various cases of heart disease patients. Health Technol. 11, 5 (2021), 1013–1033. · DOI
  2. Osamah Shihab Albahri, A. A. Zaidan, B. B. Zaidan, Ahmed Shihab Albahri, Ali H. Mohsin, K. I. Mohammed, and M. A. Alsalem. 2022. New mHealth hospital selection framework supporting decentralised telemedicine architecture for outpatient cardiovascular disease-based integrated techniques: Haversine-GPS and AHP-VIKOR. J. Amb. Intell. Human. Comput. 13, 1 (2022), 219–239. · DOI
  3. Ethem Alpaydin. 2020. Introduction to Machine Learning. MIT Press.
  4. 10.1109/TFUZZ.2008.925904 · DOI · Paper page
  5. 10.1007/978-3-540-76298-0_52 · DOI · Paper page
  6. Abhijit Bendale and Terrance Boult. 2015. Towards open world recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1893–1902.
  7. 10.1109/CVPR.2016.173 · DOI · Paper page
  8. Christopher M. Bishop and M. Nasrabadi Nasser. 2006. Pattern recognition and machine learning. Vol. 4 no. 4.
  9. Paul Bodesheim, Alexander Freytag, Erik Rodner, and Joachim Denzler. 2015. Local novelty detection in multi-class recognition problems. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. IEEE, 813–820.
  10. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th International Conference on Neural Information Processing Systems. 2787–2795.
  11. 10.1007/s10994-013-5335-x · DOI · Paper page
  12. 10.1613/JAIR.1.12228 · DOI · Paper page
  13. 10.1145/1961189.1961199 · DOI · Paper page
  14. Zhiyuan Chen and Bing Liu. 2014. Mining topics in documents: Standing on the shoulders of big data. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1116–1125. · DOI · Paper page
  15. Gouenou Coatrieux, Laurent Lecornu, Bülent Sankur, and Ch Roux. 2006. A review of image watermarking applications in healthcare. In Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society. 4691–4694.
  16. Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’17). IEEE, 2921–2926.
  17. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces. arXiv:1805.10190. Retrieved from https://arxiv.org/abs/1805.10190. · arXiv
  18. Laurens De Haan, Ana Ferreira, and Ana Ferreira. 2006. Extreme Value Theory: An Introduction, Vol. 21. Springer. · DOI · Paper page
  19. Rocco De Rosa, Thomas Mensink, and Barbara Caputo. 2016. Online open world recognition. arXiv:1604.02275. Retrieved from https://arxiv.org/abs/1604.02275. · arXiv
  20. 10.1007/s10489-015-0661-2 · DOI · Paper page
  21. Tri Doan and Jugal Kalita. 2017. Overcoming the challenge for text classification in the open world. In Proceedings of the IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC). 1–7.
  22. Geli Fei and Bing Liu. 2016. Breaking the closed world assumption in text classification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 506–514.
  23. 10.1023/A:1022606404104 · doi:10.1023/a:1022606404104 · Paper page
  24. Yang Gao, Yi-Fan Li, Bo Dong, Yu Lin, and Latifur Khan. 2019. SIM: Open-world multi-task stream classifier with integral similarity metrics. In Proceedings of the IEEE International Conference on Big Data (Big Data’19). IEEE, 751–760.
  25. Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. 2020. Recent advances in open set recognition: A survey. IEEE Trans. Pattern Anal. Machine Intell. 43, 10 (2020), 3614–3631.
  26. 10.1023/B:VISI.0000042993.50813.60 · DOI · Paper page
  27. 10.5555/3086952 · doi:10.5555/3086952
  28. 10.1145/3357384.3357981 · DOI
  29. Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2787–2792.
  30. Mehadi Hassen and Philip K. Chan. 2020. Learning a neural-network-based representation for open set recognition. In Proceedings of the SIAM International Conference on Data Mining. SIAM, 154–162.
  31. Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. 2020. Incremental learning in online scenario. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13926–13935.
  32. 10.1145/2872427.2883037 · DOI · Paper page
  33. Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. 2021. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1074–1083.
  34. Andrei De Souza Inácio, Matheus Gutoski, André Eugênio Lazzaretti, and Heitor Silvério Lopes. 2021. OSVidCap: A framework for the simultaneous recognition and description of concurrent actions in videos in an open-set scenario. IEEE Access 9 (2021), 137029–137041.
  35. 10.1109/91.413232 · doi:10.1109/91.413232
  36. Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. 2014. Multi-class open set recognition using probability of inclusion. In Proceedings of the 13th European Conference on Computer Vision. 393–409.
  37. A. Jeya Christy and K. Dhanalakshmi. 2021. Content-based image recognition and tagging by deep learning methods. Wireless Pers. Commun. (2021), 1–26.
  38. 10.1145/2736277.2741669 · DOI · Paper page
  39. 10.1007/s10994-016-5610-8 · DOI · Paper page
  40. Dan Jurafsky. 1997. Switchboard SWBD-DAMSL Shallow-discourse-function Annotation Coders Manual. Institute of Cognitive Science Technical Report.
  41. James M. Keller, Michael R. Gray, and James A. Givens. 1985. A fuzzy k-nearest neighbor algorithm. IEEE Trans. Syst. Man Cybernet.4 (1985), 580–585. · DOI · Paper page
  42. Zubair Ahmed Khan and Asma Rizvi. 2021. AI based facial recognition technology and criminal justice: Issues and challenges. Turk. J. Comput. Math. Educ. 12, 14 (2021), 3384–3392.
  43. Shu Kong and Deva Ramanan. 2021. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 813–822.
  44. Sotiris B. Kotsiantis, I. Zaharakis, P. Pintelas, et al. 2007. Supervised machine learning: A review of classification techniques. Emerg. Artif. Intell. Appl. Comput. Eng. 160, 1 (2007), 3–24.
  45. Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master’s thesis. Department of Computer Science University of Toronto.
  46. Ludmila Kuncheva. 2000. Fuzzy Classifier Design. Vol. 49. Springer Science & Business Media. · DOI
  47. Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings, LakeTahoe, CA, 331–339.
  48. 10.1109/TPAMI.2005.92 · DOI · Paper page
  49. 10.1109/TCSVT.2019.2898940 · DOI · Paper page
  50. Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 5491–5496.
  51. 10.1016/j.knosys.2019.104979 · DOI
  52. Fan Liu and Yong Deng. 2020. Determine the number of unknown targets in open world based on elbow method. IEEE Trans. Fuzzy Syst. 29, 5 (2020), 986–995.
  53. 10.1145/3544013 · doi:10.1145/3544013
  54. Abbas Saliimi Lokman and Mohamed Ariff Ameedeen. 2018. Modern chatbot systems: A technical review. In Proceedings of the Future Technologies Conference. 1012–1023.
  55. Vincent Lonij, Ambrish Rawat, and Maria-Irina Nicolae. 2017. Open-world visual recognition using knowledge graphs. arXiv:1708.08310. Retrieved from https://arxiv.org/abs/1708.08310. · arXiv
  56. 10.1016/j.ins.2022.03.014 · DOI
  57. 10.1016/j.asoc.2015.06.038 · DOI
  58. Sahisnu Mazumder, Nianzu Ma, and Bing Liu. 2018. Towards a continuous knowledge learning engine for chatbots. arXiv:1802.06024. Retrieved from https://arxiv.org/abs/1802.06024. · arXiv
  59. 10.1145/2766462.2767755 · DOI · Paper page
  60. 10.1109/TPAMI.2013.83 · DOI · Paper page
  61. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781. Retrieved from https://arxiv.org/abs/1301.3781. · arXiv
  62. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. 3111–3119.
  63. 10.1109/WACV48630.2021.00361 · DOI
  64. Philip Moore and Hai Van Pham. 2015. On context and the open world assumption. In Proceedings of the IEEE 29th International Conference on Advanced Information Networking and Applications Workshops. 387–392.
  65. 10.1109/TKDE.2017.2691702 · DOI
  66. Susan M. Mudambi and David Schuff. 2010. Research note: What makes a helpful online review? A study of customer reviews on Amazon. com. MIS Quart. (2010), 185–200. · DOI · Paper page
  67. Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. MIT Press.
  68. 10.5555/550675 · doi:10.5555/550675
  69. David Nister and Henrik Stewenius. 2006. Scalable recognition with a vocabulary tree. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 2. 2161–2168.
  70. Manuel Olave, Vladislav Rajkovic, and Marko Bohanec. 1989. An application for admission in public school systems. Expert Syst. Publ. Admin. 1 (1989), 145–160.
  71. Poojan Oza and Vishal M. Patel. 2019. C2ae: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2307–2316.
  72. Poojan Oza and Vishal M. Patel. 2019. Deep cnn-based multi-task learning for open-set recognition. arXiv:1903.03161. Retrieved from https://arxiv.org/abs/1903.03161. · arXiv
  73. Jitendra Parmar, Sanskar Soni, and Satyendra Singh Chouhan. 2020. OWI: Open-world intent identification framework for dialog based system. In Proceedings of the International Conference on Big Data Analytics. 329–343. · DOI
  74. 10.3115/v1/D14-1162 · DOI · Paper page
  75. John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, Vol. 10. MIT Press, 61–74.
  76. Sridhama Prakhya, Vinodini Venkataram, and Jugal Kalita. 2017. Open set text classification using convolutional neural networks. In Proceedings of the International Conference on Natural Language Processing.
  77. Hemant Purohit, Carlos Castillo, Fernando Diaz, Amit Sheth, and Patrick Meier. 2014. Emergency-relief coordination on social media: Automatically matching resource requests and offers. First Monday (2014).
  78. David Ratcliffe and Kerry Taylor. 2014. Closed-world concept induction for learning in OWL knowledge bases. In Proceedings of the International Conference on Knowledge Engineering and Knowledge Management. Springer, 429–440. · doi:10.1007/978-3-319-13704-9_33
  79. Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib. 2018. Deep learning for medical image processing: Overview, challenges and the future. Classif. BioApps (2018), 323–350. · DOI · Paper page
  80. Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. 2018. Meta-learning for semi-supervised few-shot classification. In Proceedings of the International Conference on Learning Representations.
  81. Jason Rennie and Ken Lang. 2008. The 20 Newsgroups Data Set. Retrieved from http://qwone.com/jason/20Newsgroups.
  82. 10.5555/1005332.1005336 · doi:10.5555/1005332.1005336
  83. 10.1016/S0020-0255(02)00369-9 · doi:10.1016/s0020-0255(02)00369-9
  84. 10.1007/s11263-015-0816-y · DOI · Paper page
  85. 10.1145/3446374 · DOI · Paper page
  86. 10.1109/TPAMI.2011.54 · DOI · Paper page
  87. 10.1145/3338501.3357372 · DOI · Paper page
  88. Haseeb Shah, Johannes Villmow, Adrian Ulges, Ulrich Schwanecke, and Faisal Shafait. 2019. An open-world extension to knowledge graph completion models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 3044–3051.
  89. Baoxu Shi and Tim Weninger. 2018. Open-world knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
  90. 10.1177/002383099804100410 · DOI · Paper page
  91. Lei Shu, Hu Xu, and Bing Liu. 2017. DOC: Deep open classification of text documents. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2911–2916.
  92. Lei Shu, Hu Xu, and Bing Liu. 2018. Unseen class discovery in open-world classification. arXiv:1801.05609. Retrieved from https://arxiv.org/abs/1801.05609. · arXiv
  93. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https://arxiv.org/abs/1409.1556. · arXiv
  94. Liwei Song, Vikash Sehwag, Arjun Nitin Bhagoji, and Prateek Mittal. 2020. A critical evaluation of open-world machine learning. arXiv:2007.04391. Retrieved from https://arxiv.org/abs/2007.04391. · arXiv
  95. Wallace Stevens. 2021. Efficient uncertainty estimation for open-set object detection. In Epistemic Uncertainty Estimation for Object Detection in Open-Set Conditions, 91.
  96. 10.1162/089120100561737 · DOI · Paper page
  97. Martin Thoma. 2017. The hasyv2 dataset. arXiv:1701.08380. Retrieved from https://arxiv.org/abs/1701.08380. · arXiv
  98. 10.1109/SLT.2010.5700816 · DOI
  99. Nikhita Vedula, Rahul Gupta, Aman Alok, and Mukund Sridhar. 2020. Automatic discovery of novel intents & domains from text utterances. arXiv:2006.01208. Retrieved from https://arxiv.org/abs/2006.01208. · arXiv
  100. Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2019. Towards open intent discovery for conversational text. arXiv:1904.08524. Retrieved from https://arxiv.org/abs/1904.08524. · arXiv
  101. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, Vol. 29. 3630–3638.
  102. Alex Hai Wang. 2010. Detecting spam bots in online social networking sites: A machine learning approach. In Proceedings of the IFIP Annual Conference on Data and Applications Security and Privacy. 335–342.
  103. Yuhan Wang, Weidong Xiao, Zhen Tan, and Xiang Zhao. 2021. Caps-OWKG: A capsule network model for open-world knowledge graph. Int. J. Mach. Learn. Cybernet. 12, 6 (2021), 1627–1637. · DOI
  104. 10.1109/ACCESS.2022.3155451 · DOI
  105. John Willes, James Harrison, Ali Harakeh, Chelsea Finn, Marco Pavone, and Steven Waslander. 2021. Bayesian embeddings for few-shot open world recognition. arXiv:2107.13682. Retrieved from https://arxiv.org/abs/2107.13682. · arXiv
  106. 10.1109/TPAMI.2008.79 · DOI · Paper page
  107. Qitian Wu, Hengrui Zhang, Xiaofeng Gao, Junchi Yan, and Hongyuan Zha. 2021. Towards open-world recommendation: An inductive model-based collaborative filtering approach. In Proceedings of the International Conference on Machine Learning. 11329–11339.
  108. Zhi-Fan Wu Tong Wei Jianwen Jiang Chaojie Mao Mingqian Tang and Yu-Feng Li. 2021. NGC: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 62–71.
  109. 10.1145/3308558.3313644 · DOI · Paper page
  110. 10.5555/2464909 · doi:10.5555/2464909
  111. J. Yang and Joseph. Coughlin. 2014. In-vehicle technology for self-driving cars: Advantages and challenges for aging drivers. Int. J. Automot. Technol. 15, 2 (2014), 333–340. · DOI
  112. Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi. 2021. Deep learning for person re-identification: A survey and outlook. IEEE Trans. Pattern Anal. Mach. Intell. (2021).
  113. Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto Iida, and Takeshi Naemura. 2019. Classification-reconstruction learning for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4016–4025.
  114. Kun Yue, Jiahui Wang, Xinbai Li, and Kuang Hu. 2020. Representation-based completion of knowledge graph with open-world data. In Proceedings of the 5th International Conference on Computer and Communication Systems (ICCCS’20). 1–8.
  115. He Zhang and Vishal M. Patel. 2016. Sparse representation-based open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39, 8 (2016), 1690–1696. · DOI · Paper page
  116. Tianchen Zhang, Zhongqin Bi, Meijing Shan, and Yongbin Li. 2022. IFGAN: Information fusion generative adversarial network for knowledge base completion. Expert Syst. (2022), e12984. · DOI
  117. 10.1007/s10115-015-0897-5 · DOI
  118. Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. 2021. OpenMix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). 9457–9465.
  119. 10.1109/SP.2012.16 · DOI · Paper page

Cited by

Metadata

Authors
Parmar, J., Chouhan, S., Raychoudhury, V., & Rathore, S.
Venue
ACM Computing Surveys
Published
2023-02-02
Volume
55
Issue
10
Pages
1-37
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W3164246203
Cited by
154
References
37
FWCI
9.61
Type
journal article
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.