CoChat

Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools

Lotfi, Fariba, Beheshti, Amin, Farhood, Helia, Pooshideh, Matineh, Jamzad, Mansour, Beigy, Hamid · 2023 · Algorithms
Cited by 24

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.

Abstract

In our digital age, data are generated constantly from public and private sources, social media platforms, and the Internet of Things. A significant portion of this information comes in the form of unstructured images and videos, such as the 95 million daily photos and videos shared on Instagram and the 136 billion images available on Google Images. Despite advances in image processing and analytics, the current state of the art lacks effective methods for discovering, linking, and comprehending image data. Consider, for instance, the images from a crime scene that hold critical information for a police investigation. Currently, no system can interactively generate a comprehensive narrative of events from the incident to the conclusion of the investigation. To address this gap in research, we have conducted a thorough systematic literature review of existing methods, from labeling and captioning to extraction, enrichment, and transforming image data into contextualized information and knowledge. Our review has led us to propose the vision of storytelling with image data, an innovative framework designed to address fundamental challenges in image data comprehension. In particular, we focus on the research problem of understanding image data in general and, specifically, curating, summarizing, linking, and presenting large amounts of image data in a digestible manner to users. In this context, storytelling serves as an appropriate metaphor, as it can capture and depict the narratives and insights locked within the relationships among data stored across different islands. Additionally, a story can be subjective and told from various perspectives, ranging from a highly abstract narrative to a highly detailed one.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/a16030135

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 An end-to-end storytelling example. Storytelling with image data could be a critical tool in preventing events like the 2016 Brussels bombings. By analyzing and connecting images from various sources, including security cameras and social media, authorities can identify potential threats and act preemptively to ensure public safety.
Figure 1 An end-to-end storytelling example. Storytelling with image data could be a critical tool in preventing events like the 2016 Brussels bombings. By analyzing and connecting images from various sources, including security cameras and social media, authorities can identify potential threats and act preemptively to ensure public safety.CC-BY · From the PDF
Figure 2 The proposed taxonomy for storytelling with image data.
Figure 2 The proposed taxonomy for storytelling with image data.CC-BY · From the PDF
Figure 3 Storytelling with image data research questions/motivations and keywords word cloud. ( a ) Research questions/motivations word cloud. ( b ) Search keywords word cloud.
Figure 3 Storytelling with image data research questions/motivations and keywords word cloud. ( a ) Research questions/motivations word cloud. ( b ) Search keywords word cloud.CC-BY · From the PDF
Figure 4 Yearwise distribution of the selected articles.
Figure 4 Yearwise distribution of the selected articles.CC-BY · From the PDF
Figure 5 Quality assessment process.
Figure 5 Quality assessment process.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 167

  1. Beheshti, A.; Ghodratnama, S.; Elahi, M.; Farhood, H. Social Data Analytics; CRC Press: Boca Raton, FL, USA, 2022. · doi:10.1201/9781003260141
  2. Lindeberg, T. Scale Invariant Feature Transform; KTH: Stockholm, Sweden, 2012. · DOI · Paper page
  3. Calonder, M.; Lepetit, V.; Strecha, C.; Fua, P. Brief: Binary robust independent elementary features. In Proceedings of the European Conference on Computer Vision, Heraklion, Greece, 5-11 September 2010; Springer: Berlin/Heidelberg, Germany, 2010; pp. 778-792. · DOI · Paper page
  4. Li, Q.; Li, J.; Sheng, J.; Cui, S.; Wu, J.; Hei, Y.; Peng, H.; Guo, S.; Wang, L.; Beheshti, A.; et al. A Survey on Deep Learning Event Extraction: Approaches and Applications. IEEE Trans. Neural Netw. Learn. Syst. 2022, Early Access. [CrossRef] · DOI
  5. Beheshti, A.; Tabebordbar, A.; Benatallah, B. istory: Intelligent storytelling with social data. In Proceedings of the Companion Proceedings of the Web Conference 2020, Taipei, Taiwan, 20-24 April 2020; pp. 253-256. · doi:10.1145/3366424.3383553
  6. Thöny, M.; Schnürer, R.; Sieber, R.; Hurni, L.; Pajarola, R. Storytelling in interactive 3D geographic visualization systems. ISPRS Int. J. Geo-Inf. 2018, 7, 123. [CrossRef] · doi:10.3390/ijgi7030123
  7. Beheshti, A. Knowledge base 4.0: Using crowdsourcing services for mimicking the knowledge of domain experts. In Proceedings of the 2022 IEEE International Conference on Web Services (ICWS), Barcelona, Spain, 11-15 July 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 425-427. · DOI
  8. Beheshti, A.; Benatallah, B.; Sheng, Q.Z.; Schiliro, F. Intelligent knowledge lakes: The age of artificial intelligence and big data. In Proceedings of the International Conference on Web Information Systems Engineering, Amsterdam, The Netherlands, 20-24 October 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 24-34. · doi:10.1007/978-981-15-3281-8_3
  9. Zhang, D.; Cui, M.; Yang, Y.; Yang, P.; Xie, C.; Liu, D.; Yu, B.; Chen, Z. Knowledge Graph-based image classification refinement. IEEE Access 2019, 7, 57678-57690. [CrossRef] · DOI
  10. Gong, W.; Zhang, X.; Chen, Y.; He, Q.; Beheshti, A.; Xu, X.; Yan, C.; Qi, L. DAWAR: Diversity-aware web APIs recommendation for mashup creation based on correlation graph. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, 11-15 July 2022; pp. 395-404. · DOI
  11. Sagheer, S.V.M.; George, S.N. A review on medical image denoising algorithms. Biomed. Signal Process. Control 2020, 61, 102036. [CrossRef] · DOI · Paper page
  12. Brooks, T.; Mildenhall, B.; Xue, T.; Chen, J.; Sharlet, D.; Barron, J.T. Unprocessing images for learned raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16-17 June 2019; pp. 11036-11045. · DOI · Paper page
  13. Saafin, W.; Schaefer, G. Pre-processing techniques for colour digital pathology image analysis. In Proceedings of the Annual Conference on Medical Image Understanding and Analysis, Edinburgh, UK, 11-13 July 2017; Springer: Berlin/Heidelberg, Germany, 2017; pp. 551-560. · DOI
  14. Krig, S. Image pre-processing. In Computer Vision Metrics; Springer: Berlin/Heidelberg, Germany, 2016; pp. 35-74. · DOI
  15. Heilbronner, R.; Barrett, S. Pre-processing. In Image Analysis in Earth Sciences; Springer: Berlin/Heidelberg, Germany, 2014; pp. 59-73. · doi:10.1007/978-3-642-10343-8
  16. Wang, Y.; Cao, Y.; Zha, Z.J.; Zhang, J.; Xiong, Z.; Zhang, W.; Wu, F. Progressive retinex: Mutually reinforced illumination-noise perception network for low-light image enhancement. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21-25 October 2019; pp. 2015-2023. · DOI
  17. Heilbronner, R.; Barrett, S. Image Analysis in Earth Sciences: Microstructures and Textures of Earth Materials; Springer Science & Business Media: Berlin, Germany, 2013; Volume 129. · doi:10.1007/978-3-642-10343-8
  18. Kim, W. Low-light image enhancement by diffusion pyramid with residuals. J. Vis. Commun. Image Represent. 2021, 81, 103364. [CrossRef] · DOI
  19. Pang, B.; Zhai, D.; Jiang, J.; Liu, X. Single image deraining via scale-space invariant attention neural network. In Proceedings of the 28th ACM International Conference on Multimedia, Virtual/Seattle, WA, USA, 12-16 October 2020; pp. 375-383. · DOI
  20. Liu, Y.; Yeoh, J.K.; Chua, D.K. Deep learning-based enhancement of motion blurred UAV concrete crack images. J. Comput. Civ. Eng. 2020, 34, 04020028. [CrossRef] · DOI
  21. Bai, C.; Liu, C.; Yu, X.; Peng, T.; Min, J.; Yan, S.; Dan, D.; Yao, B. Imaging enhancement of light-sheet fluorescence microscopy via deep learning. IEEE Photonics Technol. Lett. 2019, 31, 1803-1806. [CrossRef] · DOI
  22. Dong, J.; Dickfeld, T. Image integration in electroanatomic mapping. Herzschrittmachertherapie Elektrophysiologie 2007, 18, 122-130. [CrossRef] [PubMed] · doi:10.1007/s00399-007-0571-z
  23. Zach, C.; Pock, T.; Bischof, H. A globally optimal algorithm for robust tv-l 1 range image integration. In Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio De Janeiro, Brazil, 14-21 October 2007; IEEE: Piscataway, NJ, USA, 2007; pp. 1-8. · DOI · Paper page
  24. Dogra, A.; Kadry, S.; Goyal, B.; Agrawal, S. An efficient image integration algorithm for night mode vision applications. Multimed. Tools Appl. 2020, 79, 10995-11012. [CrossRef] · doi:10.1007/s11042-018-6631-z
  25. Bavirisetti, D.P.; Dhuli, R. Multi-focus image fusion using multi-scale image decomposition and saliency detection. Ain Shams Eng. J. 2018, 9, 1103-1117. [CrossRef] · doi:10.1016/j.asej.2016.06.011
  26. Wang, C.; Xu, C.; Wang, C.; Tao, D. Perceptual adversarial networks for image-to-image transformation. IEEE Trans. Image Process. 2018, 27, 4066-4079. [CrossRef] · DOI · Paper page
  27. Sarid, O.; Huss, E. Image formation and image transformation. Arts Psychother. 2011, 38, 252-255. [CrossRef] · DOI
  28. Jia, K.; Wang, X.; Tang, X. Image transformation based on learning dictionaries across image spaces. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 35, 367-380. [CrossRef] [PubMed] · DOI · Paper page
  29. Vial, A.; Stirling, D.; Field, M.; Ros, M.; Ritz, C.; Carolan, M.; Holloway, L.; Miller, A.A. The role of deep learning and radiomic feature extraction in cancer-specific predictive modelling: A review. Transl. Cancer Res. 2018, 7, 803-816. [CrossRef] · DOI · Paper page
  30. Tofighi, G.; Venetsanopoulos, A.N.; Raahemifar, K.; Beheshti, S.; Mohammadi, H. Hand posture recognition using K-NN and Support Vector Machine classifiers evaluated on our proposed HandReader dataset. In Proceedings of the 2013 18th International Conference on Digital Signal Processing (DSP), Fira, Greece, 1-3 July 2013; IEEE: Piscataway, NJ, USA, 2013; pp. 1-5. · DOI
  31. Zhao, S.; Ge, D.; Zhao, J.; Xiang, W. Fingerprint pre-processing and feature engineering to enhance agricultural products categorization. Future Gener. Comput. Syst. 2021, 125, 944-948. [CrossRef] · doi:10.1016/j.future.2021.07.005
  32. Wiggers, K.L.; Britto, A.S.; Heutte, L.; Koerich, A.L.; Oliveira, L.E.S. Document image retrieval using deep features. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil, 8-13 July 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1-8. · DOI
  33. Farhood, H.; He, X.; Jia, W.; Blumenstein, M.; Li, H. Counting people based on linear, weighted, and local random forests. In Proceedings of the 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, Australia, 29 November-1 December 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1-7. · DOI
  34. Singh, A.; Sharma, D.K. Image collection summarization: Past, present and future. In Data Visualization and Knowledge Engineering; Springer: Berlin/Heidelberg, Germany, 2020; pp. 49-78.
  35. Zou, X. A Review of object detection techniques. In Proceedings of the 2019 International Conference on Smart Grid and Electrical Automation (ICSGEA), Xiangtan, China, 10-11 August 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 251-254. · DOI · Paper page
  36. Ballard, D.H. Generalizing the Hough transform to detect arbitrary shapes. Pattern Recognit. 1981, 13, 111-122. [CrossRef] · DOI · Paper page
  37. Harris, C.; Stephens, M. A combined corner and edge detector. In Proceedings of the Alvey Vision Conference, Manchester, UK, 31 August-2 September 1988; Citeseer: Manchester, UK, 1988; Volume 15, pp. 1-6.
  38. He, K.; Zhang, X.; Ren, S.; Sun, J. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 1904-1916. [CrossRef] · DOI · Paper page
  39. Girshick, R. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7-13 December 2015; pp. 1440-1448. · DOI · Paper page
  40. Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23-28 June 2014; pp. 580-587. · DOI · Paper page
  41. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11-14 October 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 21-37. · DOI · Paper page
  42. Lin, T.Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22-29 October 2017; pp. 2980-2988. · DOI · Paper page
  43. Dash, S.K.; Acharya, S.; Pakray, P.; Das, R.; Gelbukh, A. Topic-based image caption generation. Arab. J. Sci. Eng. 2020, 45, 3025-3034. [CrossRef] · DOI
  44. Guo, H.; Zheng, K.; Fan, X.; Yu, H.; Wang, S. Visual attention consistency under image transforms for multi-label image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15-20 June 2019; pp. 729-739. · DOI · Paper page
  45. Ghiasi, G.; Lin, T.Y.; Le, Q.V. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15-20 June 2019; pp. 7036-7045. · DOI · Paper page
  46. Lotfi, F.; Jamzad, M.; Beigy, H. Automatic Image Annotation using Tag Relations and Graph Convolutional Networks. In Proceedings of the 2021 5th International Conference on Pattern Recognition and Image Analysis (IPRIA), Kashan, Iran, 28-29 April 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 1-6. · DOI
  47. Joseph, K.; Khan, S.; Khan, F.S.; Balasubramanian, V.N. Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 19-25 June 2021; pp. 5830-5840. · DOI · Paper page
  48. Lotfi, F.; Jamzad, M.; Beigy, H. Automatic Image Annotation Using Quantization Reweighting Function and Graph Neural Networks. In Proceedings of the International Conference on Service-Oriented Computing, Seville, Spain, 29 November-2 December 2022; Springer: Berlin/Heidelberg, Germany, 2022; pp. 46-60.
  49. Yang, P.; Luo, F.; Chen, P.; Li, L.; Yin, Z.; He, X.; Sun, X. Knowledgeable Storyteller: A Commonsense-Driven Generative Model for Visual Storytelling. In Proceedings of the IJCAI, Macao, China, 11-12 August 2019; Volume 3, p. 7. · DOI · Paper page
  50. Speer, R.; Havasi, C. Representing general relational knowledge in conceptnet 5. In Proceedings of the LREC, Istanbul, Turkey, 23-25 May 2012; Volume 2012, pp. 3679-3786.
  51. Chen, H.; Huang, Y.; Takamura, H.; Nakayama, H. Commonsense knowledge aware concept selection for diverse and informative visual storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual, 2-9 February 2021; Volume 35, pp. 999-1008. · DOI
  52. Li, J.; Shi, H.; Tang, S.; Wu, F.; Zhuang, Y. Informative visual storytelling with cross-modal rules. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21-25 October 2019; pp. 2314-2322. · DOI
  53. Aggarwal, C.C. Data classification. In Data Mining; Springer: Berlin/Heidelberg, Germany, 2015; pp. 285-344. · DOI · Paper page
  54. Ahmed, M. Data summarization: A survey. Knowl. Inf. Syst. 2019, 58, 249-273. [CrossRef] · DOI
  55. Mao, J.; Xu, W.; Yang, Y.; Wang, J.; Huang, Z.; Yuille, A. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv 2014, arXiv:1412.6632. · arXiv
  56. Karpathy, A.; Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7-12 June 2015; pp. 3128-3137. · DOI · Paper page
  57. Vinyals, O.; Toshev, A.; Bengio, S.; Erhan, D. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7-12 June 2015; pp. 3156-3164. · DOI · Paper page
  58. Rennie, S.J.; Marcheret, E.; Mroueh, Y.; Ross, J.; Goel, V. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21-26 July 2017; pp. 7008-7024. · DOI · Paper page
  59. Ren, Z.; Wang, X.; Zhang, N.; Lv, X.; Li, L.J. Deep reinforcement learning-based image captioning with embedding reward. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21-26 July 2017; pp. 290-298. · DOI · Paper page
  60. Gordon, D.; Kembhavi, A.; Rastegari, M.; Redmon, J.; Fox, D.; Farhadi, A. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-22 June 2018; pp. 4089-4098. · DOI · Paper page
  61. Patro, B.; Patel, S.; Namboodiri, V. Robust explanations for visual question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA, 1-5 March 2020; pp. 1577-1586. · DOI
  62. Wu, Q.; Wang, P.; Shen, C.; Reid, I.; Van Den Hengel, A. Are you talking to me? reasoned visual dialog generation through adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-22 June 2018; pp. 6106-6115. · DOI · Paper page
  63. Chen, C.; Mu, S.; Xiao, W.; Ye, Z.; Wu, L.; Ju, Q. Improving image captioning with conditional generative adversarial nets. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January-1 February 2019; Volume 33, pp. 8142-8150.
  64. Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.; Zemel, R.; Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, Lille, France, 7-9 July 2015; 2015; pp. 2048-2057.
  65. Bahdanau, D.; Cho, K.; Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv 2014, arXiv:1409.0473. · arXiv
  66. Mnih, V.; Heess, N.; Graves, A. Recurrent models of visual attention. Adv. Neural Inf. Process. Syst. 2014, 27, 2204-2212.
  67. Lu, J.; Xiong, C.; Parikh, D.; Socher, R. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21-26 July 2017; pp. 375-383. · DOI · Paper page
  68. Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; Zhang, L. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-22 June 2018; pp. 6077-6086. · DOI · Paper page
  69. Qin, Y.; Du, J.; Zhang, Y.; Lu, H. Look back and predict forward in image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16-17 June 2019; pp. 8367-8375. · DOI · Paper page
  70. Ke, L.; Pei, W.; Li, R.; Shen, X.; Tai, Y.W. Reflective decoding network for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea, 27 October-2 November 2019; pp. 8888-8897.
  71. Guo, L.; Liu, J.; Zhu, X.; Yao, P.; Lu, S.; Lu, H. Normalized and geometry-aware self-attention network for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14-19 June 2020; pp. 10327-10336. · DOI · Paper page
  72. Li, G.; Zhu, L.; Liu, P.; Yang, Y. Entangled transformer for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea, 27 October-2 November 2019; pp. 8928-8937.
  73. Yao, T.; Pan, Y.; Li, Y.; Mei, T. Exploring visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8-14 September 2018; pp. 684-699. · DOI · Paper page
  74. Yang, X.; Tang, K.; Zhang, H.; Cai, J. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16-17 June 2019; pp. 10685-10694. · DOI · Paper page
  75. Yao, T.; Pan, Y.; Li, Y.; Mei, T. Hierarchy parsing for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea, 27 October-2 November 2019; pp. 2621-2629.
  76. Aneja, J.; Deshpande, A.; Schwing, A.G. Convolutional image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-22 June 2018; pp. 5561-5570. · DOI · Paper page
  77. Wang, Q.; Chan, A.B. Cnn+ cnn: Convolutional decoders for image captioning. arXiv 2018, arXiv:1805.09019. · arXiv
  78. Herdade, S.; Kappeler, A.; Boakye, K.; Soares, J. Image captioning: Transforming objects into words. Adv. Neural Inf. Process. Syst. 2019, 32, 11135-11145.
  79. Johnson, J.; Karpathy, A.; Fei-Fei, L. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27-30 June 2016; pp. 4565-4574. · DOI · Paper page
  80. Yang, L.; Tang, K.; Yang, J.; Li, L.J. Dense captioning with joint inference and visual context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21-26 July 2017; pp. 2193-2202. · DOI · Paper page
  81. Kim, D.J.; Choi, J.; Oh, T.H.; Kweon, I.S. Dense relational captioning: Triple-stream networks for relationship-based captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16-17 June 2019; pp. 6271-6280. 100. · DOI
  82. Uehara, K., Mori, Y., Mukuta, Y., and Harada, T. (2022, January 25–29). ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer. Proceedings of the Companion Proceedings of the Web Conference 2022, Virtual Event/Lyon, France. · DOI
  83. Su, 2021, BERT-hLSTMs: BERT and hierarchical LSTMs for visual storytelling, Comput. Speech Lang., 67, 101169, 10.1016/j.csl.2020.101169 · DOI
  84. 2020, Image Collection Summarization Method Based on Semantic Hierarchies, AI, 1, 209, 10.3390/ai1020014 · DOI
  85. Sharma, V., Kumar, A., Agrawal, N., Singh, P., and Kulshreshtha, R. (2015, January 19–21). Image summarization using topic modelling. Proceedings of the 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), Kuala Lumpur, Malaysia. · DOI
  86. Camargo, J.E., and González, F.A. (2009, January 15–18). A multi-class kernel alignment method for image collection summarization. Proceedings of the Iberoamerican Congress on Pattern Recognition, Guadalajara, Mexico. · doi:10.1007/978-3-642-10268-4_64
  87. Sreelakshmi, 2021, Image Summarization Using Unsupervised Learning, Proceedings of the 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS), Volume 1, 100
  88. Chen, 2019, Extractive summarization of documents with images based on multi-modal RNN, Future Gener. Comput. Syst., 99, 186, 10.1016/j.future.2019.04.045 · DOI
  89. Qian, 2019, Social media based event summarization by user–text–image co-clustering, Knowl. Based Syst., 164, 107, 10.1016/j.knosys.2018.10.028 · doi:10.1016/j.knosys.2018.10.028
  90. Kuzovkin, D., Pouli, T., Cozot, R., Meur, O.L., Kervec, J., and Bouatouch, K. (2017, January 29–30). Context-aware clustering and assessment of photo collections. Proceedings of the Symposium on Computational Aesthetics, Los Angeles, CA, USA. · DOI
  91. Camargo, J.E., and González, F.A. (2011, January 4–6). Multimodal image collection summarization using non-negative matrix factorization. Proceedings of the 2011 6th Colombian Computing Congress (CCC), Manizales, Colombia. · DOI
  92. Huang, T.H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., and Batra, D. (2016, January 12–17). Visual storytelling. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA. · DOI · Paper page
  93. Yu, L., Bansal, M., and Berg, T.L. (2017). Hierarchically-attentive rnn for album summarization and storytelling. arXiv. · DOI
  94. Wang, X., Chen, W., Wang, Y.F., and Wang, W.Y. (2018). No metrics are perfect: Adversarial reward learning for visual storytelling. arXiv. · DOI · Paper page
  95. Wang, J., Fu, J., Tang, J., Li, Z., and Mei, T. (2018, January 2–7). Show, reward and tell: Automatic generation of narrative paragraph from photo stream by adversarial training. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA. · DOI
  96. Huang, Q., Gan, Z., Celikyilmaz, A., Wu, D., Wang, J., and He, X. (February, January 27). Hierarchically structured reinforcement learning for topically coherent visual story generation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
  97. Hsu, C.C., Chen, Z.Y., Hsu, C.Y., Li, C.C., Lin, T.Y., Huang, T.H., and Ku, L.W. (2020, January 7–12). Knowledge-enriched visual storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
  98. Jung, Y., Kim, D., Woo, S., Kim, K., Kim, S., and Kweon, I.S. (2020, January 7–12). Hide-and-tell: Learning to bridge photo streams for visual storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA. · DOI
  99. Hu, J.; Cheng, Y.; Gan, Z.; Liu, J.; Gao, J.; Neubig, G. What makes a good story? Designing composite rewards for visual storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7-12 February 2020; Volume 34, pp. 7969-7976. 118. Yu, Y.; Chung, J.; Yun, H.; Kim, J.; Kim, G. Transitional Adaptation of Pretrained Models for Visual Storytelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 19-25 June 2021; pp. 12658-12668. 119. · DOI
  100. Yu, Y., Chung, J., Yun, H., Kim, J., and Kim, G. (2021, January 19–25). Transitional Adaptation of Pretrained Models for Visual Storytelling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA. · DOI
  101. Lukin, S.M., Hobbs, R., and Voss, C.R. (2018). A pipeline for creative visual storytelling. arXiv. · DOI
  102. Xu, C., Yang, M., Li, C., Shen, Y., Ao, X., and Xu, R. (2021, January 2–9). Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual. · DOI
  103. Wang, E., Han, C., and Poon, J. (2022). RoViST: Learning Robust Metrics for Visual Storytelling. arXiv. · DOI
  104. Lukin, S.M.; Hobbs, R.; Voss, C.R. A pipeline for creative visual storytelling. arXiv 2018, arXiv:1807.08077. 120. Xu, C.; Yang, M.; Li, C.; Shen, Y.; Ao, X.; Xu, R. Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual, 2-9 February 2021; Volume 35, pp. 3022-3029. 121. Wang, E.; Han, C.; Poon, J. RoViST: Learning Robust Metrics for Visual Storytelling. arXiv 2022, arXiv:2205.03774. 122. Li, T.; Wang, H.; He, B.; Chen, C.W. Knowledge-enriched attention network with group-wise semantic for visual storytelling. IEEE Trans. Pattern Anal. Mach. Intell. 2022, Early Access. [CrossRef] 123. Knaflic, C.N. Storytelling with Data: Let's Practice!; John Wiley & Sons: Hoboken, NJ, USA, 2019. 124. Toker, D.; Conati, C.; Carenini, G.; Haraty, M. Towards adaptive information visualization: On the influence of user characteristics. In Proceedings of the International Conference on User Modeling, Adaptation, and Personalization, Montreal, QC, Canada, 16-20 July 2012; Springer: Berlin/Heidelberg, Germany, 2012; pp. 274-285. 125. Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 7-12 July 2002; pp. 311-318. 126. Banerjee, S.; Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA, 29 June 2005; pp. 65-72. 127. Anderson, P.; Fernando, B.; Johnson, M.; Gould, S. Spice: Semantic propositional image caption evaluation. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11-14 October 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 382-398. 128. Vedantam, R.; Lawrence Zitnick, C.; Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7-12 June 2015; pp. 4566-4575. 129. Yao, L.; Peng, N.; Weischedel, R.; Knight, K.; Zhao, D.; Yan, R. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January-1 February 2019; Volume 33, pp. 7378-7385. 130. Chen, Y.C.; Bansal, M. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv 2018, arXiv:1805.11080. 131. · DOI
  105. Knaflic, C.N. (2019). Storytelling with Data: Let’s Practice!, John Wiley & Sons.
  106. Toker, D., Conati, C., Carenini, G., and Haraty, M. (2012, January 16–20). Towards adaptive information visualization: On the influence of user characteristics. Proceedings of the International Conference on User Modeling, Adaptation, and Personalization, Montreal, QC, Canada. · DOI
  107. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA. · DOI · Paper page
  108. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
  109. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  110. Yao, L., Peng, N., Weischedel, R., Knight, K., Zhao, D., and Yan, R. (February, January 27). Plan-and-write: Towards better automatic storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
  111. Chen, Y.C., and Bansal, M. (2018). Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv. · DOI · Paper page
  112. Aafaq, 2019, Video description: A survey of methods, datasets, and evaluation metrics, ACM Comput. Surv. (CSUR), 52, 1, 10.1145/3355390 · DOI · Paper page
  113. Kilickaya, M., Erdem, A., Ikizler-Cinbis, N., and Erdem, E. (2016). Re-evaluating automatic metrics for image captioning. arXiv. · DOI · Paper page
  114. Sai, 2022, A survey of evaluation metrics used for NLG systems, ACM Comput. Surv. (CSUR), 55, 1, 10.1145/3485766 · DOI · Paper page
  115. Engel, 2013, Breaking with trends in pre-processing?, TrAC Trends Anal. Chem., 50, 96, 10.1016/j.trac.2013.04.015 · DOI · Paper page
  116. Hemanth, D.J., and Anitha, J. (2012, January 12–14). Image pre-processing and feature extraction techniques for magnetic resonance brain image analysis. Proceedings of the International Conference on Future Generation Communication and Networking, London, UK.
  117. Rajesh, S.D., Almeida, J.M., and Martins, A. (2019, January 17–20). Image Cleaning and Enhancement Technique for Underwater Mining. Proceedings of the OCEANS 2019, Marseille, France. · doi:10.1109/oceanse.2019.8866882
  118. Jiang, Z., Zhang, Y., Zou, D., Ren, J., Lv, J., and Liu, Y. (2020, January 14–19). Learning event-based motion deblurring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA. · DOI · Paper page
  119. Uijlings, 2013, Selective search for object recognition, Int. J. Comput. Vis., 104, 154, 10.1007/s11263-013-0620-5 · DOI · Paper page
  120. Lazebnik, 2006, Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Volume 2, 2169, 10.1109/CVPR.2006.68 · DOI · Paper page
  121. Shin, A., Ushiku, Y., and Harada, T. (2018, January 18–22). Customized Image Narrative Generation via Interactive Visual Question Generation and Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. · DOI
  122. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7-12 June 2015; pp. 1-9. 142. · DOI · Paper page
  123. Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., and Platt, J.C. (2015, January 7–12). From captions to visual concepts and back. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  124. Wu, Q., Shen, C., Liu, L., Dick, A., and Van Den Hengel, A. (2016, January 27–30). What value do explicit high level concepts have in vision to language problems?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. · DOI · Paper page
  125. Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., and Zhou, M. (2018, January 18–22). Visual question generation as dual task of visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. · DOI · Paper page
  126. Kipf, T.N.; Welling, M. Semi-supervised classification with graph convolutional networks. arXiv 2016, arXiv:1609.02907. 147. · arXiv
  127. Malakan, Z.M., Hassan, G.M., and Mian, A. (2022, January 5–8). Vision Transformer Based Model for Describing a Set of Images as a Story. Proceedings of the AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia. · DOI
  128. Malakan, Z.M.; Hassan, G.M.; Mian, A. Vision Transformer Based Model for Describing a Set of Images as a Story. In Proceedings of the AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, 5-8 December 2022; Springer: Berlin/Heidelberg, Germany, 2022; pp. 15-28. 148. Cao, S.; An, G.; Zheng, Z.; Wang, Z. Vision-Enhanced and Consensus-Aware Transformer for Image Captioning. IEEE Trans. Circuits Syst. Video Technol. 2022, 32, 7005-7018. [CrossRef] 149. Fang, Z.; Wang, J.; Hu, X.; Liang, L.; Gan, Z.; Wang, L.; Yang, Y.; Liu, Z. Injecting semantic concepts into end-to-end image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18-24 June 2022; pp. 18009-18019. 150. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929. 151. · DOI
  129. Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., and Liu, Z. (2022, January 18–24). Injecting semantic concepts into end-to-end image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA. · DOI · Paper page
  130. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
  131. Vinyals, 2016, Show and tell: Lessons learned from the 2015 mscoco image captioning challenge, IEEE Trans. Pattern Anal. Mach. Intell., 39, 652, 10.1109/TPAMI.2016.2587640 · DOI · Paper page
  132. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  133. Gu, J., Wang, G., Cai, J., and Chen, T. (2017, January 22–29). An empirical study of language cnn for image captioning. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy. · DOI · Paper page
  134. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
  135. Zhu, Z., Wei, Y., Wang, J., Gan, Z., Zhang, Z., Wang, L., Hua, G., Wang, L., Liu, Z., and Hu, H. (2022). Exploring Discrete Diffusion Models for Image Captioning. arXiv.
  136. Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., and Mei, T. (2022). Semantic-Conditional Diffusion Networks for Image Captioning. arXiv.
  137. Xu, S. (2022). CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning. arXiv.
  138. Cheung, J.C.K., and Li, X. (2012, January 8–12). Sequence clustering and labeling for unsupervised query intent discovery. Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, Seattle, WA, USA. · DOI
  139. Vedula, N., Lipka, N., Maneriker, P., and Parthasarathy, S. (2019). Towards open intent discovery for conversational text. arXiv.
  140. Narayanan, A., Chandramohan, M., Venkatesan, R., Chen, L., Liu, Y., and Jaiswal, S. (2017). graph2vec: Learning distributed representations of graphs. arXiv.
  141. Beheshti, A., Benatallah, B., Nouri, R., Chhieng, V.M., Xiong, H., and Zhao, X. (2017, January 6–10). Coredb: A data lake service. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore. · doi:10.1145/3132847.3133171
  142. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018, arXiv:1810.04805. 156. Zhu, Z.; Wei, Y.; Wang, J.; Gan, Z.; Zhang, Z.; Wang, L.; Hua, G.; Wang, L.; Liu, Z.; Hu, H. Exploring Discrete Diffusion Models for Image Captioning. arXiv 2022, arXiv:2211.11694. 157. Luo, J.; Li, Y.; Pan, Y.; Yao, T.; Feng, J.; Chao, H.; Mei, T. Semantic-Conditional Diffusion Networks for Image Captioning. arXiv 2022, arXiv:2212.03099. 158. Xu, S. CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning. arXiv 2022, arXiv:2210.04559. 159. Cheung, J.C.K.; Li, X. Sequence clustering and labeling for unsupervised query intent discovery. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, Seattle, WA, USA, 8-12 February 2012; pp. 383-392. 160. Vedula, N.; Lipka, N.; Maneriker, P.; Parthasarathy, S. Towards open intent discovery for conversational text. arXiv 2019, arXiv:1904.08524. 161. Narayanan, A.; Chandramohan, M.; Venkatesan, R.; Chen, L.; Liu, Y.; Jaiswal, S. graph2vec: Learning distributed representations of graphs. arXiv 2017, arXiv:1707.05005. 162. Beheshti, A.; Benatallah, B.; Nouri, R.; Chhieng, V.M.; Xiong, H.; Zhao, X. Coredb: A data lake service. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore, 6-10 November 2017; pp. 2451-2454. 163. Beheshti, A.; Benatallah, B.; Nouri, R.; Tabebordbar, A. CoreKG: A knowledge lake service. Proc. VLDB Endow. 2018, 11, 1942-1945. [CrossRef] 164. Li, N.; Liu, B.; Han, Z.; Liu, Y.S.; Fu, J. Emotion reinforced visual storytelling. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada, 10-13 June 2019; pp. 297-305. 165. Hsu, C.Y.; Chu, Y.W.; Huang, T.H.; Ku, L.W. Plot and Rework: Modeling Storylines for Visual Storytelling. arXiv 2021, arXiv:2105.06950. 166. · arXiv · doi:10.14778/3229863.3236230
  143. Li, N., Liu, B., Han, Z., Liu, Y.S., and Fu, J. (2019, January 10–13). Emotion reinforced visual storytelling. Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada. · DOI
  144. Hsu, C.Y., Chu, Y.W., Huang, T.H., and Ku, L.W. (2021). Plot and Rework: Modeling Storylines for Visual Storytelling. arXiv. · DOI
  145. Nahian, M., Al, S., Tasrin, T., Gandhi, S., Gaines, R., and Harrison, B. (2019, January 19–22). A hierarchical approach for visual storytelling using image description. Proceedings of the International Conference on Interactive Digital Storytelling, Little Cottonwood Canyon, UT, USA. · DOI
  146. Kim, T., Heo, M.O., Son, S., Park, K.W., and Zhang, B.T. (2018). Glac net: Glocal attention cascading networks for multi-image cued story generation. arXiv.
  147. Wang, R., Wei, Z., Li, P., Zhang, Q., and Huang, X. (2020, January 7–12). Storytelling from an image stream using scene graphs. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
  148. Zhang, B., Hu, H., and Sha, F. (2020). Visual storytelling via predicting anchor word embeddings in the stories. arXiv.
  149. Gonzalez-Rico, D., and Fuentes-Pineda, G. (2018). Contextualize, show and tell: A neural visual storyteller. arXiv.
  150. Nahian, M.; Al, S.; Tasrin, T.; Gandhi, S.; Gaines, R.; Harrison, B. A hierarchical approach for visual storytelling using image description. In Proceedings of the International Conference on Interactive Digital Storytelling, Little Cottonwood Canyon, UT, USA, 19-22 November 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 304-317. 167. Kim, T.; Heo, M.O.; Son, S.; Park, K.W.; Zhang, B.T. Glac net: Glocal attention cascading networks for multi-image cued story generation. arXiv 2018, arXiv:1805.10973. 168. Wang, R.; Wei, Z.; Li, P.; Zhang, Q.; Huang, X. Storytelling from an image stream using scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7-12 February 2020; Volume 34, pp. 9185-9192. 169. Zhang, B.; Hu, H.; Sha, F. Visual storytelling via predicting anchor word embeddings in the stories. arXiv 2020, arXiv:2001.04541. 170. Gonzalez-Rico, D.; Fuentes-Pineda, G. Contextualize, show and tell: A neural visual storyteller. arXiv 2018, arXiv:1806.00738. 171. Wang, P.; Zamora, J.; Liu, J.; Ilievski, F.; Chen, M.; Ren, X. Contextualized scene imagination for generative commonsense reasoning. arXiv 2021, arXiv:2112.06318. 172. Smilevski, M.; Lalkovski, I.; Madjarov, G. Stories for images-in-sequence by using visual and narrative components. In Proceed- ings of the International Conference on Telecommunications, Saint Malo, France, 26-28 June 2018; Springer: Berlin/Heidelberg, Germany, 2018; pp. 148-159. 173. · arXiv
  151. Smilevski, M., Lalkovski, I., and Madjarov, G. (2018, January 26–28). Stories for images-in-sequence by using visual and narrative components. Proceedings of the International Conference on Telecommunications, Saint Malo, France. · DOI
  152. Chandu, K., Nyberg, E., and Black, A.W. (August, January 28). Storyboarding of recipes: Grounded contextual generation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
  153. Salvador, A., Gundogdu, E., Bazzani, L., and Donoser, M. (2021, January 19–25). Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA. · DOI
  154. Chandu, K.; Nyberg, E.; Black, A.W. Storyboarding of recipes: Grounded contextual generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 28 July-2 August 2019; pp. 6040-6046. 174. Salvador, A.; Gundogdu, E.; Bazzani, L.; Donoser, M. Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 19-25 June 2021; pp. 15475-15484. 175. Nishimura, T.; Hashimoto, A.; Ushiku, Y.; Kameko, H.; Yamakata, Y.; Mori, S. Structure-aware procedural text generation from an image sequence. IEEE Access 2020, 9, 2125-2141. [CrossRef] 176. Qi, M.; Qin, J.; Huang, D.; Shen, Z.; Yang, Y.; Luo, J. Latent Memory-augmented Graph Transformer for Visual Storytelling. In Proceedings of the 29th ACM International Conference on Multimedia, Virtual, 20-24 October 2021; pp. 4892-4901. 177. · DOI
  155. Qi, M., Qin, J., Huang, D., Shen, Z., Yang, Y., and Luo, J. (2021, January 20–24). Latent Memory-augmented Graph Transformer for Visual Storytelling. Proceedings of the 29th ACM International Conference on Multimedia, Virtual. · DOI
  156. Hong, X.; Shetty, R.; Sayeed, A.; Mehra, K.; Demberg, V.; Schiele, B. Diverse and Relevant Visual Storytelling with Scene Graph Embeddings. In Proceedings of the 24th Conference on Computational Natural Language Learning, Online, 19-20 November 2020; pp. 420-430. 178. · DOI
  157. Joana, K.; Chan, S.W.; Chu, S.K. Quality assessment for digital stories by young authors. Data Inf. Manag. 2021, 5, 174-183. 179.
  158. Kusner, M., Sun, Y., Kolkin, N., and Weinberger, K. (2015, January 7–9). From word embeddings to document distances. Proceedings of the International Conference on Machine Learning, Lille, France.
  159. Lin, C.Y. (2004). Text Summarization Branches Out, Association for Computational Linguistics.
  160. Hsu, T.Y., Huang, C.Y., Hsu, Y.C., and Huang, T.H. (2019). Visual story post-editing. arXiv. · DOI · Paper page
  161. Ravi, H., Kafle, K., Cohen, S., Brandt, J., and Kapadia, M. (2021, January 10–17). AESOP: Abstract Encoding of Stories, Objects, and Pictures. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada. · DOI
  162. Hsu, C.Y., Chu, Y.W., Chen, V., Lo, K.C., Chen, C., Huang, T.H., and Ku, L.W. (2022, January 22–27). Learning to Rank Visual Stories From Human Ranking Data. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland. · DOI
  163. Kusner, M.; Sun, Y.; Kolkin, N.; Weinberger, K. From word embeddings to document distances. In Proceedings of the International Conference on Machine Learning, Lille, France, 7-9 July 2015; pp. 957-966. 180. Lin, C.Y. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out; Association for Computational Linguistics: Barcelona, Spain, 2004; pp. 74-81. 181. Hsu, T.Y.; Huang, C.Y.; Hsu, Y.C.; Huang, T.H. Visual story post-editing. arXiv 2019, arXiv:1906.01764. 182. Ravi, H.; Kafle, K.; Cohen, S.; Brandt, J.; Kapadia, M. AESOP: Abstract Encoding of Stories, Objects, and Pictures. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 10-17 October 2021; pp. 2052-2063. 183. Hsu, C.Y.; Chu, Y.W.; Chen, V.; Lo, K.C.; Chen, C.; Huang, T.H.; Ku, L.W. Learning to Rank Visual Stories From Human Ranking Data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, 22-27 May 2022; pp. 6365-6378. 184. Marin, J.; Biswas, A.; Ofli, F.; Hynes, N.; Salvador, A.; Aytar, Y.; Weber, I.; Torralba, A. Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images. IEEE Trans. Pattern Anal. Mach. Intell. 2019, 43, 187-203. [CrossRef] 185. Wang, H.; Lin, G.; Hoi, S.C.; Miao, C. Decomposed generation networks with structure prediction for recipe generation from food images. arXiv 2020, arXiv:2007.13374. 186. Nishimura, T.; Hashimoto, A.; Ushiku, Y.; Kameko, H.; Mori, S. Recipe Generation from Unsegmented Cooking Videos. arXiv 2022, arXiv:2209.10134. 187. · DOI · Paper page
  164. Wang, H., Lin, G., Hoi, S.C., and Miao, C. (2020). Decomposed generation networks with structure prediction for recipe generation from food images. arXiv.
  165. Nishimura, T., Hashimoto, A., Ushiku, Y., Kameko, H., and Mori, S. (2022). Recipe Generation from Unsegmented Cooking Videos. arXiv.
  166. Fain, M.; Twomey, N.; Ponikar, A.; Fox, R.; Bollegala, D. Dividing and conquering cross-modal recipe retrieval: From nearest neighbours baselines to sota. arXiv 2019, arXiv:1911.12763. 188. · arXiv
  167. Sakib, M.S.; Paulius, D.; Sun, Y. Approximate task tree retrieval in a knowledge network for robotic cooking. IEEE Robot. Autom. Lett. 2022, 7, 11492-11499. · DOI

Metadata

Authors
Lotfi, F., Beheshti, A., Farhood, H., Pooshideh, M., Jamzad, M., & Beigy, H.
Affiliations
  • School of Computing, Macquarie University, Sydney 2109, Australia
  • Computer Engineering Department, Sharif University of Technology, Tehran 14588-89694, Iran
Venue
Algorithms
Publisher
MDPI AG
Published
2023-03-02
Volume
16
Issue
3
Pages
135
ISSN
1999-4893
Cited by
24
References
19
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.