Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 24

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools

Lotfi, Fariba, Beheshti, Amin, Farhood, Helia, Pooshideh, Matineh, Jamzad, Mansour, Beigy, Hamid · 2023 · Algorithms

Abstract

In our digital age, data are generated constantly from public and private sources, social media platforms, and the Internet of Things. A significant portion of this information comes in the form of unstructured images and videos, such as the 95 million daily photos and videos shared on Instagram and the 136 billion images available on Google Images. Despite advances in image processing and analytics, the current state of the art lacks effective methods for discovering, linking, and comprehending image data. Consider, for instance, the images from a crime scene that hold critical information for a police investigation. Currently, no system can interactively generate a comprehensive narrative of events from the incident to the conclusion of the investigation. To address this gap in research, we have conducted a thorough systematic literature review of existing methods, from labeling and captioning to extraction, enrichment, and transforming image data into contextualized information and knowledge. Our review has led us to propose the vision of storytelling with image data, an innovative framework designed to address fundamental challenges in image data comprehension. In particular, we focus on the research problem of understanding image data in general and, specifically, curating, summarizing, linking, and presenting large amounts of image data in a digestible manner to users. In this context, storytelling serves as an appropriate metaphor, as it can capture and depict the narratives and insights locked within the relationships among data stored across different islands. Additionally, a story can be subjective and told from various perspectives, ranging from a highly abstract narrative to a highly detailed one.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/a16030135

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 An end-to-end storytelling example. Storytelling with image data could be a critical tool in preventing events like the 2016 Brussels bombings. By analyzing and connecting images from various sources, including security cameras and social media, authorities can identify potential threats and act preemptively to ensure public safety.
Figure 1 An end-to-end storytelling example. Storytelling with image data could be a critical tool in preventing events like the 2016 Brussels bombings. By analyzing and connecting images from various sources, including security cameras and social media, authorities can identify potential threats and act preemptively to ensure public safety.CC-BY · From the PDF
Figure 2 The proposed taxonomy for storytelling with image data.
Figure 2 The proposed taxonomy for storytelling with image data.CC-BY · From the PDF
Figure 3 Storytelling with image data research questions/motivations and keywords word cloud. ( a ) Research questions/motivations word cloud. ( b ) Search keywords word cloud.
Figure 3 Storytelling with image data research questions/motivations and keywords word cloud. ( a ) Research questions/motivations word cloud. ( b ) Search keywords word cloud.CC-BY · From the PDF
Figure 4 Yearwise distribution of the selected articles.
Figure 4 Yearwise distribution of the selected articles.CC-BY · From the PDF
Figure 5 Quality assessment process.
Figure 5 Quality assessment process.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 169

  1. Beheshti, A., Ghodratnama, S., Elahi, M., and Farhood, H. (2022). Social Data Analytics, CRC Press. · doi:10.1201/9781003260141
  2. Lindeberg, T. (2012). Scale Invariant Feature Transform, KTH. · DOI · Paper page
  3. Calonder, M., Lepetit, V., Strecha, C., and Fua, P. (2010, January 5–11). Brief: Binary robust independent elementary features. Proceedings of the European Conference on Computer Vision, Heraklion, Greece. · DOI · Paper page
  4. Li, Q., Li, J., Sheng, J., Cui, S., Wu, J., Hei, Y., Peng, H., Guo, S., Wang, L., and Beheshti, A. (IEEE Trans. Neural Netw. Learn. Syst., 2022). A Survey on Deep Learning Event Extraction: Approaches and Applications, IEEE Trans. Neural Netw. Learn. Syst., Early Access. · DOI
  5. Beheshti, A., Tabebordbar, A., and Benatallah, B. (2020, January 20–24). istory: Intelligent storytelling with social data. Proceedings of the Companion Proceedings of the Web Conference 2020, Taipei, Taiwan. · doi:10.1145/3366424.3383553
  6. Thöny, M., Schnürer, R., Sieber, R., Hurni, L., and Pajarola, R. (2018). Storytelling in interactive 3D geographic visualization systems. ISPRS Int. J. Geo-Inf., 7. · doi:10.3390/ijgi7030123
  7. Beheshti, A. (2022, January 11–15). Knowledge base 4.0: Using crowdsourcing services for mimicking the knowledge of domain experts. Proceedings of the 2022 IEEE International Conference on Web Services (ICWS), Barcelona, Spain. · DOI
  8. Beheshti, A., Benatallah, B., Sheng, Q.Z., and Schiliro, F. (2020, January 20–24). Intelligent knowledge lakes: The age of artificial intelligence and big data. Proceedings of the International Conference on Web Information Systems Engineering, Amsterdam, The Netherlands. · doi:10.1007/978-981-15-3281-8_3
  9. Zhang, 2019, Knowledge Graph-based image classification refinement, IEEE Access, 7, 57678, 10.1109/ACCESS.2019.2912627 · DOI
  10. Gong, W., Zhang, X., Chen, Y., He, Q., Beheshti, A., Xu, X., Yan, C., and Qi, L. (2022, January 11–15). DAWAR: Diversity-aware web APIs recommendation for mashup creation based on correlation graph. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain. · DOI
  11. Sagheer, 2020, A review on medical image denoising algorithms, Biomed. Signal Process. Control, 61, 102036, 10.1016/j.bspc.2020.102036 · DOI · Paper page
  12. Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., and Barron, J.T. (2019, January 16–17). Unprocessing images for learned raw denoising. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI · Paper page
  13. Saafin, W., and Schaefer, G. (2017, January 11–13). Pre-processing techniques for colour digital pathology image analysis. Proceedings of the Annual Conference on Medical Image Understanding and Analysis, Edinburgh, UK. · DOI
  14. Krig, S. (2016). Computer Vision Metrics, Springer. · DOI
  15. Heilbronner, R., and Barrett, S. (2014). Image Analysis in Earth Sciences, Springer. · doi:10.1007/978-3-642-10343-8
  16. Wang, Y., Cao, Y., Zha, Z.J., Zhang, J., Xiong, Z., Zhang, W., and Wu, F. (2019, January 21–25). Progressive retinex: Mutually reinforced illumination-noise perception network for low-light image enhancement. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France. · DOI
  17. Heilbronner, R., and Barrett, S. (2013). Image Analysis in Earth Sciences: Microstructures and Textures of Earth Materials, Springer Science & Business Media. · doi:10.1007/978-3-642-10343-8
  18. Kim, 2021, Low-light image enhancement by diffusion pyramid with residuals, J. Vis. Commun. Image Represent., 81, 103364, 10.1016/j.jvcir.2021.103364 · DOI
  19. Pang, B., Zhai, D., Jiang, J., and Liu, X. (2020, January 12–16). Single image deraining via scale-space invariant attention neural network. Proceedings of the 28th ACM International Conference on Multimedia, Virtual/Seattle, WA, USA. · DOI
  20. Liu, 2020, Deep learning–based enhancement of motion blurred UAV concrete crack images, J. Comput. Civ. Eng., 34, 04020028, 10.1061/(ASCE)CP.1943-5487.0000907 · DOI
  21. Bai, 2019, Imaging enhancement of light-sheet fluorescence microscopy via deep learning, IEEE Photonics Technol. Lett., 31, 1803, 10.1109/LPT.2019.2948030 · DOI
  22. Dong, 2007, Image integration in electroanatomic mapping, Herzschrittmachertherapie Elektrophysiologie, 18, 122, 10.1007/s00399-007-0571-z · doi:10.1007/s00399-007-0571-z
  23. Zach, C., Pock, T., and Bischof, H. (2007, January 14–21). A globally optimal algorithm for robust tv-l 1 range image integration. Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio De Janeiro, Brazil. · DOI · Paper page
  24. Dogra, 2020, An efficient image integration algorithm for night mode vision applications, Multimed. Tools Appl., 79, 10995, 10.1007/s11042-018-6631-z · doi:10.1007/s11042-018-6631-z
  25. Bavirisetti, 2018, Multi-focus image fusion using multi-scale image decomposition and saliency detection, Ain Shams Eng. J., 9, 1103, 10.1016/j.asej.2016.06.011 · doi:10.1016/j.asej.2016.06.011
  26. Wang, 2018, Perceptual adversarial networks for image-to-image transformation, IEEE Trans. Image Process., 27, 4066, 10.1109/TIP.2018.2836316 · DOI · Paper page
  27. Sarid, 2011, Image formation and image transformation, Arts Psychother., 38, 252, 10.1016/j.aip.2011.07.001 · DOI
  28. Jia, 2012, Image transformation based on learning dictionaries across image spaces, IEEE Trans. Pattern Anal. Mach. Intell., 35, 367, 10.1109/TPAMI.2012.95 · DOI
  29. Vial, 2018, The role of deep learning and radiomic feature extraction in cancer-specific predictive modelling: A review, Transl. Cancer Res., 7, 803, 10.21037/tcr.2018.05.02 · DOI · Paper page
  30. Tofighi, G., Venetsanopoulos, A.N., Raahemifar, K., Beheshti, S., and Mohammadi, H. (2013, January 1–3). Hand posture recognition using K-NN and Support Vector Machine classifiers evaluated on our proposed HandReader dataset. Proceedings of the 2013 18th International Conference on Digital Signal Processing (DSP), Fira, Greece. · DOI
  31. Zhao, 2021, Fingerprint pre-processing and feature engineering to enhance agricultural products categorization, Future Gener. Comput. Syst., 125, 944, 10.1016/j.future.2021.07.005 · doi:10.1016/j.future.2021.07.005
  32. Wiggers, K.L., Britto, A.S., Heutte, L., Koerich, A.L., and Oliveira, L.E.S. (2018, January 8–13). Document image retrieval using deep features. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil. · DOI
  33. Farhood, H., He, X., Jia, W., Blumenstein, M., and Li, H. (December, January 29). Counting people based on linear, weighted, and local random forests. Proceedings of the 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, Australia. · DOI
  34. Singh, A., and Sharma, D.K. (2020). Data Visualization and Knowledge Engineering, Springer.
  35. Zou, X. (2019, January 10–11). A Review of object detection techniques. Proceedings of the 2019 International Conference on Smart Grid and Electrical Automation (ICSGEA), Xiangtan, China. · DOI
  36. Ballard, 1981, Generalizing the Hough transform to detect arbitrary shapes, Pattern Recognit., 13, 111, 10.1016/0031-3203(81)90009-1 · DOI · Paper page
  37. Harris, 1988, A combined corner and edge detector, Proceedings of the Alvey Vision Conference, Volume 15, 1
  38. He, 2015, Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE Trans. Pattern Anal. Mach. Intell., 37, 1904, 10.1109/TPAMI.2015.2389824 · DOI · Paper page
  39. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile. · DOI · Paper page
  40. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA. · DOI · Paper page
  41. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands. · DOI · Paper page
  42. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy. · DOI · Paper page
  43. Dash, 2020, Topic-based image caption generation, Arab. J. Sci. Eng., 45, 3025, 10.1007/s13369-019-04262-2 · DOI
  44. Guo, H., Zheng, K., Fan, X., Yu, H., and Wang, S. (2019, January 15–20). Visual attention consistency under image transforms for multi-label image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI · Paper page
  45. Ghiasi, G., Lin, T.Y., and Le, Q.V. (2019, January 15–20). Nas-fpn: Learning scalable feature pyramid architecture for object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI · Paper page
  46. Lotfi, F., Jamzad, M., and Beigy, H. (2021, January 28–29). Automatic Image Annotation using Tag Relations and Graph Convolutional Networks. Proceedings of the 2021 5th International Conference on Pattern Recognition and Image Analysis (IPRIA), Kashan, Iran. · DOI
  47. Joseph, K., Khan, S., Khan, F.S., and Balasubramanian, V.N. (2021, January 19–25). Towards open world object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA. · DOI · Paper page
  48. Lotfi, F., Jamzad, M., and Beigy, H. (December, January 29). Automatic Image Annotation Using Quantization Reweighting Function and Graph Neural Networks. Proceedings of the International Conference on Service-Oriented Computing, Seville, Spain.
  49. Yang, P., Luo, F., Chen, P., Li, L., Yin, Z., He, X., and Sun, X. (2019, January 11–12). Knowledgeable Storyteller: A Commonsense-Driven Generative Model for Visual Storytelling. Proceedings of the IJCAI, Macao, China. · DOI
  50. Speer, R., and Havasi, C. (2012, January 23–25). Representing general relational knowledge in conceptnet 5. Proceedings of the LREC, Istanbul, Turkey.
  51. Chen, H., Huang, Y., Takamura, H., and Nakayama, H. (2021, January 2–9). Commonsense knowledge aware concept selection for diverse and informative visual storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual. · DOI
  52. Li, J., Shi, H., Tang, S., Wu, F., and Zhuang, Y. (2019, January 21–25). Informative visual storytelling with cross-modal rules. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France. · DOI
  53. Aggarwal, C.C. (2015). Data Mining, Springer. · DOI · Paper page
  54. Ahmed, 2019, Data summarization: A survey, Knowl. Inf. Syst., 58, 249, 10.1007/s10115-018-1183-0 · DOI
  55. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., and Yuille, A. (2014). Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv.
  56. Karpathy, A., and Fei-Fei, L. (2015, January 7–12). Deep visual-semantic alignments for generating image descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  57. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  58. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (2017, January 21–26). Self-critical sequence training for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA. · DOI · Paper page
  59. Ren, Z., Wang, X., Zhang, N., Lv, X., and Li, L.J. (2017, January 21–26). Deep reinforcement learning-based image captioning with embedding reward. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA. · DOI · Paper page
  60. Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., and Farhadi, A. (2018, January 18–22). Iqa: Visual question answering in interactive environments. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. · DOI · Paper page
  61. Patro, B., Patel, S., and Namboodiri, V. (2020, January 1–5). Robust explanations for visual question answering. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA. · DOI
  62. Wu, Q., Wang, P., Shen, C., Reid, I., and Van Den Hengel, A. (2018, January 18–22). Are you talking to me? reasoned visual dialog generation through adversarial learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. · DOI
  63. Chen, C., Mu, S., Xiao, W., Ye, Z., Wu, L., and Ju, Q. (February, January 27). Improving image captioning with conditional generative adversarial nets. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
  64. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015;, January 7–9). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France.
  65. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
  66. Mnih, 2014, Recurrent models of visual attention, Adv. Neural Inf. Process. Syst., 27, 2204
  67. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 21–26). Knowing when to look: Adaptive attention via a visual sentinel for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA. · DOI · Paper page
  68. Qin, Y., Du, J., Zhang, Y., and Lu, H. (2019, January 16–17). Look back and predict forward in image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI
  69. Ke, L., Pei, W., Li, R., Shen, X., and Tai, Y.W. (November, January 27). Reflective decoding network for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea.
  70. Huang, L., Wang, W., Chen, J., and Wei, X.Y. (November, January 27). Attention on attention for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea.
  71. Guo, L., Liu, J., Zhu, X., Yao, P., Lu, S., and Lu, H. (2020, January 14–19). Normalized and geometry-aware self-attention network for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA. · DOI · Paper page
  72. Li, G., Zhu, L., Liu, P., and Yang, Y. (November, January 27). Entangled transformer for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea.
  73. Yao, T., Pan, Y., Li, Y., and Mei, T. (2018, January 8–14). Exploring visual relationship for image captioning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany. · DOI · Paper page
  74. Yang, X., Tang, K., Zhang, H., and Cai, J. (2019, January 16–17). Auto-encoding scene graphs for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI · Paper page
  75. Yao, T., Pan, Y., Li, Y., and Mei, T. (November, January 27). Hierarchy parsing for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea.
  76. Aneja, J., Deshpande, A., and Schwing, A.G. (2018, January 18–22). Convolutional image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. · DOI · Paper page
  77. Wang, Q., and Chan, A.B. (2018). Cnn+ cnn: Convolutional decoders for image captioning. arXiv.
  78. Herdade, 2019, Image captioning: Transforming objects into words, Adv. Neural Inf. Process. Syst., 32, 11135
  79. Luo, Y., Ji, J., Sun, X., Cao, L., Wu, Y., Huang, F., Lin, C.W., and Ji, R. (2021). Dual-level collaborative transformer for image captioning. arXiv. · DOI · Paper page
  80. Johnson, J., Karpathy, A., and Fei-Fei, L. (2016, January 27–30). Densecap: Fully convolutional localization networks for dense captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. · DOI · Paper page
  81. Yang, L., Tang, K., Yang, J., and Li, L.J. (2017, January 21–26). Dense captioning with joint inference and visual context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA. · DOI · Paper page
  82. Kim, D.J., Choi, J., Oh, T.H., and Kweon, I.S. (2019, January 16–17). Dense relational captioning: Triple-stream networks for relationship-based captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. · DOI
  83. Uehara, K., Mori, Y., Mukuta, Y., and Harada, T. (2022, January 25–29). ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer. Proceedings of the Companion Proceedings of the Web Conference 2022, Virtual Event/Lyon, France. · DOI
  84. Su, 2021, BERT-hLSTMs: BERT and hierarchical LSTMs for visual storytelling, Comput. Speech Lang., 67, 101169, 10.1016/j.csl.2020.101169 · DOI
  85. 2020, Image Collection Summarization Method Based on Semantic Hierarchies, AI, 1, 209, 10.3390/ai1020014 · DOI
  86. Sharma, V., Kumar, A., Agrawal, N., Singh, P., and Kulshreshtha, R. (2015, January 19–21). Image summarization using topic modelling. Proceedings of the 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), Kuala Lumpur, Malaysia. · DOI
  87. Camargo, J.E., and González, F.A. (2009, January 15–18). A multi-class kernel alignment method for image collection summarization. Proceedings of the Iberoamerican Congress on Pattern Recognition, Guadalajara, Mexico. · doi:10.1007/978-3-642-10268-4_64
  88. Sreelakshmi, 2021, Image Summarization Using Unsupervised Learning, Proceedings of the 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS), Volume 1, 100
  89. Chen, 2019, Extractive summarization of documents with images based on multi-modal RNN, Future Gener. Comput. Syst., 99, 186, 10.1016/j.future.2019.04.045 · DOI
  90. Qian, 2019, Social media based event summarization by user–text–image co-clustering, Knowl. Based Syst., 164, 107, 10.1016/j.knosys.2018.10.028 · doi:10.1016/j.knosys.2018.10.028
  91. Kuzovkin, D., Pouli, T., Cozot, R., Meur, O.L., Kervec, J., and Bouatouch, K. (2017, January 29–30). Context-aware clustering and assessment of photo collections. Proceedings of the Symposium on Computational Aesthetics, Los Angeles, CA, USA. · DOI
  92. Camargo, J.E., and González, F.A. (2011, January 4–6). Multimodal image collection summarization using non-negative matrix factorization. Proceedings of the 2011 6th Colombian Computing Congress (CCC), Manizales, Colombia. · doi:10.1109/colomcc.2011.5936291
  93. Huang, T.H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., and Batra, D. (2016, January 12–17). Visual storytelling. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA. · DOI · Paper page
  94. Yu, L., Bansal, M., and Berg, T.L. (2017). Hierarchically-attentive rnn for album summarization and storytelling. arXiv. · DOI
  95. Wang, X., Chen, W., Wang, Y.F., and Wang, W.Y. (2018). No metrics are perfect: Adversarial reward learning for visual storytelling. arXiv. · DOI · Paper page
  96. Wang, J., Fu, J., Tang, J., Li, Z., and Mei, T. (2018, January 2–7). Show, reward and tell: Automatic generation of narrative paragraph from photo stream by adversarial training. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA. · DOI
  97. Huang, Q., Gan, Z., Celikyilmaz, A., Wu, D., Wang, J., and He, X. (February, January 27). Hierarchically structured reinforcement learning for topically coherent visual story generation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
  98. Hsu, C.C., Chen, Z.Y., Hsu, C.Y., Li, C.C., Lin, T.Y., Huang, T.H., and Ku, L.W. (2020, January 7–12). Knowledge-enriched visual storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
  99. Jung, Y., Kim, D., Woo, S., Kim, K., Kim, S., and Kweon, I.S. (2020, January 7–12). Hide-and-tell: Learning to bridge photo streams for visual storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA. · DOI
  100. Hu, J., Cheng, Y., Gan, Z., Liu, J., Gao, J., and Neubig, G. (2020, January 7–12). What makes a good story? Designing composite rewards for visual storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA. · DOI
  101. Yu, Y., Chung, J., Yun, H., Kim, J., and Kim, G. (2021, January 19–25). Transitional Adaptation of Pretrained Models for Visual Storytelling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA. · DOI
  102. Lukin, S.M., Hobbs, R., and Voss, C.R. (2018). A pipeline for creative visual storytelling. arXiv. · DOI
  103. Xu, C., Yang, M., Li, C., Shen, Y., Ao, X., and Xu, R. (2021, January 2–9). Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual. · DOI
  104. Wang, E., Han, C., and Poon, J. (2022). RoViST: Learning Robust Metrics for Visual Storytelling. arXiv. · DOI
  105. Li, T., Wang, H., He, B., and Chen, C.W. (IEEE Trans. Pattern Anal. Mach. Intell., 2022). Knowledge-enriched attention network with group-wise semantic for visual storytelling, IEEE Trans. Pattern Anal. Mach. Intell., Early Access. · DOI
  106. Knaflic, C.N. (2019). Storytelling with Data: Let’s Practice!, John Wiley & Sons.
  107. Toker, D., Conati, C., Carenini, G., and Haraty, M. (2012, January 16–20). Towards adaptive information visualization: On the influence of user characteristics. Proceedings of the International Conference on User Modeling, Adaptation, and Personalization, Montreal, QC, Canada. · DOI
  108. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA. · DOI · Paper page
  109. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
  110. Anderson, P., Fernando, B., Johnson, M., and Gould, S. (2016, January 11–14). Spice: Semantic propositional image caption evaluation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands. · DOI · Paper page
  111. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  112. Yao, L., Peng, N., Weischedel, R., Knight, K., Zhao, D., and Yan, R. (February, January 27). Plan-and-write: Towards better automatic storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
  113. Chen, Y.C., and Bansal, M. (2018). Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv. · DOI · Paper page
  114. Aafaq, 2019, Video description: A survey of methods, datasets, and evaluation metrics, ACM Comput. Surv. (CSUR), 52, 1, 10.1145/3355390 · DOI · Paper page
  115. Kilickaya, M., Erdem, A., Ikizler-Cinbis, N., and Erdem, E. (2016). Re-evaluating automatic metrics for image captioning. arXiv. · DOI
  116. Sai, 2022, A survey of evaluation metrics used for NLG systems, ACM Comput. Surv. (CSUR), 55, 1, 10.1145/3485766 · DOI · Paper page
  117. Engel, 2013, Breaking with trends in pre-processing?, TrAC Trends Anal. Chem., 50, 96, 10.1016/j.trac.2013.04.015 · DOI · Paper page
  118. Hemanth, D.J., and Anitha, J. (2012, January 12–14). Image pre-processing and feature extraction techniques for magnetic resonance brain image analysis. Proceedings of the International Conference on Future Generation Communication and Networking, London, UK.
  119. Rajesh, S.D., Almeida, J.M., and Martins, A. (2019, January 17–20). Image Cleaning and Enhancement Technique for Underwater Mining. Proceedings of the OCEANS 2019, Marseille, France. · doi:10.1109/oceanse.2019.8866882
  120. Jiang, Z., Zhang, Y., Zou, D., Ren, J., Lv, J., and Liu, Y. (2020, January 14–19). Learning event-based motion deblurring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA. · DOI · Paper page
  121. Uijlings, 2013, Selective search for object recognition, Int. J. Comput. Vis., 104, 154, 10.1007/s11263-013-0620-5 · DOI · Paper page
  122. Lazebnik, 2006, Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Volume 2, 2169, 10.1109/CVPR.2006.68 · DOI · Paper page
  123. Shin, A., Ushiku, Y., and Harada, T. (2018, January 18–22). Customized Image Narrative Generation via Interactive Visual Question Generation and Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. · DOI
  124. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  125. Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., and Platt, J.C. (2015, January 7–12). From captions to visual concepts and back. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  126. Wu, Q., Shen, C., Liu, L., Dick, A., and Van Den Hengel, A. (2016, January 27–30). What value do explicit high level concepts have in vision to language problems?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. · DOI · Paper page
  127. Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., and Zhou, M. (2018, January 18–22). Visual question generation as dual task of visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA. · DOI · Paper page
  128. Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv.
  129. Malakan, Z.M., Hassan, G.M., and Mian, A. (2022, January 5–8). Vision Transformer Based Model for Describing a Set of Images as a Story. Proceedings of the AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia. · DOI
  130. Cao, 2022, Vision-Enhanced and Consensus-Aware Transformer for Image Captioning, IEEE Trans. Circuits Syst. Video Technol., 32, 7005, 10.1109/TCSVT.2022.3178844 · DOI
  131. Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., and Liu, Z. (2022, January 18–24). Injecting semantic concepts into end-to-end image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA. · DOI
  132. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
  133. Vinyals, 2016, Show and tell: Lessons learned from the 2015 mscoco image captioning challenge, IEEE Trans. Pattern Anal. Mach. Intell., 39, 652, 10.1109/TPAMI.2016.2587640 · DOI · Paper page
  134. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. · DOI · Paper page
  135. Gu, J., Wang, G., Cai, J., and Chen, T. (2017, January 22–29). An empirical study of language cnn for image captioning. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy. · DOI
  136. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
  137. Zhu, Z., Wei, Y., Wang, J., Gan, Z., Zhang, Z., Wang, L., Hua, G., Wang, L., Liu, Z., and Hu, H. (2022). Exploring Discrete Diffusion Models for Image Captioning. arXiv.
  138. Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., and Mei, T. (2022). Semantic-Conditional Diffusion Networks for Image Captioning. arXiv.
  139. Xu, S. (2022). CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning. arXiv.
  140. Cheung, J.C.K., and Li, X. (2012, January 8–12). Sequence clustering and labeling for unsupervised query intent discovery. Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, Seattle, WA, USA. · DOI
  141. Vedula, N., Lipka, N., Maneriker, P., and Parthasarathy, S. (2019). Towards open intent discovery for conversational text. arXiv.
  142. Narayanan, A., Chandramohan, M., Venkatesan, R., Chen, L., Liu, Y., and Jaiswal, S. (2017). graph2vec: Learning distributed representations of graphs. arXiv.
  143. Beheshti, A., Benatallah, B., Nouri, R., Chhieng, V.M., Xiong, H., and Zhao, X. (2017, January 6–10). Coredb: A data lake service. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore. · doi:10.1145/3132847.3133171
  144. Beheshti, 2018, CoreKG: A knowledge lake service, Proc. VLDB Endow., 11, 1942, 10.14778/3229863.3236230 · doi:10.14778/3229863.3236230
  145. Li, N., Liu, B., Han, Z., Liu, Y.S., and Fu, J. (2019, January 10–13). Emotion reinforced visual storytelling. Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada. · DOI
  146. Hsu, C.Y., Chu, Y.W., Huang, T.H., and Ku, L.W. (2021). Plot and Rework: Modeling Storylines for Visual Storytelling. arXiv. · DOI
  147. Nahian, M., Al, S., Tasrin, T., Gandhi, S., Gaines, R., and Harrison, B. (2019, January 19–22). A hierarchical approach for visual storytelling using image description. Proceedings of the International Conference on Interactive Digital Storytelling, Little Cottonwood Canyon, UT, USA. · DOI
  148. Kim, T., Heo, M.O., Son, S., Park, K.W., and Zhang, B.T. (2018). Glac net: Glocal attention cascading networks for multi-image cued story generation. arXiv.
  149. Wang, R., Wei, Z., Li, P., Zhang, Q., and Huang, X. (2020, January 7–12). Storytelling from an image stream using scene graphs. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
  150. Zhang, B., Hu, H., and Sha, F. (2020). Visual storytelling via predicting anchor word embeddings in the stories. arXiv.
  151. Gonzalez-Rico, D., and Fuentes-Pineda, G. (2018). Contextualize, show and tell: A neural visual storyteller. arXiv.
  152. Wang, P., Zamora, J., Liu, J., Ilievski, F., Chen, M., and Ren, X. (2021). Contextualized scene imagination for generative commonsense reasoning. arXiv.
  153. Smilevski, M., Lalkovski, I., and Madjarov, G. (2018, January 26–28). Stories for images-in-sequence by using visual and narrative components. Proceedings of the International Conference on Telecommunications, Saint Malo, France. · DOI
  154. Chandu, K., Nyberg, E., and Black, A.W. (August, January 28). Storyboarding of recipes: Grounded contextual generation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
  155. Salvador, A., Gundogdu, E., Bazzani, L., and Donoser, M. (2021, January 19–25). Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA. · DOI
  156. Nishimura, 2020, Structure-aware procedural text generation from an image sequence, IEEE Access, 9, 2125, 10.1109/ACCESS.2020.3043452 · DOI
  157. Qi, M., Qin, J., Huang, D., Shen, Z., Yang, Y., and Luo, J. (2021, January 20–24). Latent Memory-augmented Graph Transformer for Visual Storytelling. Proceedings of the 29th ACM International Conference on Multimedia, Virtual. · DOI
  158. Hong, X., Shetty, R., Sayeed, A., Mehra, K., Demberg, V., and Schiele, B. (2020, January 19–20). Diverse and Relevant Visual Storytelling with Scene Graph Embeddings. Proceedings of the 24th Conference on Computational Natural Language Learning, Online. · DOI
  159. Joana, 2021, Quality assessment for digital stories by young authors, Data Inf. Manag., 5, 174
  160. Kusner, M., Sun, Y., Kolkin, N., and Weinberger, K. (2015, January 7–9). From word embeddings to document distances. Proceedings of the International Conference on Machine Learning, Lille, France.
  161. Lin, C.Y. (2004). Text Summarization Branches Out, Association for Computational Linguistics.
  162. Hsu, T.Y., Huang, C.Y., Hsu, Y.C., and Huang, T.H. (2019). Visual story post-editing. arXiv. · DOI
  163. Ravi, H., Kafle, K., Cohen, S., Brandt, J., and Kapadia, M. (2021, January 10–17). AESOP: Abstract Encoding of Stories, Objects, and Pictures. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada. · DOI
  164. Hsu, C.Y., Chu, Y.W., Chen, V., Lo, K.C., Chen, C., Huang, T.H., and Ku, L.W. (2022, January 22–27). Learning to Rank Visual Stories From Human Ranking Data. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland. · DOI
  165. Marin, 2019, Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images, IEEE Trans. Pattern Anal. Mach. Intell., 43, 187, 10.1109/TPAMI.2019.2927476 · DOI · Paper page
  166. Wang, H., Lin, G., Hoi, S.C., and Miao, C. (2020). Decomposed generation networks with structure prediction for recipe generation from food images. arXiv.
  167. Nishimura, T., Hashimoto, A., Ushiku, Y., Kameko, H., and Mori, S. (2022). Recipe Generation from Unsegmented Cooking Videos. arXiv.
  168. Fain, M., Twomey, N., Ponikar, A., Fox, R., and Bollegala, D. (2019). Dividing and conquering cross-modal recipe retrieval: From nearest neighbours baselines to sota. arXiv.
  169. Sakib, 2022, Approximate task tree retrieval in a knowledge network for robotic cooking, IEEE Robot. Autom. Lett., 7, 11492, 10.1109/LRA.2022.3191068 · DOI

Metadata

Authors
Lotfi, F., Beheshti, A., Farhood, H., Pooshideh, M., Jamzad, M., & Beigy, H.
Affiliations
  • School of Computing, Macquarie University, Sydney 2109, Australia
  • Computer Engineering Department, Sharif University of Technology, Tehran 14588-89694, Iran
Venue
Algorithms
Publisher
MDPI AG
Published
2023-03-02
Volume
16
Issue
3
Pages
135
ISSN
1999-4893
Cited by
24
References
17
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools. https://app.cochat.ai/papers/10.3390/a16030135
The paper
Lotfi, F., Beheshti, A., Farhood, H., Pooshideh, M., Jamzad, M., & Beigy, H. (2023). Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools. Algorithms. https://doi.org/10.3390/a16030135

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.