The DOI matches its Crossref record; 280 of the 280 references it cites were found in a registry or an index; none of the 144 sources its claims cite is checked for support yet.
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets
Open accessAbstract
The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This survey reviews deep multimodal learning for computer vision, organizing advances into six perspectives: multimodal data representation, multimodal fusion (traditional and deep learning-based), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. It also surveys applications and benchmark datasets and highlights limitations and challenges including dimensionality/data conflict, data availability, and real-time processing/scalability.
Key findings
- The survey organizes deep multimodal learning into six topics: multimodal data representation, multimodal fusion (traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning.
- It reviews conventional fusion methods (early, late, hybrid, kernel-based, graphical models, canonical correlation analysis) and deep fusion methods based on DBNs, DAEs, CNNs, RNNs, GANs, and attention mechanisms, noting model-specific tradeoffs such as DBNs largely ignoring spatiotemporal cues, DAEs missing spatiotemporal details, CNNs requiring large-scale multimodal datasets, RNNs having slow convergence, and GANs/attention being unstable to train.
- It surveys multimodal applications including object detection, visual tracking, semantic segmentation, human/face recognition, image retrieval, gesture recognition, image captioning, vision-and-language navigation, embodied question answering, video question answering, style transfer, medical data analysis, and autonomous systems. Table 2 reports prior best results such as face recognition on LFW (ACC = 98.43%) and CASIA-WebFace (ACC = 99.02%), gesture recognition on SKIG (ACC = 97.8%), video question answering on TDIUC (ACC = 88.20%), and autonomous systems on nuScenes (mAP = 28.9%; NDS = 44.9%).
- It presents benchmark datasets including RGB-D Object (300 object instances, 51 categories, 250,000 RGB-D images), BigBIRD (125 objects, 600 RGB-D point clouds, 600 12-megapixel images), Kinetics-400 (more than 300,000 video sequences across 400 classes), SceneNet RGB-D (5M RGB-D images), and nuScenes (1000 scenes, each about 20 s long, 23 classes, eight attributes).
Methods
- Survey/overview paper; no new experiments are reported. It reviews and synthesizes literature, compares its scope with previous surveys in Table 1, and organizes the field into six perspectives.
- It uses tables to summarize reviewed deep multimodal learning surveys, multimodal applications with technical details and best results (Table 2), and frequently used multimodal datasets (Tables 3 and 4), focusing on visual modalities such as RGB-D and RGB-optical flow.
- The main analysis is qualitative synthesis and taxonomy; no primary data collection, statistical meta-analysis, or systematic review protocol is stated.
Limitations
- The paper states main challenges of deep multimodal learning: dimensionality and data conflict, data availability, and real-time processing and scalability. It also states that despite impressive results, “no absolute conclusions can be drawn.”
- It lists model-specific limitations: DBNs largely ignore spatiotemporal cues and have computationally intensive inference; DAEs miss some spatiotemporal details and have many hidden parameters with time-consuming inference; CNNs require large-scale multimodal datasets and time-consuming inference; RNNs have slow convergence; GANs are unstable and tricky to train on discrete distributions; attention mechanisms have unstable training and a huge number of parameters.
- Evident from the text: as a survey, it does not provide new empirical validation or a quantitative meta-analysis beyond reporting prior results, and it does not state a systematic review protocol or inclusion criteria.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 144 cited sources checked · 0 supported · 0 flagged
In the paper
“Another crucial advantage of inter- and intra-modal interaction is the discriminating power of the perceptual model for multisensory stimuli by exploiting the potential synergies between modalities and their intrinsic representations [112].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal representation
What the sources say
- Not checked yetcites [112] Peng, Y., et al.: CCL: cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Trans. Multimed. 20(2), 405–420 (2017) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
- Also stated in: § Comparison with previous surveys · hedged
“However, there is still relatively little recent work directly addressing this research area [32–37].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Comparison with previous surveys
What the sources say
- Not checked yetcites [32] Marechal, C. et al.: Survey on AI-based multimodal methods for emotion detection. In: High-Performance Modelling and Simulation for Big Data Applications: Selected Results of the COST Action IC1406 cHiPSet, pp. 307–324 (2019) source
- Not checked yetcites [33] Radu, V., et al.: Multimodal deep learning for activity and context recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 157:1–157:27 (2018) source
- Not checked yetcites [34] Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process. Mag. 34, 96–108 (2017) source
- Not checked yetcites [35] Guo, W., Wang, J., Wang, S.: Deep multimodal representation learning: a survey. IEEE Access 7, 63373–63394 (2019) source
- Not checked yetcites [36] Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103(9), 1449–1477 (2015) source
- Not checked yetcites [37] Baltrušaitis, T., Ahuja, C., Morency, L.-P.: Multimodal Machine Learning: A Survey and Taxonomy. arXiv:1705.09406 (2017)
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: A majority of contemporary machine learning applications use multiple modalities, such as visual and textual data.
- Also stated in: § Comparison with previous surveys · hedged
None of the sources cited for this claim have been checked yet.
In the paper
“Recently, the valuable works of [32, 33] considered several multimodal practices that apply only to specific multimodal use cases and applications, such as emotion recognition [32], human activity and context recognition [33].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Comparison with previous surveys
What the sources say
- Not checked yetcites [32] Marechal, C. et al.: Survey on AI-based multimodal methods for emotion detection. In: High-Performance Modelling and Simulation for Big Data Applications: Selected Results of the COST Action IC1406 cHiPSet, pp. 307–324 (2019) source
- Not checked yetcites [33] Radu, V., et al.: Multimodal deep learning for activity and context recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 157:1–157:27 (2018) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Nonetheless, the applications discussed in these surveys include only a single modality as a data source for data-driven learning.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Comparison with previous surveys
What the sources say
- Not checked yetcites [4] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015) source
- Not checked yetcites [180] Hatcher, W.G., Yu, W.: A survey of deep learning: platforms, applications and emerging research trends. IEEE Access 6, 24411–24432 (2018) source
- Not checked yetcites [182] Pouyanfar, S., et al.: A survey on deep learning: algorithms, techniques, and applications. ACM Comput. Surv. 51, 92:1–92:36 (2018)
- Not checked yetcites [179] Guo, Y., et al.: Deep learning for visual understanding: a review. Neurocomputing 187, 27–48 (2016) source
- Not checked yetcites [181] Wu, X., Sahoo, D. Hoi, S.C.H.: Recent Advances in Deep Learning for Object Detection. arXiv:1908.03673 (2019) source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Comparison with previous surveys
What the sources say
- Not checked yetcites [34] Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process. Mag. 34, 96–108 (2017) source
- Not checked yetcites [36] Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103(9), 1449–1477 (2015) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“In addition to their use for unimodal tasks, RNNs have proved useful in many multimodal problems that require modeling long- and short-range dependencies across the input sequence, such as semantic segmentation [219] and image captioning [220].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [219] Abdulnabi, A.H., et al.: Multimodal recurrent neural networks with information transfer layers for indoor scene labeling. IEEE Trans. Multimed. 20, 1656–1671 (2018) source
- Not checked yetcites [220] Zhao, D., et al.: A multimodal fusion approach for image captioning. Neurocomputing 329, 476–485 (2019) source
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Since its introduction by Krizhevsky et al. [7] in 2012, the CNN model has been successfully applied to a wide range of multimodal applications, such as image dehazing [239, 240] and human activity recognition [241].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [7] Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2017) source
- Not checked yetcites [239] Zhang, S., et al.: Joint learning of image detail and transmission map for single image dehazing. Vis. Comput. 36, 305–316 (2020) source
- Not checked yetcites [240] Zhang, S., He, F.: DRCDN: learning deep residual convolutional dehazing networks. Vis. Comput. 36, 1797–1808 (2020) source
- Not checked yetcites [241] Basly, H., et al.: DTR-HAR: deep temporal residual representation for human activity recognition. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02064-y source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, an AM can be integrated into RNN models to improve the generalization capability of the former by capturing the most representative and discriminating patterns from heterogeneous datasets.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesNone of the sources cited for this claim have been checked yet.
In the paper
“Since its introduction, the AM has gained wide adoption in the computer vision community due to its spectral capabilities for many multimodal applications such as video description [233, 234], salient object detection [235], etc.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [233] Hori, C., et al.: Attention-based multimodal fusion for video description. In: IEEE International Conference on Computer Vision (ICCV), pp. 4203–4212 (2017) source
- Not checked yetcites [234] Huang, X., Wang, M., Gong, M.: Fine-grained talking face generation with video reinterpretation. Vis. Comput. 37, 95–105 (2021) source
- Not checked yetcites [235] Liu, Z., et al.: Multi-level progressive parallel attention guided salient object detection for RGB-D images. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01821-9 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIn the paper
“An adaptive multimodal mapping between two visual modalities (e.g., images and sentences) typically requires strong representations of the individual modalities [213].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [213] Ma, L., et al.: Multimodal convolutional neural networks for matching image and sentence. In: IEEE International Conference on Computer Vision (ICCV), pp. 2623–2631 (2015) source
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Probably not a substantial claim. 120
- It may state two separate claims as one. 28
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As shown in Fig. 3, the textual representation (i.e., a word embedding) is very sparse when compared to the image one, which makes it very challenging to combine these two different representations into a unified model.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal representation
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Figure 6 shows the coupling mechanism of two separate AEs (bimodal AE) for both modalities (audio and video) into a jointly shared representation hierarchy where the encoder and decoder components are independent of each other.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a powerful tool for feature extraction and dimensionality reduction, the DAE aims to learn how to efficiently represent manifolds where the training data is unbalanced or lacking.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, they also miss some spatiotemporal details in multimodal data.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Guo et al. [35] provided a comprehensive overview of deep multimodal learning frameworks and models, focusing on one of the main challenges of multimodal learning, namely multimodal representation.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Comparison with previous surveys
What the sources say
- Not checked yetcites Guo et al. [35] Guo, W., Wang, J., Wang, S.: Deep multimodal representation learning: a survey. IEEE Access 7, 63373–63394 (2019) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Here, the authors identified five levels of multimodal data combination: representation, translation, alignment, fusion, and co-learning.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Comparison with previous surveys
What the sources say
- Not checked yetcites Baltrušaitis et al. [37] Baltrušaitis, T., Ahuja, C., Morency, L.-P.: Multimodal Machine Learning: A Survey and Taxonomy. arXiv:1705.09406 (2017)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One of the main drawbacks of DAEs is that many hidden parameters have to be trained, and the inference process is time-consuming.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“the authors of [210–212] designed multimodal systems based on SAEs, where the encoder side of the architecture represents and compresses each unimodal feature separately, and the decoder side constructs the latent (shared) representation of the inputs in a unsupervised manner.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [210] Kim, J., Chung, K.: Multi-modal stacked denoising autoencoder for handling missing data in healthcare big data. IEEE Access 8, 104933–104943 (2020) source
- Not checked yetcites [211] Singh, V., et al.: Feature learning using stacked autoencoder for shared and multimodal fusion of medical images. In: Computational Intelligence: Theories, Applications and Future Directions, pp. 53–66 (2019) source
- Not checked yetcites [212] Said, A. B., et al.: Multimodal deep learning approach for joint EEG-EMG data compression and classification. In: IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6 (2017) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, most of the papers we reviewed are recent and have been published in high-quality conferences and journals such as the visual computer, ICCV, and CVPR.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Comparison with previous surveys
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Abstract
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Therefore, the AM acts as a contextual bridge between the encoding and decoding sides of an RNN to pay attention only to a particular position in the input representation.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The idea of AM is to focus on a particular position in the input image by computing the weighted sum of feature vectors and mapping them into a final contextual representation.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Structurally, GAN is a unified network consisting of two sub-networks, a generator network (G) and a discriminator network (D), which interact continuously during the learning process.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“What they have in common is that the sharing of the structure between all tasks can be learned jointly to improve performance.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multitask learning
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the only weakness of AM is that the training algorithm is unstable, which may affect the predictive power of the decision-making system.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“More recently, multitask learning (MTL) [108, 109] has become an increasingly popular topic in the deep learning community.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multitask learning
What the sources say
- Not checked yetcites [108] Ruder, S.: An Overview of Multi-Task Learning in Deep Neural Networks. arXiv:1706.05098 (2017)
- Not checked yetcites [109] Ruder, S., Bingel, J., Augenstein, I., Søgaard, A.: Latent Multi-task Architecture Learning. arXiv:1705.08142 (2018) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it requires a large-scale multimodal dataset to converge optimally during training, and the inference process is time-consuming.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In other words, it learns how to reduce some irrelevant attributes from a set of feature vectors.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Typically, MTL contributes to the performance of the target task based on knowledge gained from auxiliary tasks.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multitask learning
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generative adversarial networks (GANs) are part of deep generative architectures, designed to learn the data distribution through the adversarial learning.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it requires a large-scale multimodal dataset to converge optimally during training, and the inference process is time-consuming.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, the function that this network seeks to optimize is an adversarial loss function without any normalization.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it is extremely tricky to train the network when generating discrete distributions, and the process itself is unstable compared to other generative networks.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the model has a slow convergence rate due to the high computational cost in the hidden state transfer function.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Hard parameter sharing [110]: It consists of extracting a generic representation for different tasks using the same parameters. It is usually applied to avoid overfitting problems.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multitask learning
What the sources say
- Not checked yetcites [110] Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, the multimodal AM provides a robust solution for cross-modal data fusion by selecting the local fine-grained salient features in a multidimensional space and filtering out any hidden noise.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Generally, the alignment task can be divided into two subtasks: the attentional alignment task [170, 171] and the semantic alignment task [172, 173].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
What the sources say
- Not checked yetcites [170] Dapogny, A., et al.: Deep Entwined Learning Head Pose and Face Alignment Inside an Attentional Cascade with Doubly-Conditional fusion. arXiv:2004.06558 (2020) source
- Not checked yetcites [171] Yue, L., et al.: Attentional alignment network. In: BMVC (2018)
- Not checked yetcites [172] Liu, Z., et al.: Semantic Alignment: Finding Semantically Consistent Ground-truth for Facial Landmark Detection. arXiv:1903.10661 (2019) source
- Not checked yetcites [173] Hao, F., et al.: Collect and select: semantic alignment metric learning for few-shot learning. In: CVPR, pp. 8460–8469 (2019) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“More specifically, such models tend to model a series of nonlinear transformations by generating very abstract and informative features from highly complex datasets.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In terms of computational efficiency, GRU is a lightweight variant of LSTM since it can modulate the information flow without using its internal memory units.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the multimodal analysis, an attentional model can be designed to combine multiple modalities, each with its internal representation (e.g., spatial features, motion features, etc.).”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Multimodal alignment consists of linearly linking the features of two or more different modalities.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Patch-based methods first split each image into local patches and then consider the similarity between them by computing their cross-correlation and combination.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Recurrent neural networks (RNNs) [12] are a popular type of deep neural network architectures for processing sequential data of varying lengths.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [12] Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In recent years, the attention mechanism (AM) has become one of the most challenging tasks in computer vision and machine translation [232].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [232] Chaudhari, S., et al.: An Attentive Survey of Attention Models. arXiv:1904.02874 (2020)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the vanilla RNN model is typically incapable of capturing long-term dependencies in sequential data since they have no internal memory.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In particular, CNNs have demonstrated powerful generalization capabilities to learn how to represent visual appearance features from static data.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, unsupervised GAN is one of the most powerful generative models that can address scenarios where training data is lacking or some hidden concepts are missing.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Since their development in 2014, generative adversarial training algorithms have been widely used in various unimodal applications such as scene generation [17], image-to-image translation [18], and image super-resolution [224, 225].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [17] Turkoglu, M.O., Thong, W., Spreeuwers, L., Kicanaoglu, B.: A layer-based sequential framework for scene generation with GANs. arXiv:1902.00671 (2019) source
- Not checked yetcites [18] Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv:1611.07004 (2018) source
- Not checked yetcites [224] Song, H., et al.: $${\rm S}^2{{\rm RGAN}}$$S: sonar-image super-resolution based on generative adversarial network. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01986-3 source
- Not checked yetcites [225] Ma, T., Tian, W.: Back-projection-based progressive growing generative adversarial network for single image super-resolution. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01843-3 source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It also can handle time series and dynamic media such as text and video sequences.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Specifically, the MTL paradigm frequently arises in a context close to multimodal concepts.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multitask learning
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is also capable of modeling spatial cues from multimodal data streams with an increased number of parameters.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Feature-based methods detect and extract a set of matching features that should be structurally consistent to describe their spatial patterns.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“They learn to map input activations to the next hierarchy level and then transfer hidden states to the outputs using the recurrent feedback, which gives them the capacity to learn useful features from the previous states, unlike other deep feedforward networks such as CNNs, DBNs, etc.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the number of parameters to be trained is huge compared to other deep networks such as RNNs, CNNs, etc.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Specifically, multimodal image alignment provides a spatial mapping capability between images taken by sensors of different modalities, which may be categorized into feature-based [167, 168] and patch-based [165, 166] methods.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
What the sources say
- Not checked yetcites [167] Chen, C., et al. Progressive Feature Alignment for Unsupervised Domain Adaptation. arXiv:1811.08585 (2019) source
- Not checked yetcites [168] Jin, X., et al.: Feature Alignment and Restoration for Domain Generalization and Adaptation. arXiv:2006.12009 (2020)
- Not checked yetcites [165] Ščupáková, K., et al.: A patch-based super resolution algorithm for improving image resolution in clinical mass spectrometry. Sci. Rep. 9, 2915 (2019) source
- Not checked yetcites [166] Bashiri, F.S., et al.: Multi-modal medical image registration with full or partial data: a manifold learning approach. J. Imag. 5, 5 (2019) source
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“The main properties that distinguish CNNs from other models include their ability to capture local connectivity between units, to share weights across layers, and to block a sequence of hidden layers [4].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [4] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“To this end, several popular variants have been developed to efficiently handle this constraint and the gradient vanishing problem with impressive results, including long short-term memory (LSTM) [13] and gated recurrent linear units (GRU) [14].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [13] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997) source
- Not checked yetcites [14] Li, S., Li, W., Cook, C., Zhu, C., Gao, Y.: Independently recurrent neural network (IndRNN): building a longer and deeper RNN. arXiv:1803.04831 (2018) source
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Recently, with the advent of robust and low-cost RGB-D sensors such as the Kinect, the computer vision community has turned its attention to integrating RGB images and corresponding depth maps (2.5D) into multimodal architectures as shown in Fig. 7.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Convolutional neural networks (CNNs or ConvNets) are a class of deep feed-forward neural networks whose main purpose is to extract spatial patterns from visual input signals [20, 22].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [20] Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artif Intell Rev (2020) source
- Not checked yetcites [22] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2015) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, a multimodal CNN serves as a powerful feature extractor that learns local cross-modal features from visual modalities.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In contrast to single-task learning, the idea behind this paradigm is to learn a shared representation that can be used to respond to several tasks in order to ensure better generalizability.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multitask learning
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A formalism for integrating an AM into the basic RNN model was developed by Bahdanau et al. [1].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [1] Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 (2016)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Soft parameter sharing [111]: It consists of extracting a set of feature vectors and simultaneously drawing similarity relationships between them.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multitask learning
What the sources say
- Not checked yetcites [111] Duong, L., Cohn, T., Bird, S., Cook, P. low resource dependency parsing: cross-lingual parameter sharing in a neural network parser. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 845–850 (2015) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Since the encoding side of an RNN generates a fixed-length feature vector from its input sequence, this can lead to very tedious and time-consuming parameter tuning.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Historically, they were first developed by Goodfellow et al. [16], which demonstrated the ability to generate realistic and reasonably impractical representations from noisy data domains.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Fusion algorithms
What the sources say
- Not checked yetcites [16] Goodfellow, I.J., et al.: Generative adversarial networks. arXiv:1406.2661 (2014)
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The task has attracted a lot of interest because of its enormous relevance in many real-world applications, including video surveillance [82], autonomous driving [83], etc.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [82] 1.Mohanapriya, D., Mahesh, K.: Chapter 5—an efficient framework for object tracking in video surveillance. In: The Cognitive Approach in Cloud Computing and Internet of Things Technologies for Surveillance Tracking Systems, pp. 65–74 (2020) source
- Not checked yetcites [83] Rangesh, A., Trivedi, M.M.: No blind spots: full-surround multi-object tracking for autonomous vehicles using cameras and LiDARs. IEEE Trans. Intelli. Veh. 4, 588–599 (2019) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Transfer learning (TL) [70] is one of the model regularization techniques that have proven their effectiveness for training deep models with a limited amount of available data and avoiding overfitting problems.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal transfer learning
What the sources say
- Not checked yetcites [70] Shao, L., Zhu, F., Li, X.: Transfer learning for visual categorization: a survey. IEEE Trans. Neural Netw. Learn. Syst. 26, 1019–1034 (2015) source
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Training a deep model from scratch requires a large amount of labeled data to achieve an acceptable level of performance.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“High-precision object recognition systems with multiple sensors are aware of external noise and environmental sensitivity (e.g., lighting variations, occlusion, etc.).”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“This function allows training a one-stage detector with high accuracy by reducing the level of artifacts.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is a cascaded version of R-CNN aimed at achieving a better compromise between object localization and classification.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The objective is to project a set of synthesized features in order to make the model more generalizable toward the recognition of the unseen class in test samples [135].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Zero-shot learning
What the sources say
- Not checked yetcites [135] Liu, Y., Tuytelaars, T.: A: deep multi-modal explanation model for zero-shot learning. IEEE Trans. Image Process. 29, 4788–4803 (2020) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In order to significantly reduce this complexity, the deep learning community has pioneered a new generation of CNN-based frameworks.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, due to their complexity, this type of solution is not cost-effective, both in terms of training duration and memory consumption.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, MOT methods are mainly designed to optimize the dynamic matching of the objects of interest detected in each frame.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As mentioned before, two-stage detectors are generally based on a combination of a CNN model to perform classification and a patch proposal module to generate candidate regions like RPNs.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Multi-object tracking (MOT) has been extensively explored in recent literature for a wide range of applications [86, 138].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [86] Ciaparrone, G., et al.: Deep learning in video multi-object tracking: a survey. Neurocomputing 381, 61–88 (2020) source
- Not checked yetcites [138] Zhang, X., et al.: Online multi-object tracking with pedestrian re-identification and occlusion processing. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01854-0 source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The attentional alignment task is based on the attentional mapping between the features of the input modality and the target one, while the semantic alignment task takes the form of an alignment method that directly provides alignment capabilities to a predictive model.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In contrast, SSD (single-shot multibox detector) [78] has reduced the needs of the patches’ proposal network and, thus, accelerated the object detection pipeline.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [78] Liu, W. et al.: SSD: single shot multibox detector, pp. 21–37. arXiv:1512.02325 (2016) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In other words, it addresses multi-class learning problems when some classes do not have sufficient training data.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Zero-shot learning
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When modeling multimodal data, several compromises have to be made between system performance, computational burden, and processing speed.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Multimodal MOT is a universal aspect of MOT aimed at ensuring the accuracy of autonomous systems by mapping the motion sequence of dynamic objects [194].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [194] Zhang, W., et al.: Robust Multi-Modality Multi-Object Tracking. arXiv:1909.03850 (2019) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the context of multimodality, a multimodal mapping scheme typically combines visual and semantic attributes using only data related to the seen classes.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Zero-shot learning
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These techniques have proven effective for the accurate detection of multiple objects under normal and extreme environmental conditions.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“The recent literature has shown that it is attractive to learn shared representations from the complementarity and synergies between several modalities for increasing the discriminatory power of models [190].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [190] Michael, Y.B., Rosenhahn, V.M.: Multimodal Scene Understanding, 1st edn. Academic Press, London (2019)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this respect, two-stage detectors are generally slow in terms of real-time processing.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: In some vision applications, such as autonomous driving, security, and video surveillance, real-time conditions become necessary.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In practice, there are several possible detection methods. The most typical of these is to apply the classifier to an arbitrary region of the image or to a range of different shapes and scales.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As similar to the principle of YOLO and SSD detectors, RetinaNet [79] takes only one stage to detect dense objects by producing multi-scale semantic feature maps using a feature pyramid network (FPN) backbone and the ResNet model.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [79] Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv:1708.02002 (2018) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, the learning process can learn the data distribution in a non-parallel manner and ensure its synchronization over time.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal transfer learning
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Training a deep model from scratch requires a large amount of labeled data to achieve an acceptable level of performance.
None of the sources cited for this claim have been checked yet.
In the paper
“This is why zero-shot learning [130] takes place. This supervised learning approach opens up many valuable applications such as object detection [131], object classification and retrieval of videos [141], especially when appropriate datasets are missing.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Zero-shot learning
What the sources say
- Not checked yetcites [130] Wang, W., et al.: A survey of zero-shot learning: settings, methods, and applications. ACM Trans. Intell. Syst. Technol. 10, 13:1–13:37 (2019)
- Not checked yetcites [131] Wei, L., et al.: A single-shot multi-level feature reused neural network for object detection. Vis. Comput. (2020). https://doi.org/10.1007/s00371-019-01787-3 source
- Not checked yetcites [141] Parida, K., et al.: Coordinated joint multimodal embeddings for generalized audio-visual zero-shot classification and retrieval of videos. In: CVPR, pp. 3251–3260 (2020) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.
In the paper
“Cascade R-CNN [26] is one of the most effective multi-stage detectors that have proven their robustness over one and two-stage methods.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [26] Cai, Z., Vasconcelos, N.: Cascade R-CNN: high quality object detection and instance segmentation. arXiv:1906.09756 (2019) source
Strength:boosted· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In general, multimodality has been employed in many vision tasks and applications, such as face recognition and image retrieval.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“More recently, the availability of low-cost and robust sensors (e.g., RGB-D sensors, stereo, etc.) has encouraged the computer vision community to focus on combining the RGB modality with other sensing modalities.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the most common approach is to concatenate heterogeneous features from different modalities to generate an artificial multimodal representation.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Such methods tend to use GAN models to synthesize and reconstruct the visual features of the unseen classes, resulting in high accuracy classification and ensuring a balance between seen and unseen class labels [136, 137].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Zero-shot learning
What the sources say
- Not checked yetcites [136] Xian, Y., Lorenz, T., Schiele, B., Akata, Z.: Feature generating networks for zero-shot learning. In: CVPR, pp. 5542–5551 (2018) source
- Not checked yetcites [137] Kumar, Y. et al.: Harnessing GANs for Zero-shot Learning of New Classes in Visual Speech Recognition. arXiv:1901.10139 (2020) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.
In the paper
“To deal with the class imbalance in the training phase, a novel loss function called “focal loss” is considered by [79].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [79] Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv:1708.02002 (2018) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The most popular use of semantic alignment is to create a dataset with associated labels and then generate a semantically aligned dataset.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both of these tasks have proven effective in multimodal alignment, where attentional alignment features are better able to take into account the long-term dependencies between different concepts.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, multi-object detection in both indoor and outdoor environments under varying environmental and lighting conditions remains one of the major challenges facing the computer vision community.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It can learn multi-scale feature representation from multi-resolution images.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Over the last few decades, most deep learning-based object tracking systems have been based on CNN architectures [84, 139].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [84] Liu, L., et al.: Deep learning for generic object detection: a survey. Int. J. Comput. Vis. 128, 261–318 (2020) source
- Not checked yetcites [139] Abbass, M.Y., et al.: Efficient object tracking using hierarchical convolutional features model and correlation filters. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01833-5 source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For decades, visual tracking has been one of the major challenges for the computer vision community.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its capability to detect objects at different scales enables it to enhance the robustness of the entire chain.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The attentional alignment task is based on the attentional mapping between the features of the input modality and the target one, while the semantic alignment task takes the form of an alignment method that directly provides alignment capabilities to a predictive model.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Deep multimodal learning architectures > Multimodal alignment
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In [160], Wang et al. introduced a residual CNN architecture and loss network to transfer the artistic style of the input picture across multiple scales and dimensions.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [160] Wang, X., et al.: Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer. arXiv:1612.01895 (2017) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Similarly, Peri et al. [53] proposed a multimodal framework that encodes both images and captions using CNN and RNN as an intermediate level representation and then decodes these multimodal representations into a new caption that is similar to the input.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [53] Peri, D., Sah, S., Ptucha, R.: Show, Translate and Tell. arXiv:1903.06275 (2019) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The challenge of action recognition is mainly related to the difficulty of extracting body silhouettes from foreground rigid objects to focus on their emotions [96].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [96] Escalera, S., Athitsos, V., Guyon, I.: Challenges in multimodal gesture recognition. J. Mach. Learn. Res. 17, 1–54 (2016)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For instance, RGB-D cameras (e.g., Microsoft Kinect, Asus Xtion, and so on) can provide color and pixel-wise depth information, characterizing the distance of visual objects in a complex scene [199].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [199] Shamwell, E.J., et al.: Unsupervised deep visual-inertial odometry with online error correction for RGB-D imagery. IEEE Trans. Pattern Anal. Mach. Intell. (2019). https://doi.org/10.1109/TPAMI.2019.2909895 source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The authors of [159] proposed a universal graph-based style transfer to transform multimodal features by matching style patterns and semantic content and appearance in a way that avoids the lack of flexibility in real-world scenarios.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [159] Zhang, Y. et al.: Multimodal style transfer via graph cuts. In: ICCV, pp. 5943–5951 (2019) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Note that face detection, face identification, and face reconstruction are subtasks of face recognition [50].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [50] Wang, M., Deng, W.: Deep Face Recognition: A Survey. arXiv:1804.06655 (2019)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The authors of [128] presented an unsupervised image captioning framework based on a new alignment method that allows the simultaneous integration of visual and textual streams through semantic learning of multimodal embeddings of the language and vision domains.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [128] Laina, I., et al.: Towards unsupervised image captioning with shared multimodal embeddings. In: ICCV, pp. 7414–7424 (2019) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The authors of [93] introduced a biometric system that combines biometric traits from different modalities (face and iris) to establish an individual’s identity.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [93] Ammour, B., Boubchir, L., Bouden, T., Ramdani, M.: Face-Iris multimodal biometric identification system. Electronics 9, 85 (2020) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“It is essentially a weighted sum of the error (loss) between the input content and the output image and the loss between the original style and the applied style [161].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [161] Jing, Y., et al.: Neural Style Transfer: A Review. arXiv:1705.04058 (2018)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Most recently, Biten et al. [52] combined both visual and textual data to generate captions across two stages: template caption generation stage and entity insertion stage.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [52] Biten, A.F., Gomez, L., Rusiñol, M., Karatzas, D.: Good News, Everyone! Context driven entity-aware captioning for news images. arXiv:1904.01475 (2019) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For instance, the Mask R-CNN model offers the possibility of locating instances of objects with class labels and segmenting them with semantic masks.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In [73], Koo et al. developed a multimodal human recognition framework based on face and body information extracted from deep CNNs. They employed the late fusion policy to merge the high-level features across the different modalities.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [73] Koo, J.H., Cho, S.W., Baek, N.R., Kim, M.C., Park, K.R.: CNN-based multimodal human recognition in surveillance environments. Sensors 18, 3040 (2018) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“Regarding the multimodal dimensions of gesture recognition, the authors of [97] proposed a multi-stream architecture based on the RNN (LSTM) model to capture spatial-temporal features from gesture data.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [97] Nishida, N., Nakayama, H.: Multimodal gesture recognition using multi-stream recurrent neural network. In: Revised Selected Papers of the 7th Pacific-Rim Symposium on Image and Video Technology, pp. 682–694 (2015) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In recent years, deep learning algorithms have been developed to save time and dependability during patient care by improving clinical accuracy and detecting abnormalities in medical images [104, 140].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [104] Xu, Y.: Deep learning in multimodal medical image analysis. In: Health Information Science, pp. 193–200 (2019) source
- Not checked yetcites [140] Xi, P.: An integrated approach for medical abnormality detection using deep patch convolutional neural networks. Vis. Comput. 36, 1869–1882 (2020) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“Gatys et al. [156] demonstrated that visual features of models could be combined to represent image styles.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [156] Gatys, L., et al.: A Neural Algorithm of Artistic Style. arXiv:1508.06576 (2015) source
Strength:boosted· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Therefore, each modality will be covered individually using adaptive approaches. After joining the multi-domain features, the high-level classification will be performed by an individual classifier.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“In [48], the structure of deep multimodal Boltzmann machines (DMBM) [71] is similar to that of DBM, but it can admit more than one modality.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [48] Alam, M. R., Bennamoun, M., Togneri, R., Sohel, F.: A deep neural network for audio-visual person recognition. In: IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1–6 (2015) source
- Not checked yetcites [71] Srivastava, N., Salakhutdinov, R.: Multimodal learning with deep Boltzmann machines. J. Mach. Learn. Res. 15(1), 2949–2980 (2014)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Over the last few years, the fully convolutional network (FCN) [29] has become one of the robust models for a wide range of image types (multimedia, aerial, medical, etc.).”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Generic computer vision tasks
What the sources say
- Not checked yetcites [29] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. arXiv:1411.4038 (2015) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“According to Salakhutdinov et al. [72], a DBM is a generative model that includes several layers of hidden variables.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [72] Salakhutdinov, R., Hinton, G.: Deep Boltzmann machines. In: Artificial Intelligence and Statistics, pp. 448–455 (2009)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In [142], Lee et al. proposed a new CNN-based retinal image registration method to learn multimodal features simultaneously from several imaging modalities.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [142] Lee, J. A., et al.: Deep step pattern representation for multimodal retinal image registration. In: CVPR, pp. 5077–5086 (2019) source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In [98], the authors developed a multimodal gesture recognition system using the 3D Residual CNN (ResC3D) model [99] trained on an RGB-D dataset.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [98] Miao, Q. et al.: Multimodal gesture recognition based on the ResC3D network. In: IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 3047–3055 (2017) source
- Not checked yetcites [99] Tran, D., Ray, J., Shou, Z., Chang, S.-F., Paluri, M.: ConvNet architecture search for spatiotemporal feature learning. arXiv:1708.05038 (2017)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, capturing multimodal data streams under different acquisition conditions and increasing the data volume makes it easier to recognize visual content.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, it is important to note that each multimodal problem requires a specific fusion strategy, ranging from traditional methods to deep learning techniques.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Conclusion
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the need to reduce computing capacity remains the main challenge, which can lead to a deterioration in the overall accuracy of training algorithms.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, networks trained with a limited number of examples can no longer generalize well to a new dataset.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“For instance, the edge/cloud computing solution for mulitmodal analysis provides an effortless way to create and handle multimodal datasets for training and deploying models [107].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
What the sources say
- Not checked yetcites [107] Wang, X., et al.: Convergence of edge computing and deep learning: a comprehensive survey. IEEE Commun. Surv. Tutorials 1, 2020 (2020)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Deep learning models are often robust strategies for dealing with the linear and nonlinear combination of multimodal data.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The amount of multimodal data significantly affects the overall performance of the vision system.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, feature-level fusion is more flexible than decision-level fusion due to the homogeneity of data samples.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The deep multimodal analysis relies on a large amount of heterogeneous sensor data to achieve high performance and avoid overfitting problems.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Popular visual multimodal datasets
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In practice, autonomous vehicles, healthcare robots, and other real-time embedded systems consume more hardware resources, storage, and battery than other emerging technologies, resulting in a lack of adaptation to future needs.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, acquiring and processing a large amount of multimodal data is costly in terms of computation complexity and memory consumption.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, choosing the right fusion of different schemes remains a vital challenge for the computer vision community in terms of accuracy and efficiency.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Conclusion
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, AE can address missing patterns by generating intermediate shared representations from the input data and showing intra- and inter-pattern correlations.”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Discussion, limitations, and challenges
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Among the advantages of these types of sensors are their low computational cost, their long-range, their ability to have an internal mechanism to limit the impact of bad weather, etc. [149].”
§ A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets > Tasks and applications > Multimodal applications
What the sources say
- Not checked yetcites [149] Van Brummelen, J., et al.: Autonomous vehicle perception: the technology of today and tomorrow. Transp. Res. C Emerg. Technol. 89, 384–406 (2018) source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
A majority of contemporary machine learning applications use multiple modalities, such as visual and textual data.
“However, most modern machine learning applications involve more than one modality (e.g., visual and textual modalities), such as embodied question answering, vision-and-language navigation, etc.”
The paper relies on this without support.
The absence of labeled multimodal data in the literature can reduce flexibility and accuracy.
“Moreover, the lack of labeled multimodal data in the current literature can lead to reduced flexibility and accuracy, often requiring cooperation between different modalities.”
The paper relies on this without support.
Using multiple sensors as multimodal data sources yields complementary and richer information and can outperform a single input.
“The main reason for using multimodal data sources is that it is possible to extract complementary and richer information coming from multiple sensors, which can provide much more optimistic results than a single input.”
The paper relies on this without support.
Deep multimodal learning's success has been driven by large multimodal datasets, powerful GPU-equipped computers, and high-quality multi-scale feature representations.
“The success of deep multimodal learning techniques has been driven by many factors that have led many researchers to adopt these methods to improve model performance. These factors include large volumes of widely usable multimodal datasets, more powerful computers with fast GPUs, and high-quality feature representation at multiple scales.”
The paper relies on this without support.
Training a deep model from scratch requires a large amount of labeled data to achieve an acceptable level of performance.
“Typically, training a deep model from scratch requires a large amount of labeled data to achieve an acceptable level of performance.”
The paper relies on this without support.
The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.
“In practice, the amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image (i.e., seen and unseen classes).”
The paper relies on this without support.
In some vision applications, such as autonomous driving, security, and video surveillance, real-time conditions become necessary.
“However, in some vision applications, such as autonomous driving, security, video surveillance, etc., real-time conditions become necessary.”
The paper relies on this without support.
Most facial recognition systems, including biometric systems, are limited by a lack of universality and by variations in the appearance of visual patterns.
“However, most facial recognition systems, including biometric systems, suffer from a restriction in terms of universality and variations in the appearance of visual patterns.”
The paper relies on this without support.
Most current scene-content analysis methods cannot handle information that is not usable in real-life contexts.
“However, most current scene-content analysis methods are still limited in their ability to deal with information that is not usable in real-life contexts.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Developments in router Zloss regularization techniques plays a crucial role in addressing expert imbalance in MoE models by fine-tuning the gating mechanism, ensuring a more equitable workload distribution across experts and fostering a stable training environment, thereby enhancing model performance and reducing training time and computational overhead [301], [302].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [301]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Application-table continuation is labeled Table 3
MinorThe rows under "Table 3 continued" list applications, fusion schemes, and results, continuing the material identified as Table 2 rather than the dataset table described as Table 3.
“Table: Table 3 continued | [142] | 2019 | Medical data analysis | RGB + FA + OCT | Alignment | Color fundus-FA (ACC = 90.10%) |”
“Table: Table 2 Summary of the multimodal applications reviewed, their related technical details, and best results achieved”
CNN introduction dated both 1998 and 2012
MinorThe paper describes LeCun et al.'s convolutional neural networks in 1998, but later says the CNN model was introduced by Krizhevsky et al. in 2012.
“Since its introduction by Krizhevsky et al. [7] in 2012, the CNN model has been successfully applied”
“such as the convolutional neural networks of LeCun et al. [21] in 1998 and the spectacular deep network results of Krizhevsky et al. [7] in 2012”
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
(2016) · Cited by 31,590
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2015) · arXiv · doi:10.48550/arxiv.1512.02325 · Cited by 22,443
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Horn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Horn, B. K. P., & Schunck, B. G. (1981) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.965761 · Cited by 7,443
Cascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370
An Algorithm for Vector Quantizer Design
Linde, Y., Buzo, A., & Gray, R. (1980) · IEEE Transactions on Communications · doi:10.1109/tcom.1980.1094577 · Cited by 7,201
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras
Mur-Artal, R., & Tardos, J. D. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2705103 · Cited by 6,898
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
The Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 5,004
Generative Adversarial Networks: An Overview
Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A. A. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765202 · Cited by 4,540
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
LSD-SLAM: Large-scale direct monocular SLAM
Engel, J., Schöps, T., & Cremers, D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_54 · Cited by 3,851
An Overview of Multi-Task Learning in Deep Neural NetworksPreprint
Ruder, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05098 · Cited by 3,827
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2844175 · Cited by 3,457
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmák, S., & Shawe-Taylor, J. S. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,231
A survey of the recent architectures of deep convolutional neural networks
Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09825-6 · Cited by 3,137
Engel, J. J., Koltun, V., & Cremers, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2658577 · Cited by 2,995
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01247-4 · Cited by 2,965
Region-based convolutional networks for accurate object detection and segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2437384 · Cited by 2,912
Fast R-CNNPreprint
Girshick, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.08083 · Cited by 2,781
Multimodal Unsupervised Image-to-Image Translation
Huang, X., Liu, M.-Y., Belongie, S., & Kautz, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_11 · Cited by 2,710
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6229 · Cited by 2,680
A Neural Algorithm of Artistic Style
Gatys, L., Ecker, A., & Bethge, M. (2016) · Journal of Vision · doi:10.1167/16.12.326 · Cited by 2,612
Generative Adversarial Networks
Drori, I. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108891530.013 · Cited by 2,557
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504
Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.438 · Cited by 2,488
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,072
Cascade R-CNN: High Quality Object Detection and Instance Segmentation
Cai, Z., & Vasconcelos, N. M. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2956516 · Cited by 2,062
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809
Multiple Kernel Learning Algorithms
Gönen, M., & Alpaydın, E. (2011) · Cited by 1,651
Hybrid Task Cascade for Instance Segmentation
Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang, W., et al. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00511 · Cited by 1,587
Pouyanfar, S., Sadiq, S. B., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M., Chen, S., & Iyengar, S. S. (2018) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,510
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498 · cited 1×
“For instance, the edge/cloud computing solution for mulitmodal analysis provides an effortless way to create and handle multimodal datasets for training and deploying models [107].”
MethodA Survey of Deep Learning-Based Object Detection
Jiao, L., Zhang, F., Liu, F., Yang, S., Li, L., Feng, Z., & Qu, R. (2019) · IEEE Access · doi:10.1109/access.2019.2939201 · Cited by 1,454
Shi, F., Wang, J., Shi, J., Wu, Z., Wang, Q., Tang, Z., He, K., Shi, Y., & Shen, D. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.2987975 · Cited by 1,404
A large-scale hierarchical multi-view RGB-D object dataset
Lai, K., Bo, L., Ren, X., & Fox, D. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980382 · Cited by 1,353
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296
Mayer, N., Ilg, E., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · Cited by 1,214
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170
Factorial Hidden Markov Models
Ghahramani, Z., & Jordan, M. I. (1997) · Machine Learning · doi:10.1023/a:1007425814087 · Cited by 1,157
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
Feature Generating Networks for Zero-Shot Learning
Xian, Y., Lorenz, T., Schiele, B., & Akata, Z. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00581 · Cited by 1,080
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · doi:10.5555/2627435.2697059 · Cited by 1,070
Recent advances in deep learning for object detection
Wu, X., Sahoo, D., & Hoi, S. C. H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.085 · Cited by 1,047
Recent Advances in Deep Learning for Object DetectionPreprint
Wu, X., Sahoo, D., & Hoi, S. (2019) · Neurocomputing · doi:10.48550/arxiv.1908.03673 · Cited by 975
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972
Review on Methods to Fix Number of Hidden Neurons in Neural Networks
Sheela, K. G., & Deepa, S. N. (2013) · Mathematical Problems in Engineering · doi:10.1155/2013/425740 · Cited by 950
Neural Style Transfer: A Review
Jing, Y., Yang, Y., Feng, Z., Ye, J., Yu, Y., & Song, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2921336 · Cited by 938
HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition
Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., & Yu, Y. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.314 · Cited by 936
Transfer Learning for Visual Categorization: A Survey
Shao, L., Zhu, F., & Li, X. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2330900 · Cited by 895
A review of semantic segmentation using deep neural networks
Guo, Y., Liu, Y., Georgiou, T., & Lew, M. S. (2018) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-017-0141-z · Cited by 882
Autonomous vehicle perception: The technology of today and tomorrow
Van Brummelen, J., O’Brien, M., Gruyer, D., & Najjaran, H. (2018) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2018.02.012 · Cited by 869
An Attentive Survey of Attention Models
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3465055 · Cited by 860
Feature Pyramid Networks for Object DetectionPreprint
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03144 · Cited by 821
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.541 · Cited by 795
Deep learning in video multi-object tracking: A survey
Ciaparrone, G., Luque Sánchez, F., Tabik, S., Troiano, L., Tagliaferri, R., & Herrera, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.023 · Cited by 665
A Survey of Deep Learning: Platforms, Applications and Emerging Research Trends
Hatcher, W. G., & Yu, W. (2018) · IEEE Access · doi:10.1109/access.2018.2830661 · Cited by 654
Sangkloy, P., Burnell, N., Ham, C., & Hays, J. H. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925954 · Cited by 623
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535
A Survey of Zero-Shot Learning: Settings, Methods, and Applications
Wang, W., Zheng, V. W., Yu, H., & Miao, C. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3293318 · Cited by 528
Multi-Sensor Fusion in Automated Driving: A Survey
Wang, Z., Wu, Y., & Niu, Q. (2020) · IEEE Access · doi:10.1109/access.2019.2962554 · Cited by 526
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522
Overfitting and neural networks: conjugate gradient and backpropagation
Lawrence, S., & Giles, C. L. (2000) · Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium · doi:10.1109/ijcnn.2000.857823 · Cited by 491
Indoor scene segmentation using a structured light sensor
Silberman, N., & Fergus, R. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130298 · Cited by 485
Progressive Feature Alignment for Unsupervised Domain Adaptation
Chen, C., Xie, W., Huang, W., Rong, Y., Ding, X., Huang, Y., Xu, T., & Huang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00072 · Cited by 481
COVID-19 Detection Through Transfer Learning Using Multimodal Imaging Data
Horry, M. J., Chakraborty, S., Paul, M., Ulhaq, A., Pradhan, B., Saha, M., & Shukla, N. (2020) · IEEE Access · doi:10.1109/access.2020.3016780 · Cited by 472
Robust Face Recognition via Multimodal Deep Face Representation
Ding, C., & Tao, D. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477042 · Cited by 449
Independent Component Analysis
Stone, J. V. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/3717.001.0001 · Cited by 447
Composing Text and Image for Image Retrieval - an Empirical Odyssey
Vo, N., Jiang, L., Sun, C., Murphy, K. P., Li, L.-J., Fei-Fei, L., & Hays, J. H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00660 · Cited by 433
ConvNet Architecture Search for Spatiotemporal Feature LearningPreprint
Tran, D., Ray, J., Zheng, S.-F., Chang, S.-F., & Paluri, M. (2017) · arXiv.org · doi:10.48550/arxiv.1708.05038 · Cited by 421
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00008 · Cited by 410
Learning deep representations for ground-to-aerial geolocalization
Lin, T.-Y., Cui, Y., Belongie, S., & Hays, J. H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299135 · Cited by 409
Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network Parser
Duong, L., Cohn, T., Bird, S., & Cook, P. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.3115/v1/p15-2139 · Cited by 402
Attention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401
People detection in RGB-D Data
Spinello, L., & Arras, K. O. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6095074 · Cited by 367
Santosh, K. C. (2020) · Journal of Medical Systems · doi:10.1007/s10916-020-01562-1 · Cited by 341
MMTM: Multimodal Transfer Module for CNN Fusion
Joze, H. R. V., Shaban, A., Iuzzolino, M. L., & Koishida, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01330 · Cited by 337
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334
RGB-D salient object detection: A survey
Zhou, T., Fan, D.-P., Cheng, M.-M., Shen, J., & Shao, L. (2021) · Computational Visual Media · doi:10.1007/s41095-020-0199-z · Cited by 331
MUREL: Multimodal Relational Reasoning for Visual Question Answering
Cadene, R., Ben-younes, H., Cord, M., & Thome, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00209 · Cited by 330
Dargan, S., & Kumar, M. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.113114 · Cited by 329
BigBIRD: A large-scale 3D database of object instances
Singh, A., Sha, J., Narayan, K. S., Achim, T., & Abbeel, P. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6906903 · Cited by 321
Multimodal end-to-end autonomous driving
Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., & Lopez, A. M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3013234 · Cited by 319
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00279 · Cited by 306
Frank Rosenblatt: Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
Malsburg, C. V. D. (1986) · Springer eBooks · doi:10.1007/978-3-642-70911-1_20 · Cited by 305
Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds
Simon, M., Amende, K., Kraus, A., Honer, J., Samann, T., Kaulbersch, H., Milz, S., & Gross, H. M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00158 · Cited by 304
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., & Huang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00210 · Cited by 293
Multimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation
Pan, B., Cai, H., Huang, D.-A., Lee, K.-H., Gaidon, A., Adeli, E., & Niebles, J. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01088 · Cited by 281
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos
Shahroudy, A., Ng, T.-T., Gong, Y., & Wang, G. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2691321 · Cited by 269
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
Krantz, J., Wijmans, E., Majumdar, A., Batra, D., & Lee, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58604-1_7 · Cited by 266
Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Guo, L., Liu, J., Zhu, X., Yao, P., Lu, S., & Lu, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01034 · Cited by 261
Hierarchical Conditional Relation Networks for Video Question Answering
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00999 · Cited by 260
Multimodal Unsupervised Image-to-Image TranslationPreprint
Huang, X., Liu, M.-Y., Belongie, S., & Kautz, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.04732 · Cited by 259
Robust Multi-Modality Multi-Object Tracking
Zhang, W., Zhou, H., Sun, S., Wang, Z., Shi, J., & Loy, C. C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00245 · Cited by 250
Understanding RealWorld Indoor Scenes with Synthetic Data
Handa, A., Pătrăucean, V., Badrinarayanan, V., Stent, S., & Cipolla, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.442 · Cited by 248
Perceptron Simulation Experiments
Rosenblatt, F. F. (1960) · Proceedings of the IRE · doi:10.1109/jrproc.1960.287598 · Cited by 232
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228
Multimodal vehicle detection: fusing 3D-LIDAR and color camera data
Asvadi, A., Garrote, L., Premebida, C., Peixoto, P., & J. Nunes, U. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.09.038 · Cited by 225
A Survey on Canonical Correlation Analysis
Yang, X., Liu, W., Liu, W., & Tao, D. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2958342 · Cited by 209
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network
Wang, B., Ma, L., Zhang, W., Jiang, W., Wang, J., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00273 · Cited by 199
Pouyanfar, S., Yang, Y., Chen, S., Shyu, M., & Iyengar, S. S. (2018) · ACM Computing Surveys · doi:10.1145/3150226 · Cited by 195
Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training
Abavisani, M., Joze, H. R. V., & Patel, V. M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00126 · Cited by 190
EasyMKL: a scalable multiple kernel learning algorithm
Aiolli, F., & Donini, M. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.11.078 · Cited by 186
Wang, X., Oxholm, G., Zhang, D., & Wang, Y.-F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.759 · Cited by 182
A Survey of Semantic SegmentationPreprint
Thoma, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.06541 · Cited by 175
Hybrid Task Cascade for Instance SegmentationPreprint
Chen, K., Pang, J., Wang, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.07518 · Cited by 175
Attribute Attention for Semantic Disambiguation in Zero-Shot Learning
Liu, Y., Guo, J., Cai, D., & He, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00680 · Cited by 175
A Large Dataset of Object ScansPreprint
Choi, S., Zhou, Q.-Y., Miller, S., & Koltun, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02481 · Cited by 173
Multimodal Gesture Recognition Based on the ResC3D Network
Miao, Q., Li, Y., Ouyang, W., Ma, Z., Xu, X., Shi, W., & Cao, X. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.360 · Cited by 168
Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images
Tu, S., Pang, J., Liu, H., Zhuang, N., Chen, Y., Zheng, C., Wan, H., & Xue, Y. (2020) · Precision Agriculture · doi:10.1007/s11119-020-09709-3 · Cited by 168
Collect and Select: Semantic Alignment Metric Learning for Few-Shot Learning
Hao, F., He, F., Cheng, J., Wang, L., Cao, J., & Tao, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00855 · Cited by 167
No Blind Spots: Full-Surround Multi-Object Tracking for Autonomous Vehicles Using Cameras and LiDARs
Rangesh, A., & Trivedi, M. M. (2019) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2938110 · Cited by 158
DRCDN: learning deep residual convolutional dehazing networks
Zhang, S., & He, F. (2020) · The Visual Computer · doi:10.1007/s00371-019-01774-8 · Cited by 158
Multimodal Machine Learning: A Survey and TaxonomyPreprint
Baltrušaitis, T., Ahuja, C., & Morency, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09406 · Cited by 152
Rethinking Zero-Shot Learning: A Conditional Visual Classification Perspective
Li, K., Min, M. R., & Fu, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00368 · Cited by 144
Majority Vote of Diverse Classifiers for Late Fusion
Morvant, E., Habrard, A., & Ayache, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44415-3_16 · Cited by 138
Survey on AI-Based Multimodal Methods for Emotion Detection
Marechal, C., Mikołajewski, D., Tyburek, K., Prokopowicz, P., Bougueroua, L., Ancourt, C., & Węgrzyn-Wolska, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16272-6_11 · Cited by 137
Good News, Everyone! Context Driven Entity-Aware Captioning for News Images
Biten, A. F., Gomez, L., Rusiñol, M., & Karatzas, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01275 · Cited by 136
Chandar, S., Khapra, M. M., Larochelle, H., & Ravindran, B. (2015) · Neural Computation · doi:10.1162/neco_a_00801 · Cited by 133
A review of monocular visual odometry
He, M., Zhu, C., Huang, Q., Ren, B., & Liu, J. (2019) · The Visual Computer · doi:10.1007/s00371-019-01714-6 · Cited by 127
A study on Deep Neural Networks framework
Huang, Y., Shiyu, S., Duan, X., & Zhigang, C. (2016) · IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC) · doi:10.1109/imcec.2016.7867471 · Cited by 123
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNPreprint
Li, S., Li, W., Cook, C., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04831 · Cited by 123
Modality-correlation-aware sparse representation for RGB-infrared object tracking
Lan, X., Ye, M., Zhang, S., Zhou, H., & Yuen, P. C. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.10.002 · Cited by 121
AutoencodersPreprint
Bank, D., Koenigstein, N., & Giryes, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05991 · Cited by 115
Nguyen, K., Dey, D., Brockett, C., & Dolan, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01281 · Cited by 112
Towards Unsupervised Image Captioning With Shared Multimodal Embeddings
Laina, I., Rupprecht, C., & Navab, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00751 · Cited by 111
Face–Iris Multimodal Biometric Identification System
Ammour, B., Boubchir, L., Bouden, T., & Ramdani, M. (2020) · Electronics · doi:10.3390/electronics9010085 · Cited by 111
Bilbao, I., & Bilbao, J. (2017) · 2017 Eighth International Conference on Intelligent Computing and Information Systems (ICICIS) · doi:10.1109/intelcis.2017.8260032 · Cited by 109
Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00419 · Cited by 108
Facial Expression Recognition with Faster R-CNN
Li, J., Zhang, D., Zhang, J., et al. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.03.069 · Cited by 107
Neural Style Transfer: A ReviewPreprint
Jing, Y., Yang, Y., Feng, Z., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04058 · Cited by 102
Cascade R-CNN: High Quality Object Detection and Instance SegmentationPreprint
Cai, Z., & Vasconcelos, N. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09756 · Cited by 100
Adversarial Image Registration with Application for MR and TRUS Image Fusion
Yan, P., Xu, S., Rastinehad, A. R., & Wood, B. J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00919-9_23 · Cited by 100
Multimodal Style Transfer via Graph Cuts
Zhang, Y., Fang, C., Wang, Y., Wang, Z., Lin, Z., Fu, Y., & Yang, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00604 · Cited by 98
Multimodal video classification with stacked contractive autoencoders
Liu, Y., Xiao-qing, F., & Zhou, Z. (2015) · Signal Processing · doi:10.1016/j.sigpro.2015.01.001 · Cited by 96
Multi-Target Embodied Question Answering
Yu, L., Chen, X., Gkioxari, G., Bansal, M., Berg, T. L., & Batra, D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00647 · Cited by 95
Challenges in Multi-modal Gesture Recognition
Escalera, S., Athitsos, V., & Guyon, I. (2017) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-57021-1_1 · Cited by 91
A Convolutional Neural Network Hand Tracker
Nowlan, S. J., & Platt, J. C. (1994) · Cited by 90
A survey on deep neural network-based image captioning
Liu, X., Xu, Q., & Wang, N. (2018) · The Visual Computer · doi:10.1007/s00371-018-1566-y · Cited by 89
Bayoudh, K., Hamdaoui, F., & Mtibaa, A. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01801-5 · Cited by 89
Self-citationThe Infinite Factorial Hidden Markov Model
Gael, J. V., Teh, Y. W., & Ghahramani, Z. (2008) · UCL Discovery (University College London) · Cited by 88
SceneNet: Understanding Real World Indoor Scenes With Synthetic DataPreprint
Handa, A., Pătrăucean, V., Badrinarayanan, V., Stent, S., & Cipolla, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07041 · Cited by 87
Joint learning of image detail and transmission map for single image dehazing
Zhang, S., He, F., Ren, W., & Yao, J. (2020) · The Visual Computer · doi:10.1007/s00371-018-1612-9 · Cited by 87
Multimodal Deep Learning Approach for Joint EEG-EMG Data Compression and Classification
Ben Said, A., Mohamed, A., Elfouly, T., Harras, K., & Wang, Z. J. (2017) · 2017 IEEE Wireless Communications and Networking Conference (WCNC) · doi:10.1109/wcnc.2017.7925709 · Cited by 86
Unsupervised Deep Visual-Inertial Odometry with Online Error Correction for RGB-D Imagery
Shamwell, E. J., Lindgren, K., Leung, S., & Nothwang, W. D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2909895 · Cited by 86
Efficient Low-rank Multimodal Fusion with Modality-Specific FactorsPreprint
Liu, Z., Shen, Y., Lakshminarasimhan, V., Liang, P. P., Zadeh, A., & Morency, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00064 · Cited by 80
Exploring RGB+Depth Fusion for Real-Time Object Detection
Ophoff, T., Van Beeck, K., & Goedemé, T. (2019) · Sensors · doi:10.3390/s19040866 · Cited by 77
Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks
Sano, A., Chen, W. '., Lopez-Martinez, D., Taylor, S., & Picard, R. W. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2867619 · Cited by 73
Semantic Alignment: Finding Semantically Consistent Ground-Truth for Facial Landmark Detection
Liu, Z., Zhu, X., Hu, G., Guo, H., Tang, M., Lei, Z., Robertson, N. M., & Wang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00358 · Cited by 73
Direct Sparse OdometryPreprint
Engel, J. J., Koltun, V., & Cremers, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02565 · Cited by 70
3D-SSD: Learning hierarchical features from RGB-D images for amodal 3D object detection
Luo, Q., Ma, H., Tang, L., Wang, Y., & Xiong, R. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.10.025 · Cited by 69
A survey on online learning for visual tracking
Abbass, M. Y., Kwon, K.-C., Kim, N., Abdelwahab, S. A., El-Samie, F. E. A., & Khalaf, A. A. M. (2021) · The Visual Computer · doi:10.1007/s00371-020-01848-y · Cited by 64
RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey
Gao, M., Jiang, J., Zou, G., John, V., & Liu, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2907071 · Cited by 63
A multimodal fusion approach for image captioning
Zhao, D., Chang, Z., & Guo, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.004 · Cited by 61
Feature Alignment and Restoration for Domain Generalization and AdaptationPreprint
Jin, X., Lan, C., Zeng, W., & Chen, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12009 · Cited by 61
Enhanced situation awareness through CNN-based deep multimodal image fusion
Liu, S., Liu, H., John, V., Liu, Z., & Blasch, E. (2020) · Optical Engineering · doi:10.1117/1.oe.59.5.053103 · Cited by 60
Mahmud, T., Sazzad Sayyed, A. Q. M., Fattah, S. A., & Kung, S.-Y. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3015781 · Cited by 58
A Deep Step Pattern Representation for Multimodal Retinal Image Registration
Lee, J., Liu, P., Cheng, J., & Fu, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00518 · Cited by 56
Multi-Modal Stacked Denoising Autoencoder for Handling Missing Data in Healthcare Big Data
Kim, J.-C., & Chung, K. (2020) · IEEE Access · doi:10.1109/access.2020.2997255 · Cited by 56
Multimodal Gesture Recognition Using Multi-stream Recurrent Neural Network
Nishida, N., & Nakayama, H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-29451-3_54 · Cited by 53
A Review of Point Feature Based Medical Image Registration
Guan, S., Wang, T.-M., Meng, C., & Wang, J.-C. (2018) · Chinese Journal of Mechanical Engineering · doi:10.1186/s10033-018-0275-9 · Cited by 53
MultiXNet: Multiclass Multistage Multimodal Motion Prediction
Djuric, N., Cui, H., Su, Z., Wu, S., Wang, H., Chou, F.-C., Martin, L. S., Feng, S., Hu, R., Xu, Y., et al. (2021) · 2021 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/iv48863.2021.9575718 · Cited by 53
Classification-aware Semi-supervised Domain Adaptation
He, G., Liu, X., Fan, F., & You, J. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00490 · Cited by 53
Multi-modal multiple kernel learning for accurate identification of Tourette syndrome children
Wen, H., Liu, Y., Rekik, I., Wang, S., Chen, Z., Zhang, J., Zhang, Y., Peng, Y., & He, H. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.09.039 · Cited by 52
Audio Caption: Listen and Tell
Wu, M., Dinkel, H., & Yu, K. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682377 · Cited by 52
ASoVS: Abstractive Summarization of Video Sequences
Dilawari, A., & Khan, M. U. G. (2019) · IEEE Access · doi:10.1109/access.2019.2902507 · Cited by 51
Diagnosing the Environment Bias in Vision-and-Language Navigation
Zhang, Y., Tan, H., & Bansal, M. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/124 · Cited by 51
Flagged first, then by citations
Research works printed but not linked: 6
- Bayoudh, K.: From machine learning to deep learning, (1st ed.), Ebook, ISBN: 9781387465606 (2017)
- Wang, M., Deng, W.: Deep Face Recognition: A Survey. arXiv:1804.06655 (2019) · arXiv
- Salakhutdinov, R., Hinton, G.: Deep Boltzmann machines. In: Artificial Intelligence and Statistics, pp. 448–455 (2009)
- Escalera, S., Athitsos, V., Guyon, I.: Challenges in multimodal gesture recognition. J. Mach. Learn. Res. 17, 1–54 (2016)
- Yue, L., et al.: Attentional alignment network. In: BMVC (2018)
- Srivastava, N., et al.: Multimodal learning with deep Boltzmann machines. J. Mach. Learn. Res. 15, 2949–2980 (2014)
Cited by
Sorted by citations
Effective Techniques for Multimodal Data Fusion: A Comparative Analysis
Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S. (2023) · Sensors · doi:10.3390/s23052381 · Cited by 180
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“…the field of multimodal learning still confronts ongoing challenges, such as refining the architectures to handle diverse data types more effectively [269], [270], developing comprehensive datasets that accurately represent multifaceted information [269], [271], and establishing benchmarks for…”
BackgroundA systematic review of intermediate fusion in multimodal deep learning for biomedical applications
Guarrasi, V., Aksu, F., Caruso, C. M., Di Feola, F., Rofena, A., Ruffini, F., & Soda, P. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105509 · Cited by 95
Detection in adverse weather conditions for autonomous vehicles via deep learning
Al-Haija, Q. A., Gharaibeh, M., & Odeh, A. (2022) · AI · doi:10.3390/ai3020019 · Cited by 81
A Systematic Review of Adversarial Machine Learning Attacks, Defensive Controls and Technologies
Malik, J., Muthalagu, R., & Pawar, P. M. (2024) · IEEE Access · doi:10.1109/access.2024.3423323 · Cited by 76
Metadata
- Authors
- Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A.
- Affiliations
- University of Monastir (TN)
- Corresponding author
- Bayoudh, Khaled
- Venue
- The Visual Computer
- Published
- 2021-06-10
- Volume
- 38
- Issue
- 8
- Pages
- 2939-2970
- ISSN
- 0178-2789 (ISSN-L)
- 1432-2315
- PMC
- PMC8192112
- OpenAlex
- W3168463823
- Semantic Scholar
- 2a9b33f66ccc3806af58bdab2319559f4f9d2c5e
- Cited by
- 456
- References
- 280
- Citations by year
- 2026: 70 · 2025: 148 · 2024: 119 · 2023: 70 · 2022: 42 · 2021: 7
- FWCI
- 21.32
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- The Visual Computer (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation, multi-task learning, multimodal alignment, zero-shot learning, multimodal dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
