A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Textual representation (word embedding) is much sparser than image representation, which makes combining them into a unified model difficult.

“As shown in Fig. 3, the textual representation (i.e., a word embedding) is very sparse when compared to the image one, which makes it very challenging to combine these two different representations into a unified model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A bimodal autoencoder couples two separate autoencoders for audio and video into a jointly shared representation hierarchy with independent encoder and decoder components.

“Figure 6 shows the coupling mechanism of two separate AEs (bimodal AE) for both modalities (audio and video) into a jointly shared representation hierarchy where the encoder and decoder components are independent of each other.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Denoising autoencoders aim to learn efficient representations of manifolds when training data is unbalanced or lacking.

“As a powerful tool for feature extraction and dimensionality reduction, the DAE aims to learn how to efficiently represent manifolds where the training data is unbalanced or lacking.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Denoising autoencoders miss some spatiotemporal details in multimodal data.

“Moreover, they also miss some spatiotemporal details in multimodal data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Guo et al. [35] gave a comprehensive overview of deep multimodal learning frameworks and models, with a focus on multimodal representation.

“Guo et al. [35] provided a comprehensive overview of deep multimodal learning frameworks and models, focusing on one of the main challenges of multimodal learning, namely multimodal representation.”

• cites Guo et al. [35] Guo, W., Wang, J., Wang, S.: Deep multimodal representation learning: a survey. IEEE Access 7, 63373–63394 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Baltrušaitis et al. [37] identified five levels of multimodal data combination: representation, translation, alignment, fusion, and co-learning.

“Here, the authors identified five levels of multimodal data combination: representation, translation, alignment, fusion, and co-learning.”

• cites Baltrušaitis et al. [37] Baltrušaitis, T., Ahuja, C., Morency, L.-P.: Multimodal Machine Learning: A Survey and Taxonomy. arXiv:1705.09406 (2017)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Denoising autoencoders have the drawback that many hidden parameters must be trained and inference is time-consuming.

“One of the main drawbacks of DAEs is that many hidden parameters have to be trained, and the inference process is time-consuming.”

May combine two claims.

Stacked-autoencoder-based multimodal systems compress each modality separately in the encoder and build a shared latent representation in the decoder without supervision.

“the authors of [210–212] designed multimodal systems based on SAEs, where the encoder side of the architecture represents and compresses each unimodal feature separately, and the decoder side constructs the latent (shared) representation of the inputs in a unsupervised manner.”

• cites [210] Kim, J., Chung, K.: Multi-modal stacked denoising autoencoder for handling missing data in healthcare big data. IEEE Access 8, 104933–104943 (2020) sourceNot checked yet

  Not checked in the free run.

• cites [211] Singh, V., et al.: Feature learning using stacked autoencoder for shared and multimodal fusion of medical images. In: Computational Intelligence: Theories, Applications and Future Directions, pp. 53–66 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [212] Said, A. B., et al.: Multimodal deep learning approach for joint EEG-EMG data compression and classification. In: IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6 (2017) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Most papers reviewed in this survey are recent and appeared in high-quality conferences and journals such as the Visual Computer, ICCV, and CVPR.

“Moreover, most of the papers we reviewed are recent and have been published in high-quality conferences and journals such as the visual computer, ICCV, and CVPR.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Research on deep multimodal learning has expanded quickly over the past decade, particularly in computer vision.

“The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The attention mechanism acts as a contextual bridge between the encoding and decoding sides of a recurrent neural network to pay attention only to a particular position in the input representation.

“Therefore, the AM acts as a contextual bridge between the encoding and decoding sides of an RNN to pay attention only to a particular position in the input representation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The idea of the attention mechanism is to focus on a particular position in the input image by computing the weighted sum of feature vectors and mapping them into a final contextual representation.

“The idea of AM is to focus on a particular position in the input image by computing the weighted sum of feature vectors and mapping them into a final contextual representation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A generative adversarial network is a unified network consisting of a generator and a discriminator that interact continuously during learning.

“Structurally, GAN is a unified network consisting of two sub-networks, a generator network (G) and a discriminator network (D), which interact continuously during the learning process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sharing the structure between all tasks can be learned jointly to improve performance.

“What they have in common is that the sharing of the structure between all tasks can be learned jointly to improve performance.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The only weakness of the attention mechanism is that the training algorithm is unstable, which may affect the predictive power of the decision-making system.

“However, the only weakness of AM is that the training algorithm is unstable, which may affect the predictive power of the decision-making system.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multitask learning has become an increasingly popular topic in the deep learning community.

“More recently, multitask learning (MTL) [108, 109] has become an increasingly popular topic in the deep learning community.”

• cites [108] Ruder, S.: An Overview of Multi-Task Learning in Deep Neural Networks. arXiv:1706.05098 (2017)Not checked yet

  Not checked in the free run.

• cites [109] Ruder, S., Bingel, J., Augenstein, I., Søgaard, A.: Latent Multi-task Architecture Learning. arXiv:1705.08142 (2018) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A multimodal convolutional neural network's inference process is time-consuming.

“However, it requires a large-scale multimodal dataset to converge optimally during training, and the inference process is time-consuming.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The attention mechanism learns how to reduce some irrelevant attributes from a set of feature vectors.

“In other words, it learns how to reduce some irrelevant attributes from a set of feature vectors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multitask learning contributes to the performance of the target task based on knowledge gained from auxiliary tasks.

“Typically, MTL contributes to the performance of the target task based on knowledge gained from auxiliary tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Generative adversarial networks are deep generative architectures designed to learn the data distribution through adversarial learning.

“Generative adversarial networks (GANs) are part of deep generative architectures, designed to learn the data distribution through the adversarial learning.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A multimodal convolutional neural network requires a large-scale multimodal dataset to converge optimally during training.

“However, it requires a large-scale multimodal dataset to converge optimally during training, and the inference process is time-consuming.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The function that a generative adversarial network seeks to optimize is an adversarial loss function without any normalization.

“Moreover, the function that this network seeks to optimize is an adversarial loss function without any normalization.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

It is extremely tricky to train a generative adversarial network when generating discrete distributions, and the process is unstable compared to other generative networks.

“However, it is extremely tricky to train the network when generating discrete distributions, and the process itself is unstable compared to other generative networks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The multimodal recurrent neural network model has a slow convergence rate due to high computational cost in the hidden state transfer function.

“However, the model has a slow convergence rate due to the high computational cost in the hidden state transfer function.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hard parameter sharing extracts a generic representation for different tasks using the same parameters and is usually applied to avoid overfitting.

“Hard parameter sharing [110]: It consists of extracting a generic representation for different tasks using the same parameters. It is usually applied to avoid overfitting problems.”

• cites [110] Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The multimodal attention mechanism provides a robust solution for cross-modal data fusion by selecting local fine-grained salient features in a multidimensional space and filtering out hidden noise.

“In summary, the multimodal AM provides a robust solution for cross-modal data fusion by selecting the local fine-grained salient features in a multidimensional space and filtering out any hidden noise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The alignment task can be divided into attentional alignment and semantic alignment.

“Generally, the alignment task can be divided into two subtasks: the attentional alignment task [170, 171] and the semantic alignment task [172, 173].”

• cites [170] Dapogny, A., et al.: Deep Entwined Learning Head Pose and Face Alignment Inside an Attentional Cascade with Doubly-Conditional fusion. arXiv:2004.06558 (2020) sourceNot checked yet

  Not checked in the free run.

• cites [171] Yue, L., et al.: Attentional alignment network. In: BMVC (2018)Not checked yet

  Not checked in the free run.

• cites [172] Liu, Z., et al.: Semantic Alignment: Finding Semantically Consistent Ground-truth for Facial Landmark Detection. arXiv:1903.10661 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [173] Hao, F., et al.: Collect and select: semantic alignment metric learning for few-shot learning. In: CVPR, pp. 8460–8469 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Convolutional neural networks model nonlinear transformations by generating abstract and informative features from highly complex datasets.

“More specifically, such models tend to model a series of nonlinear transformations by generating very abstract and informative features from highly complex datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Gated recurrent linear units are a lightweight variant of long short-term memory because they can modulate information flow without using internal memory units.

“In terms of computational efficiency, GRU is a lightweight variant of LSTM since it can modulate the information flow without using its internal memory units.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In multimodal analysis, an attentional model can combine multiple modalities, each with its internal representation.

“In the multimodal analysis, an attentional model can be designed to combine multiple modalities, each with its internal representation (e.g., spatial features, motion features, etc.).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multimodal alignment consists of linearly linking the features of two or more different modalities.

“Multimodal alignment consists of linearly linking the features of two or more different modalities.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Patch-based methods first split each image into local patches and then consider the similarity between them by computing their cross-correlation and combination.

“Patch-based methods first split each image into local patches and then consider the similarity between them by computing their cross-correlation and combination.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recurrent neural networks are popular deep neural network architectures for processing sequential data of varying lengths.

“Recurrent neural networks (RNNs) [12] are a popular type of deep neural network architectures for processing sequential data of varying lengths.”

• cites [12] Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The attention mechanism has become one of the most challenging tasks in computer vision and machine translation.

“In recent years, the attention mechanism (AM) has become one of the most challenging tasks in computer vision and machine translation [232].”

• cites [232] Chaudhari, S., et al.: An Attentive Survey of Attention Models. arXiv:1904.02874 (2020)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The vanilla recurrent neural network is typically incapable of capturing long-term dependencies in sequential data because it has no internal memory.

“However, the vanilla RNN model is typically incapable of capturing long-term dependencies in sequential data since they have no internal memory.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Convolutional neural networks have demonstrated powerful generalization capabilities for learning visual appearance features from static data.

“In particular, CNNs have demonstrated powerful generalization capabilities to learn how to represent visual appearance features from static data.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Unsupervised generative adversarial networks are one of the most powerful generative models that can address scenarios where training data is lacking or hidden concepts are missing.

“In summary, unsupervised GAN is one of the most powerful generative models that can address scenarios where training data is lacking or some hidden concepts are missing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Since 2014, generative adversarial training algorithms have been widely used in unimodal applications such as scene generation, image-to-image translation, and image super-resolution.

“Since their development in 2014, generative adversarial training algorithms have been widely used in various unimodal applications such as scene generation [17], image-to-image translation [18], and image super-resolution [224, 225].”

• cites [17] Turkoglu, M.O., Thong, W., Spreeuwers, L., Kicanaoglu, B.: A layer-based sequential framework for scene generation with GANs. arXiv:1902.00671 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [18] Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv:1611.07004 (2018) sourceNot checked yet

  Not checked in the free run.

• cites [224] Song, H., et al.: $${\rm S}^2{{\rm RGAN}}$$S: sonar-image super-resolution based on generative adversarial network. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01986-3 sourceNot checked yet

  Not checked in the free run.

• cites [225] Ma, T., Tian, W.: Back-projection-based progressive growing generative adversarial network for single image super-resolution. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01843-3 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Recurrent neural networks can handle time series and dynamic media such as text and video sequences.

“It also can handle time series and dynamic media such as text and video sequences.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The multitask learning paradigm frequently arises in a context close to multimodal concepts.

“Specifically, the MTL paradigm frequently arises in a context close to multimodal concepts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A multimodal convolutional neural network can model spatial cues from multimodal data streams with an increased number of parameters.

“It is also capable of modeling spatial cues from multimodal data streams with an increased number of parameters.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Feature-based methods detect and extract a set of matching features that should be structurally consistent to describe their spatial patterns.

“Feature-based methods detect and extract a set of matching features that should be structurally consistent to describe their spatial patterns.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Recurrent neural networks learn to map input activations to the next hierarchy level and transfer hidden states to outputs using recurrent feedback, giving them capacity to learn useful features from previous states unlike feedforward networks.

“They learn to map input activations to the next hierarchy level and then transfer hidden states to the outputs using the recurrent feedback, which gives them the capacity to learn useful features from the previous states, unlike other deep feedforward networks such as CNNs, DBNs, etc.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The number of parameters to be trained in the attention mechanism is huge compared to other deep networks such as recurrent neural networks and convolutional neural networks.

“Furthermore, the number of parameters to be trained is huge compared to other deep networks such as RNNs, CNNs, etc.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multimodal image alignment provides spatial mapping between images taken by sensors of different modalities and can be categorized into feature-based and patch-based methods.

“Specifically, multimodal image alignment provides a spatial mapping capability between images taken by sensors of different modalities, which may be categorized into feature-based [167, 168] and patch-based [165, 166] methods.”

• cites [167] Chen, C., et al. Progressive Feature Alignment for Unsupervised Domain Adaptation. arXiv:1811.08585 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [168] Jin, X., et al.: Feature Alignment and Restoration for Domain Generalization and Adaptation. arXiv:2006.12009 (2020)Not checked yet

  Not checked in the free run.

• cites [165] Ščupáková, K., et al.: A patch-based super resolution algorithm for improving image resolution in clinical mass spectrometry. Sci. Rep. 9, 2915 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [166] Bashiri, F.S., et al.: Multi-modal medical image registration with full or partial data: a manifold learning approach. J. Imag. 5, 5 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Convolutional neural networks are distinguished from other models by local connectivity between units, weight sharing across layers, and blocking a sequence of hidden layers.

“The main properties that distinguish CNNs from other models include their ability to capture local connectivity between units, to share weights across layers, and to block a sequence of hidden layers [4].”

• cites [4] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Long short-term memory and gated recurrent linear units were developed to handle the long-term dependency constraint and gradient vanishing problem with impressive results.

“To this end, several popular variants have been developed to efficiently handle this constraint and the gradient vanishing problem with impressive results, including long short-term memory (LSTM) [13] and gated recurrent linear units (GRU) [14].”

• cites [13] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997) sourceNot checked yet

  Not checked in the free run.

• cites [14] Li, S., Li, W., Cook, C., Zhu, C., Gao, Y.: Independently recurrent neural network (IndRNN): building a longer and deeper RNN. arXiv:1803.04831 (2018) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

With the advent of robust low-cost RGB-D sensors, the computer vision community turned its attention to integrating RGB images and depth maps into multimodal architectures.

“Recently, with the advent of robust and low-cost RGB-D sensors such as the Kinect, the computer vision community has turned its attention to integrating RGB images and corresponding depth maps (2.5D) into multimodal architectures as shown in Fig. 7.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Convolutional neural networks are deep feed-forward networks whose main purpose is to extract spatial patterns from visual input signals.

“Convolutional neural networks (CNNs or ConvNets) are a class of deep feed-forward neural networks whose main purpose is to extract spatial patterns from visual input signals [20, 22].”

• cites [20] Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artif Intell Rev (2020) sourceNot checked yet

  Not checked in the free run.

• cites [22] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2015) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A multimodal convolutional neural network serves as a powerful feature extractor that learns local cross-modal features from visual modalities.

“In summary, a multimodal CNN serves as a powerful feature extractor that learns local cross-modal features from visual modalities.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The idea behind multitask learning is to learn a shared representation that can be used to respond to several tasks in order to ensure better generalizability.

“In contrast to single-task learning, the idea behind this paradigm is to learn a shared representation that can be used to respond to several tasks in order to ensure better generalizability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Bahdanau et al. developed a formalism for integrating an attention mechanism into the basic recurrent neural network model.

“A formalism for integrating an AM into the basic RNN model was developed by Bahdanau et al. [1].”

• cites [1] Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 (2016)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Soft parameter sharing extracts a set of feature vectors and simultaneously draws similarity relationships between them.

“Soft parameter sharing [111]: It consists of extracting a set of feature vectors and simultaneously drawing similarity relationships between them.”

• cites [111] Duong, L., Cohn, T., Bird, S., Cook, P. low resource dependency parsing: cross-lingual parameter sharing in a neural network parser. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 845–850 (2015) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The encoding side of a recurrent neural network generates a fixed-length feature vector from its input sequence, which can lead to tedious and time-consuming parameter tuning.

“Since the encoding side of an RNN generates a fixed-length feature vector from its input sequence, this can lead to very tedious and time-consuming parameter tuning.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Generative adversarial networks were first developed by Goodfellow et al., demonstrating the ability to generate realistic representations from noisy data domains.

“Historically, they were first developed by Goodfellow et al. [16], which demonstrated the ability to generate realistic and reasonably impractical representations from noisy data domains.”

• cites [16] Goodfellow, I.J., et al.: Generative adversarial networks. arXiv:1406.2661 (2014)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The visual tracking task has attracted interest because of its relevance in applications including video surveillance and autonomous driving.

“The task has attracted a lot of interest because of its enormous relevance in many real-world applications, including video surveillance [82], autonomous driving [83], etc.”

• cites [82] 1.Mohanapriya, D., Mahesh, K.: Chapter 5—an efficient framework for object tracking in video surveillance. In: The Cognitive Approach in Cloud Computing and Internet of Things Technologies for Surveillance Tracking Systems, pp. 65–74 (2020) sourceNot checked yet

  Not checked in the free run.

• cites [83] Rangesh, A., Trivedi, M.M.: No blind spots: full-surround multi-object tracking for autonomous vehicles using cameras and LiDARs. IEEE Trans. Intelli. Veh. 4, 588–599 (2019) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Transfer learning is a model regularization technique proven effective for training deep models with limited data and avoiding overfitting.

“Transfer learning (TL) [70] is one of the model regularization techniques that have proven their effectiveness for training deep models with a limited amount of available data and avoiding overfitting problems.”

• cites [70] Shao, L., Zhu, F., Li, X.: Transfer learning for visual categorization: a survey. IEEE Trans. Neural Netw. Learn. Syst. 26, 1019–1034 (2015) sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: Training a deep model from scratch requires a large amount of labeled data to achieve an acceptable level of performance.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

High-precision object recognition systems with multiple sensors are aware of external noise and environmental sensitivity.

“High-precision object recognition systems with multiple sensors are aware of external noise and environmental sensitivity (e.g., lighting variations, occlusion, etc.).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Focal loss allows training a one-stage detector with high accuracy by reducing the level of artifacts.

“This function allows training a one-stage detector with high accuracy by reducing the level of artifacts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cascade R-CNN is a cascaded version of R-CNN aimed at achieving a better compromise between object localization and classification.

“It is a cascaded version of R-CNN aimed at achieving a better compromise between object localization and classification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The objective of zero-shot learning is to project a set of synthesized features in order to make the model more generalizable toward recognition of unseen classes in test samples.

“The objective is to project a set of synthesized features in order to make the model more generalizable toward the recognition of the unseen class in test samples [135].”

• cites [135] Liu, Y., Tuytelaars, T.: A: deep multi-modal explanation model for zero-shot learning. IEEE Trans. Image Process. 29, 4788–4803 (2020) sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.

Probably not a substantial claim: may not pass S3 (load-bearing).

The deep learning community has pioneered a new generation of CNN-based frameworks to significantly reduce the complexity of object detection.

“In order to significantly reduce this complexity, the deep learning community has pioneered a new generation of CNN-based frameworks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to their complexity, sliding-window-based solutions are not cost-effective in terms of training duration and memory consumption.

“However, due to their complexity, this type of solution is not cost-effective, both in terms of training duration and memory consumption.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multi-object tracking methods are mainly designed to optimize the dynamic matching of the objects of interest detected in each frame.

“However, MOT methods are mainly designed to optimize the dynamic matching of the objects of interest detected in each frame.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Two-stage detectors are generally based on a combination of a CNN model for classification and a patch proposal module to generate candidate regions like RPNs.

“As mentioned before, two-stage detectors are generally based on a combination of a CNN model to perform classification and a patch proposal module to generate candidate regions like RPNs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multi-object tracking has been extensively explored in recent literature for a wide range of applications.

“Multi-object tracking (MOT) has been extensively explored in recent literature for a wide range of applications [86, 138].”

• cites [86] Ciaparrone, G., et al.: Deep learning in video multi-object tracking: a survey. Neurocomputing 381, 61–88 (2020) sourceNot checked yet

  Not checked in the free run.

• cites [138] Zhang, X., et al.: Online multi-object tracking with pedestrian re-identification and occlusion processing. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01854-0 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Semantic alignment directly provides alignment capabilities to a predictive model.

“The attentional alignment task is based on the attentional mapping between the features of the input modality and the target one, while the semantic alignment task takes the form of an alignment method that directly provides alignment capabilities to a predictive model.”

May combine two claims.

SSD reduced the needs of the patches' proposal network and accelerated the object detection pipeline.

“In contrast, SSD (single-shot multibox detector) [78] has reduced the needs of the patches’ proposal network and, thus, accelerated the object detection pipeline.”

• cites [78] Liu, W. et al.: SSD: single shot multibox detector, pp. 21–37. arXiv:1512.02325 (2016) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Zero-shot learning addresses multi-class learning problems when some classes do not have sufficient training data.

“In other words, it addresses multi-class learning problems when some classes do not have sufficient training data.”

• AssumptionRests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

When modeling multimodal data, several compromises have to be made between system performance, computational burden, and processing speed.

“When modeling multimodal data, several compromises have to be made between system performance, computational burden, and processing speed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multimodal multi-object tracking is a universal aspect of multi-object tracking aimed at ensuring the accuracy of autonomous systems by mapping the motion sequence of dynamic objects.

“Multimodal MOT is a universal aspect of MOT aimed at ensuring the accuracy of autonomous systems by mapping the motion sequence of dynamic objects [194].”

• cites [194] Zhang, W., et al.: Robust Multi-Modality Multi-Object Tracking. arXiv:1909.03850 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the context of multimodality, a multimodal mapping scheme typically combines visual and semantic attributes using only data related to the seen classes.

“In the context of multimodality, a multimodal mapping scheme typically combines visual and semantic attributes using only data related to the seen classes.”

• AssumptionRests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

These techniques have proven effective for accurate detection of multiple objects under normal and extreme environmental conditions.

“These techniques have proven effective for the accurate detection of multiple objects under normal and extreme environmental conditions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent literature has shown that it is attractive to learn shared representations from the complementarity and synergies between several modalities for increasing the discriminatory power of models.

“The recent literature has shown that it is attractive to learn shared representations from the complementarity and synergies between several modalities for increasing the discriminatory power of models [190].”

• cites [190] Michael, Y.B., Rosenhahn, V.M.: Multimodal Scene Understanding, 1st edn. Academic Press, London (2019)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Two-stage detectors are generally slow in terms of real-time processing.

“In this respect, two-stage detectors are generally slow in terms of real-time processing.”

• AssumptionRests on an assumption: In some vision applications, such as autonomous driving, security, and video surveillance, real-time conditions become necessary.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The most typical detection method is to apply the classifier to an arbitrary region of the image or to a range of different shapes and scales.

“In practice, there are several possible detection methods. The most typical of these is to apply the classifier to an arbitrary region of the image or to a range of different shapes and scales.”

Probably not a substantial claim: may not pass S3 (load-bearing).

RetinaNet takes only one stage to detect dense objects by producing multi-scale semantic feature maps using a feature pyramid network backbone and the ResNet model.

“As similar to the principle of YOLO and SSD detectors, RetinaNet [79] takes only one stage to detect dense objects by producing multi-scale semantic feature maps using a feature pyramid network (FPN) backbone and the ResNet model.”

• cites [79] Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv:1708.02002 (2018) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Transfer learning can learn the data distribution in a non-parallel manner and ensure its synchronization over time.

“Moreover, the learning process can learn the data distribution in a non-parallel manner and ensure its synchronization over time.”

• AssumptionRests on an assumption: Training a deep model from scratch requires a large amount of labeled data to achieve an acceptable level of performance.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Zero-shot learning is a supervised learning approach that opens applications such as object detection, object classification and retrieval of videos, especially when appropriate datasets are missing.

“This is why zero-shot learning [130] takes place. This supervised learning approach opens up many valuable applications such as object detection [131], object classification and retrieval of videos [141], especially when appropriate datasets are missing.”

• cites [130] Wang, W., et al.: A survey of zero-shot learning: settings, methods, and applications. ACM Trans. Intell. Syst. Technol. 10, 13:1–13:37 (2019)Not checked yet

  Not checked in the free run.

• cites [131] Wei, L., et al.: A single-shot multi-level feature reused neural network for object detection. Vis. Comput. (2020). https://doi.org/10.1007/s00371-019-01787-3 sourceNot checked yet

  Not checked in the free run.

• cites [141] Parida, K., et al.: Coordinated joint multimodal embeddings for generalized audio-visual zero-shot classification and retrieval of videos. In: CVPR, pp. 3251–3260 (2020) sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.

May combine two claims.

Cascade R-CNN is one of the most effective multi-stage detectors and has proven robust over one- and two-stage methods.

“Cascade R-CNN [26] is one of the most effective multi-stage detectors that have proven their robustness over one and two-stage methods.”

• cites [26] Cai, Z., Vasconcelos, N.: Cascade R-CNN: high quality object detection and instance segmentation. arXiv:1906.09756 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multimodality has been employed in many vision tasks and applications, such as face recognition and image retrieval.

“In general, multimodality has been employed in many vision tasks and applications, such as face recognition and image retrieval.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The availability of low-cost and robust sensors has encouraged the computer vision community to focus on combining the RGB modality with other sensing modalities.

“More recently, the availability of low-cost and robust sensors (e.g., RGB-D sensors, stereo, etc.) has encouraged the computer vision community to focus on combining the RGB modality with other sensing modalities.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The most common approach is to concatenate heterogeneous features from different modalities to generate an artificial multimodal representation.

“However, the most common approach is to concatenate heterogeneous features from different modalities to generate an artificial multimodal representation.”

May combine two claims.

Such methods tend to use generative adversarial network models to synthesize and reconstruct visual features of unseen classes, resulting in high accuracy classification and ensuring a balance between seen and unseen class labels.

“Such methods tend to use GAN models to synthesize and reconstruct the visual features of the unseen classes, resulting in high accuracy classification and ensuring a balance between seen and unseen class labels [136, 137].”

• cites [136] Xian, Y., Lorenz, T., Schiele, B., Akata, Z.: Feature generating networks for zero-shot learning. In: CVPR, pp. 5542–5551 (2018) sourceNot checked yet

  Not checked in the free run.

• cites [137] Kumar, Y. et al.: Harnessing GANs for Zero-shot Learning of New Classes in Visual Speech Recognition. arXiv:1901.10139 (2020) sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: The amount of labeled data samples for effective model training is often insufficient to recognize all possible object categories in an image.

Probably not a substantial claim: may not pass S3 (load-bearing).

A novel loss function called focal loss is considered by [79] to deal with class imbalance in the training phase.

“To deal with the class imbalance in the training phase, a novel loss function called “focal loss” is considered by [79].”

• cites [79] Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv:1708.02002 (2018) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The most popular use of semantic alignment is to create a dataset with associated labels and then generate a semantically aligned dataset.

“The most popular use of semantic alignment is to create a dataset with associated labels and then generate a semantically aligned dataset.”

May combine two claims.

Both attentional and semantic alignment tasks have proven effective in multimodal alignment, with attentional alignment features better accounting for long-term dependencies between different concepts.

“Both of these tasks have proven effective in multimodal alignment, where attentional alignment features are better able to take into account the long-term dependencies between different concepts.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multi-object detection in both indoor and outdoor environments under varying environmental and lighting conditions remains one of the major challenges facing the computer vision community.

“However, multi-object detection in both indoor and outdoor environments under varying environmental and lighting conditions remains one of the major challenges facing the computer vision community.”

Probably not a substantial claim: may not pass S3 (load-bearing).

SSD can learn multi-scale feature representation from multi-resolution images.

“It can learn multi-scale feature representation from multi-resolution images.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most deep learning-based object tracking systems have been based on CNN architectures.

“Over the last few decades, most deep learning-based object tracking systems have been based on CNN architectures [84, 139].”

• cites [84] Liu, L., et al.: Deep learning for generic object detection: a survey. Int. J. Comput. Vis. 128, 261–318 (2020) sourceNot checked yet

  Not checked in the free run.

• cites [139] Abbass, M.Y., et al.: Efficient object tracking using hierarchical convolutional features model and correlation filters. Vis. Comput. (2020). https://doi.org/10.1007/s00371-020-01833-5 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Visual tracking has been one of the major challenges for the computer vision community for decades.

“For decades, visual tracking has been one of the major challenges for the computer vision community.”

Probably not a substantial claim: may not pass S3 (load-bearing).

SSD's capability to detect objects at different scales enables it to enhance the robustness of the entire chain.

“Its capability to detect objects at different scales enables it to enhance the robustness of the entire chain.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Attentional alignment is based on attentional mapping between the features of the input modality and the target one.

“The attentional alignment task is based on the attentional mapping between the features of the input modality and the target one, while the semantic alignment task takes the form of an alignment method that directly provides alignment capabilities to a predictive model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Wang and colleagues introduced a residual CNN architecture and a loss network to transfer an input picture's artistic style across multiple scales and dimensions.

“In [160], Wang et al. introduced a residual CNN architecture and loss network to transfer the artistic style of the input picture across multiple scales and dimensions.”

• cites [160] Wang, X., et al.: Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer. arXiv:1612.01895 (2017) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Peri and colleagues proposed a multimodal framework that encodes images and captions with CNN and RNN into an intermediate representation and decodes it into a new caption resembling the input.

“Similarly, Peri et al. [53] proposed a multimodal framework that encodes both images and captions using CNN and RNN as an intermediate level representation and then decodes these multimodal representations into a new caption that is similar to the input.”

• cites [53] Peri, D., Sah, S., Ptucha, R.: Show, Translate and Tell. arXiv:1903.06275 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The main difficulty of action recognition lies in extracting body silhouettes from foreground rigid objects.

“The challenge of action recognition is mainly related to the difficulty of extracting body silhouettes from foreground rigid objects to focus on their emotions [96].”

• cites [96] Escalera, S., Athitsos, V., Guyon, I.: Challenges in multimodal gesture recognition. J. Mach. Learn. Res. 17, 1–54 (2016)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

RGB-D cameras provide color and pixel-wise depth information that characterizes the distance of visual objects in a complex scene.

“For instance, RGB-D cameras (e.g., Microsoft Kinect, Asus Xtion, and so on) can provide color and pixel-wise depth information, characterizing the distance of visual objects in a complex scene [199].”

• cites [199] Shamwell, E.J., et al.: Unsupervised deep visual-inertial odometry with online error correction for RGB-D imagery. IEEE Trans. Pattern Anal. Mach. Intell. (2019). https://doi.org/10.1109/TPAMI.2019.2909895 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A universal graph-based style transfer was proposed that transforms multimodal features by matching style patterns with semantic content and appearance, avoiding inflexibility in real-world scenarios.

“The authors of [159] proposed a universal graph-based style transfer to transform multimodal features by matching style patterns and semantic content and appearance in a way that avoids the lack of flexibility in real-world scenarios.”

• cites [159] Zhang, Y. et al.: Multimodal style transfer via graph cuts. In: ICCV, pp. 5943–5951 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Face recognition comprises the subtasks of detecting, identifying and reconstructing faces.

“Note that face detection, face identification, and face reconstruction are subtasks of face recognition [50].”

• cites [50] Wang, M., Deng, W.: Deep Face Recognition: A Survey. arXiv:1804.06655 (2019)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

An unsupervised image captioning framework was presented that uses an alignment method to integrate visual and textual streams through semantic learning of multimodal embeddings.

“The authors of [128] presented an unsupervised image captioning framework based on a new alignment method that allows the simultaneous integration of visual and textual streams through semantic learning of multimodal embeddings of the language and vision domains.”

• cites [128] Laina, I., et al.: Towards unsupervised image captioning with shared multimodal embeddings. In: ICCV, pp. 7414–7424 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A biometric system was introduced that fuses face and iris traits to establish a person's identity.

“The authors of [93] introduced a biometric system that combines biometric traits from different modalities (face and iris) to establish an individual’s identity.”

• cites [93] Ammour, B., Boubchir, L., Bouden, T., Ramdani, M.: Face-Iris multimodal biometric identification system. Electronics 9, 85 (2020) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The style transfer loss is essentially a weighted sum of the content loss between input and output and the style loss between original and applied style.

“It is essentially a weighted sum of the error (loss) between the input content and the output image and the loss between the original style and the applied style [161].”

• cites [161] Jing, Y., et al.: Neural Style Transfer: A Review. arXiv:1705.04058 (2018)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Biten and colleagues combined visual and textual data to generate captions in two stages: template caption generation and entity insertion.

“Most recently, Biten et al. [52] combined both visual and textual data to generate captions across two stages: template caption generation stage and entity insertion stage.”

• cites [52] Biten, A.F., Gomez, L., Rusiñol, M., Karatzas, D.: Good News, Everyone! Context driven entity-aware captioning for news images. arXiv:1904.01475 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Mask R-CNN model offers the possibility of locating instances of objects with class labels and segmenting them with semantic masks.

“For instance, the Mask R-CNN model offers the possibility of locating instances of objects with class labels and segmenting them with semantic masks.”

May combine two claims.

Koo et al. developed a multimodal human recognition framework based on face and body information extracted from deep CNNs, employing late fusion to merge high-level features across modalities.

“In [73], Koo et al. developed a multimodal human recognition framework based on face and body information extracted from deep CNNs. They employed the late fusion policy to merge the high-level features across the different modalities.”

• cites [73] Koo, J.H., Cho, S.W., Baek, N.R., Kim, M.C., Park, K.R.: CNN-based multimodal human recognition in surveillance environments. Sensors 18, 3040 (2018) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A multi-stream architecture built on an RNN (LSTM) was proposed to capture spatial-temporal features from gesture data.

“Regarding the multimodal dimensions of gesture recognition, the authors of [97] proposed a multi-stream architecture based on the RNN (LSTM) model to capture spatial-temporal features from gesture data.”

• cites [97] Nishida, N., Nakayama, H.: Multimodal gesture recognition using multi-stream recurrent neural network. In: Revised Selected Papers of the 7th Pacific-Rim Symposium on Image and Video Technology, pp. 682–694 (2015) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Deep learning algorithms have been developed to improve clinical accuracy and detect abnormalities in medical images, saving time and increasing dependability in patient care.

“In recent years, deep learning algorithms have been developed to save time and dependability during patient care by improving clinical accuracy and detecting abnormalities in medical images [104, 140].”

• cites [104] Xu, Y.: Deep learning in multimodal medical image analysis. In: Health Information Science, pp. 193–200 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [140] Xi, P.: An integrated approach for medical abnormality detection using deep patch convolutional neural networks. Vis. Comput. 36, 1869–1882 (2020) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Gatys and colleagues showed that visual features of models can be combined to represent image styles.

“Gatys et al. [156] demonstrated that visual features of models could be combined to represent image styles.”

• cites [156] Gatys, L., et al.: A Neural Algorithm of Artistic Style. arXiv:1508.06576 (2015) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In deep multimodal Boltzmann machines, each modality is covered individually using adaptive approaches, and after joining multi-domain features, high-level classification is performed by an individual classifier.

“Therefore, each modality will be covered individually using adaptive approaches. After joining the multi-domain features, the high-level classification will be performed by an individual classifier.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The structure of deep multimodal Boltzmann machines is similar to that of DBM, but it can admit more than one modality.

“In [48], the structure of deep multimodal Boltzmann machines (DMBM) [71] is similar to that of DBM, but it can admit more than one modality.”

• cites [48] Alam, M. R., Bennamoun, M., Togneri, R., Sohel, F.: A deep neural network for audio-visual person recognition. In: IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1–6 (2015) sourceNot checked yet

  Not checked in the free run.

• cites [71] Srivastava, N., Salakhutdinov, R.: Multimodal learning with deep Boltzmann machines. J. Mach. Learn. Res. 15(1), 2949–2980 (2014)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The fully convolutional network has become one of the robust models for a wide range of image types.

“Over the last few years, the fully convolutional network (FCN) [29] has become one of the robust models for a wide range of image types (multimedia, aerial, medical, etc.).”

• cites [29] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. arXiv:1411.4038 (2015) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

According to Salakhutdinov et al., a DBM is a generative model that includes several layers of hidden variables.

“According to Salakhutdinov et al. [72], a DBM is a generative model that includes several layers of hidden variables.”

• cites [72] Salakhutdinov, R., Hinton, G.: Deep Boltzmann machines. In: Artificial Intelligence and Statistics, pp. 448–455 (2009)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Lee and colleagues proposed a CNN-based retinal image registration method that learns multimodal features simultaneously from several imaging modalities.

“In [142], Lee et al. proposed a new CNN-based retinal image registration method to learn multimodal features simultaneously from several imaging modalities.”

• cites [142] Lee, J. A., et al.: Deep step pattern representation for multimodal retinal image registration. In: CVPR, pp. 5077–5086 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A multimodal gesture recognition system was developed using the 3D Residual CNN (ResC3D) model trained on an RGB-D dataset, with extracted features combined through a canonical correlation scheme.

“In [98], the authors developed a multimodal gesture recognition system using the 3D Residual CNN (ResC3D) model [99] trained on an RGB-D dataset.”

• cites [98] Miao, Q. et al.: Multimodal gesture recognition based on the ResC3D network. In: IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 3047–3055 (2017) sourceNot checked yet

  Not checked in the free run.

• cites [99] Tran, D., Ray, J., Shou, Z., Chang, S.-F., Paluri, M.: ConvNet architecture search for spatiotemporal feature learning. arXiv:1708.05038 (2017)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Capturing multimodal data streams under varied acquisition conditions and increasing data volume makes visual content easier to recognize.

“In general, capturing multimodal data streams under different acquisition conditions and increasing the data volume makes it easier to recognize visual content.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Each multimodal problem requires a specific fusion strategy, ranging from traditional methods to deep learning techniques.

“Furthermore, it is important to note that each multimodal problem requires a specific fusion strategy, ranging from traditional methods to deep learning techniques.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Reducing computing capacity remains the main challenge and can lead to deterioration in the overall accuracy of training algorithms.

“However, the need to reduce computing capacity remains the main challenge, which can lead to a deterioration in the overall accuracy of training algorithms.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Networks trained with a limited number of examples cannot generalize well to a new dataset.

“However, networks trained with a limited number of examples can no longer generalize well to a new dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Edge/cloud computing provides an effortless way to create and handle multimodal datasets for training and deploying models.

“For instance, the edge/cloud computing solution for mulitmodal analysis provides an effortless way to create and handle multimodal datasets for training and deploying models [107].”

• cites [107] Wang, X., et al.: Convergence of edge computing and deep learning: a comprehensive survey. IEEE Commun. Surv. Tutorials 1, 2020 (2020)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Deep learning models are often robust strategies for handling linear and nonlinear combinations of multimodal data.

“Deep learning models are often robust strategies for dealing with the linear and nonlinear combination of multimodal data.”

Probably not a substantial claim: may not pass S4 (based).

The amount of multimodal data significantly affects the overall performance of a vision system.

“The amount of multimodal data significantly affects the overall performance of the vision system.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Feature-level fusion is more flexible than decision-level fusion because data samples are homogeneous.

“However, feature-level fusion is more flexible than decision-level fusion due to the homogeneity of data samples.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Deep multimodal analysis depends on large amounts of heterogeneous sensor data to reach high performance and avoid overfitting.

“The deep multimodal analysis relies on a large amount of heterogeneous sensor data to achieve high performance and avoid overfitting problems.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Autonomous vehicles, healthcare robots and other real-time embedded systems consume more hardware resources, storage and battery than other emerging technologies, resulting in a lack of adaptation to future needs.

“In practice, autonomous vehicles, healthcare robots, and other real-time embedded systems consume more hardware resources, storage, and battery than other emerging technologies, resulting in a lack of adaptation to future needs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Acquiring and processing large amounts of multimodal data is costly in computation complexity and memory consumption.

“In other words, acquiring and processing a large amount of multimodal data is costly in terms of computation complexity and memory consumption.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Choosing the right fusion of different schemes remains a vital challenge for the computer vision community in terms of accuracy and efficiency.

“Nevertheless, choosing the right fusion of different schemes remains a vital challenge for the computer vision community in terms of accuracy and efficiency.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Autoencoders can address missing patterns by generating intermediate shared representations from input data and revealing intra- and inter-pattern correlations.

“Also, AE can address missing patterns by generating intermediate shared representations from the input data and showing intra- and inter-pattern correlations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

RGB-D sensors have advantages including low computational cost, long range, and an internal mechanism that limits the impact of bad weather.

“Among the advantages of these types of sensors are their low computational cost, their long-range, their ability to have an internal mechanism to limit the impact of bad weather, etc. [149].”

• cites [149] Van Brummelen, J., et al.: Autonomous vehicle perception: the technology of today and tomorrow. Transp. Res. C Emerg. Technol. 89, 384–406 (2018) sourceNot checked yet

  Not checked in the free run.