Multimodal learning with transformers: A survey

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

[1] proposed a structured, acknowledged taxonomy by five challenges, which this survey also adopts as part of its structure.

“In particular, [1] proposed a structured, acknowledged taxonomy by five challenges, which we also adopt as part of our structure.”

• cites [1]Not checked yet

  Not checked in the free run.

May combine two claims.

Token embedding fusion is flexible and widely applied to both uni-modal and multimodal Transformer applications.

“Similar to the flexible tokenization, token embedding fusion is also flexible and widely applied to both uni-modal and multimodal Transformer applications.”

May combine two claims.

There is a recent surge of research attempts and activities across distinct disciplines exploring Transformer architectures, resulting in many novel multimodal learning methods and significant advances.

“Critically, there is a recent surge of research attempts and activities across distinct disciplines exploring the Transformer architectures, resulting in a large number of novel MML methods being developed in recent years, along with significant and diverse advances in various areas [4], [5], [6], [7], [8].”

• cites [4] sourceNot checked yet

  Not checked in the free run.

• cites [5]Not checked yet

  Not checked in the free run.

• cites [6]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Summing position embedding is intrinsically a case of early summation.

“As discussed in Section 3.1.1and 3.3.1,summing position embedding is intrinsically a case of early summation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Several surveys dedicated to Transformers have been recently introduced, with a range of emphases.

“Several surveys dedicated to Transformers have been recently introduced, with a range of emphases including general Transformers [48], efficient designs [49], visualization [50], computer vision tasks [51], [52], [53], [54], medical imaging [55], video tasks [56], and vision language pretraining [57].”

• cites [48]Not checked yet

  Not checked in the free run.

• cites [49] sourceNot checked yet

  Not checked in the free run.

• cites [50] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Instructional videos have become increasingly popular.

“Instructional videos have become increasingly popular, e.g., cooking video YouCookII [155].”

• cites [155]Not checked yet

  Not checked in the free run.

May combine two claims.

Transformers can work in a modality-agnostic way and are compatible with various modalities and combinations of modalities.

“We highlight that Transformers have the advantage that they can work in a modality-agnostic way.Thus, they are compatible with various modalities (and combinations of modalities).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The idea of multi-head self-attention is a kind of ensemble, and it helps the model jointly attend to information from multiple representation sub-spaces.

“The idea of MHSA is a kind of ensemble.MHSA helps the model to jointly attend to information from multiple representation sub-spaces.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Aligning a sequence of instructions to a video of someone carrying out a task is an example of a powerful pretraining pretext task.

“Aligning a sequence of instructions to a video of someone carrying out a task is an example of a powerful pretraining pretext task [7], [156].”

• cites [7]Not checked yet

  Not checked in the free run.

• cites [156] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

When considered from the perspective of geometric topology, each of the modalities listed in Table 1 can be regarded as a graph.

“When considered from the perspective of geometric topology, each of the modalities listed in Table 1 can be regarded as a graph.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The mathematical formulations of self-attention variants are modality-, tokenization-, and embedding-agnostic, as self-attention models the embedding of an arbitrary token from an arbitrary modality as a node of a graph.

“Specifically, the following formulations are modality-, tokenization-, and embedding-agnostic, as self-attention models the embedding of arbitrary token from arbitrary modality as a node of a graph.”

May combine two claims.

Transformers allow each token position to contain multiple embeddings, which is essentially a kind of early-fusion of embeddings for both uni-modal and multimodal Transformer models.

“In practice, Transformers allow each token position to contain multiple embeddings.This is essentially a kind of early-fusion of embeddings, for both uni-modal and multimodal Transformer models.”

May combine two claims.

The encoder of Vanilla Transformer can be regarded as a fully-connected GNN encoder, and the Transformer family has the non-local ability of global perception.

“Therefore, the encoder of Vanilla Transformer can be regarded as a fullyconnected GNN encoder, and the Transformer family has the non-local ability of global perception, similar to the Non-Local Network [167].”

• cites [167] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The input to a Transformer can encompass one or multiple sequences of tokens, and each sequence's attribute naturally allows for multimodal learning without architectural modification.

“Concretely, the input to a Transformer could encompass one or multiple sequences of tokens, and each sequence's attribute (e.g., the modality label, the sequential order), naturally allowing for MML without architectural modification [4].”

• cites [4] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

All discussed self-attention and its variants are flexible enough to be extended to multiple modality cases.

“Please note that all discussed self-attention and its variants are such flexible that can be extended to multiple modality cases.”

May combine two claims.

Data scales are larger; various recently released multimodal datasets are million-scale.

“Data scales are larger.Various recently released datasets are million-scale, e.g., Product1M [137], Conceptual 12M [138], RUC-CAS-WenLan [139] (30M), HowToVQA69M [140], HowTo100M [141], ALT200M [142], LAION-400M [143].”

• cites [137]Not checked yet

  Not checked in the free run.

• cites [138]Not checked yet

  Not checked in the free run.

• cites [139] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Tasks are more difficult; more abstract multimodal tasks are proposed.

“Tasks are more difficult.Beyond the straightforward tasks, more abstract multimodal tasks are proposed, e.g., MultiMET [153] (a multimodal dataset for metaphor understanding), Hateful Memes [154] (hate speech in multimodal memes).”

• cites [153] sourceNot checked yet

  Not checked in the free run.

• cites [154] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

High-capacity Transformer models and multimodal big data basis co-create the prosperity of Transformer-based multimodal machine learning.

“Therefore, their highcapacity models and multimodal big data basis co-create the prosperity of the Transformer based multimodal machine learning.”

• AssumptionRests on an assumption: Transformers are data hungry.

Probably not a substantial claim: may not pass S3 (load-bearing).

Early summation's main advantage is that it does not increase computational complexity.

“Its main advantage is that it does not increase computational complexity.”

May combine two claims.

Cross-attention, if considered for each modality, fails to perform cross-modal attention globally and loses the whole context.

“however if considered for each modality, this method fails to perform cross-modal attention globally and thus loses the whole context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

TriBERT is a tri-modal cross-attention (coattention) for vision, pose, and audio, where given a Query embedding, its Key and Value embeddings are the concatenation from the other modalities.

“TriBERT [183] is a tri-modal cross-attention (coattention) for vision, pose, and audio, where given a Query embedding, its Key and Value embeddings are the concatenation from the other modalities.”

• cites [183]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hierarchical attention (multi-stream to one-stream) is an implementation of late interaction/fusion and can be treated as a special case of early concatenation.

“This kind of hierarchical attention is an implementation of late interaction/fusion, and can be treated as a special case of early concatenation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multiple cross-attention streams are used in hierarchical attention (one-stream to multi-stream) in a two-stream decoupled model, where Tf2 and Tf3 of Eq. 11 are implemented by cross-attention.

“multiple cross-attention streams are used in hierarchical attention (one-stream to multi-stream) that in a two-stream decoupled model [191] T f 2 and T f 3 of Eq. 11 are implemented by cross-attention defined in Eq. 12.”

• cites [191]Not checked yet

  Not checked in the free run.

May combine two claims.

Transformer-based models pretrained on large-scale multimodal corpora clearly outperform other competitors in a wide range of multimodal downstream tasks and achieve zero-shot generalization ability.

“Recent work has demonstrated that if pretrained on large scale multimodal corpora Transformer based models [7], [102], [103], [104], [105], [106], [110] clearly outperform other competitors in a wide range of multimodal down-stream tasks, and moreover achieve the zero-shot generalization ability.”

• cites [7]Not checked yet

  Not checked in the free run.

• cites [102] sourceNot checked yet

  Not checked in the free run.

• cites [103] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Human behaviours, emotions, events, actions, and humour are multimodal, so various human-centred multimodal learning tasks are widely studied.

“Furthermore, human behaviours, emotions, events, actions, and humour are multimodal, thus various human-centred MML tasks are widely studied, including multimodal emotion recognition [67], multimodal event representation [68], understanding multimodal humor [69], face-body-voice based video person-clustering [70], etc.”

• cites [67] sourceNot checked yet

  Not checked in the free run.

• cites [68]Not checked yet

  Not checked in the free run.

• cites [69] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Cross-attention to concatenation is applied to three modalities (language, video, and audio) in [189].

“Cross-attention to concatenation is applied to three modalities (i.e., language, video, and audio) in [189].”

• cites [189]Not checked yet

  Not checked in the free run.

May combine two claims.

Cross-attention to concatenation is widely studied and alleviates the drawback of cross-attention.

“This kind of hierarchically cross-modal interaction is also widely studied [137], [189], and alleviates the drawback of cross-attention.”

• cites [137]Not checked yet

  Not checked in the free run.

• cites [189]Not checked yet

  Not checked in the free run.

The quote only partly states this claim.

Hierarchical attention (multi-stream to one-stream) combines Transformer layers hierarchically to attend to cross-modal interactions.

“Transformer layers can be combined hierarchically to attend to the cross-modal interactions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Various multimodal Transformers work due to their internal multimodal attentions (self-attention variants).

“Essentially, various multimodal Transformers work due to their internal multimodal attentions that are the aforementioned self-attention variants.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Vision Transformer (ViT) is a seminal work that contributes an end-to-end solution by applying the encoder of Transformer to images.

“Vision Transformer (ViT) [5] is a seminal work that contributes an end-to-end solution by applying the encoder of Transformer to images.”

• cites [5]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

InterBERT's method perceives cross-modal interactions and preserves the independence of uni-modal representation.

“This method perceives the cross-modal interactions and meanwhile preserves the independence of uni-modal representation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

VideoBERT is one of the first multimodal Transformer works.

“VideoBERT [7] is the one of the first multimodal Transformer works”

• cites [7]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The longer sequence after concatenation increases computational complexity.

“However, the longer sequence after concatenation will increase computational complexity.”

May combine two claims.

Two-stream cross-attention can learn cross-modal interaction, but there is no self-attention to the self-context inside each modality.

“As discussed in [188], two-stream crossattention can learn cross-modal interaction, whereas there is no self-attention to the self-context inside each modality.”

• cites [188] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

A majority of Transformer-based multimodal pretraining works in a self-supervised manner and is overly dependent on well-aligned multimodal sample pairs/tuples.

“A majority of Transformer-based multimodal pretraining works in a self-supervised manner, however, it is overly dependent on the well-aligned multimodal sample pairs/tuples.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multimodal Transformer models have another architecture taxonomy based on the computational size of the components.

“As demonstrated in Figure 2 in [192], the multimodal Transformer models have another architecture taxonomy based on the computational size of the components.”

• cites [192] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

From the timing of interaction, multimodal attentions fall into three categories: early interaction, late interaction, and throughout interaction.

“From the perspective of timing of interaction, these multimodal attentions fall into three categories, i.e., early interaction: early summation, early concatenation, and hierarchical attention (one-stream to multi-stream), late interaction: hierarchical attention (multi-stream to one-stream), or throughout interaction: cross-attention, cross-attention to concatenation.”

May combine two claims.

Cross-attention attends to each modality conditioned on the other and does not cause higher computational complexity.

“Cross-attention attends to each modality conditioned on the other and does not cause higher computational complexity”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multimodal Transformers can be divided into single-stream, multi-stream, and hybrid-stream.

“Thus, multimodal Transformers can be divided into single-stream (e.g., Uniter [106], Visualbert [104], Vl-bert [105] , Unified VLP [110]), multi-stream (e.g., ViLBERT [102], Lxmert [103], ActBERT [114]), hybrid-stream (e.g., InterBERT [188]), etc.”

Probably not a substantial claim: may not pass S4 (based).

Using cross-modal alignment as cross-modal supervision is costly for large-scale applications.

“However, using cross-modal alignment as cross-modal supervision is costly for large-scale applications.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Vanilla Transformer benefits from a self-attention mechanism and is a breakthrough model for sequence-specific representation learning originally proposed for NLP, achieving state-of-the-art on various NLP tasks.

“Benefit from its self-attention, Vanilla Transformer [2] benefits from a self-attention mechanism, and is a breakthrough model for sequence-specific representation learning that was originally proposed for NLP, achieving the state-of-the-art on various NLP tasks.”

• cites [2] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Self-attention variants for multimodal interactions are modality-generic and can be applied in flexible strategies and for multi-granular tasks.

“All these aforementioned self-attention variants for multimodal interactions are modality-generic, and can be applied in flexible strategies and for multi-granular tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cross-attention (coattention) was first proposed in VilBERT.

“This method is termed cross-attention or coattention [190], which was first proposed in VilBERT [102]”

• cites [102] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Early summation and early concatenation work in single-stream; cross-attention works in multi-streams; hierarchical attention and cross-attention to concatenation work in hybrid-streams.

“if we consider from the angle of network structures, (1) early summation and early concatenation work in single-stream, (2) cross-attention work in multistreams, (3) hierarchical attention and cross-attention to concatenation work in hybrid-streams.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multimodal learning has been an important research area in recent decades; an early multimodal application, audiovisual speech recognition, was studied in the 1980s.

“MML [1], [60], [61] has been an important research area in recent decades; an early multimodal application -audiovisual speech recognition was studied in 1980s [62].”

• cites [62] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Speech can be converted to text by off-the-shelf ASR tools in a multimodal context.

“Thanks to recent advances in automatic speech recognition (ASR) techniques, in a multimodal context, speech can be converted to text by the offthe-shelf speech recognition tools.”

May combine two claims.

Both ViT and its variants have been widely applied to various computer vision tasks and work well for both supervised and self-supervised visual learning.

“Both ViT and its variants have been widely applied to various computer vision tasks, including low-level tasks [92], recognition [93], detection [94], segmentation [95], etc, and also work well for both supervised [93] and self-supervised [96], [97], [98] visual learning.”

• cites [92] sourceNot checked yet

  Not checked in the free run.

• cites [93]Not checked yet

  Not checked in the free run.

• cites [96]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

To the authors' knowledge, only a few surveys on video-language pretraining are relevant to multimodal learning.

“To our knowledge, only a few surveys on video-language pretraining (VLP) [57], [58], [59] are relevant to MML.”

• cites [57]Not checked yet

  Not checked in the free run.

• cites [58] sourceNot checked yet

  Not checked in the free run.

• cites [59]Not checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: Video-language pretraining is only a subdomain of multimodal learning.

Probably not a substantial claim: may not pass S3 (load-bearing).

These attentions determine the external network structures of the multimodal Transformers where they are embedded.

“Meanwhile, as illustrated in Figure 2, these attentions determine the external network structures of the multimodal Transformers where they are embedded.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The world humans live in is a multimodal environment, so both human observations and behaviours are multimodal.

“The world we humans live in is a multimodal environment, thus both our observations and behaviours are multimodal [63].”

• cites [63] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

If consider motivations, pretext tasks include masking, describing, matching, ordering, etc.

“If consider their motivations, the pretext tasks include masking, describing, matching, ordering, etc.”

Probably not a substantial claim: may not pass S3 (load-bearing).

How to use weakly-aligned or even unpaired/unaligned multimodal data as pretraining corpora is still understudied.

“Thus, how to use the weakly-aligned or even unpaired/unaligned multimodal data as the pretraining corpora is still understudied.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sentence Ordering Modelling in the text domain and Frame Ordering Modelling in the video domain share the same idea.

“while both Sentence Ordering Modelling (SOM) [201] in text domain and Frame Ordering Modelling (FOM) [116] in video domain share the same idea.”

• cites [201]Not checked yet

  Not checked in the free run.

• cites [116] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

BERT-style cross-modal pretraining models produce excellent results on various downstream vision-language tasks but fail to be applied to generative tasks directly.

“while the BERT-style cross-modal pretraining models produce excellent results on various down-stream visionlanguage tasks, they fail to be applied to generative tasks directly.”

• cites [208] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Transformer-based multimodal pretraining pipelines have three key components, from bottom to top: tokenization, Transformer representation, and objective supervision.

“In general, Transformer-based multimodal pretraining pipelines have three key components, from bottom to top, i.e., tokenization, Transformer representation, objective supervision.”

May combine two claims.

Both VideoBERT and CBT have to train a separate video-to-text decoder for video captioning.

“As discussed in [208], both VideoBERT [7] and CBT [107] have to train a separate video-to-text decoder for video captioning.”

• cites [208] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Common multimodal pretraining Transformers use well-aligned, weakly-aligned, and even unaligned multimodal sample pairs/tuples to work in supervised, weakly-supervised, and unsupervised manners, respectively.

“The common multimodal pretraining Transformers use well-aligned, weakly-aligned, and even unaligned multimodal sample pairs/tuples, to work in supervised, weakly-supervised, and unsupervised manners, respectively.”

May combine two claims.

Vision Language Pretraining follows two general pipelines: two-stage (needs object detector) and end-to-end.

“Vision Language Pretraining (VLP) follows two general pipelines: two-stage (need object detector, e.g., Faster R-CNN [202]) (e.g., LXMERT [103], ViLBert [102], VL-Bert [105], UNITER [106]) and end-to-end (e.g., Pixel-Bert [113], SOHO [203], KD-VLP [204], Simvlm [199]).”

• cites [202] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Well labelled/aligned cross-modal datasets are very costly in collecting and annotating.

“Well labelled/aligned cross-modal datasets are very costly in collecting and annotating”

May combine two claims.

Some recently successful practice used weakly aligned image-text pairs to perform pretraining and achieve both competitive performance and zero-shot learning capability for image classification, image-text retrieval, and open-ended visual question answering.

“Some recently successful practice [9], [199], [205] used weakly aligned image-text pairs to perform pretraining, and achieve both competitive performance and zero-shot learning capability for image classification, image-text retrieval, and open-ended visual question answering, etc.”

• cites [9]Not checked yet

  Not checked in the free run.

• cites [199] sourceNot checked yet

  Not checked in the free run.

• cites [205] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Nowadays self-supervised attempts are the majority.

“Nowadays, self-supervised attempts are the majority.”

May combine two claims.

There is a significant gap between pretraining models designed for discriminative and generative tasks, mainly because discriminative task oriented pretraining models do not involve the decoders of Transformer.

“This is a significant gap between the pretraining models designed for discriminative and generative tasks, as the main reason is discriminative task oriented pretraining models do not involve the decoders of Transformer.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Pretext tasks can be combined in practice.

“In practice, pretext tasks can be combined, and some representative cases are summarized in Table 3 of [57], Table 2 of [58].”

• cites [57]Not checked yet

  Not checked in the free run.

• cites [58] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Self-attention helps Transformers intrinsically work in a modality agnostic pipeline compatible with various modalities by taking the embedding of each token as a node of a graph.

“Considering from a geometrically topological perspective, self-attention helps Transformers intrinsically work in a modality agnostic pipeline that is compatible with various modalities by taking in the embedding of each token as a node of graph”

Probably not a substantial claim: may not pass S3 (load-bearing).

Masked language and region modelling as pre-training tasks have the main advantage that the Transformer encoder learned from these supervisions can encode both vision and language patterns based on bidirectional context and is naturally fit for semantic understanding tasks.

“As discussed in [208], the masked language and region modelling as pre-training task have a main advantage that the Transformer encoder learned from these supervisions can encode both vision and language patterns based on bidirectional context and it is naturally fit for the semantic understanding tasks, e.g., VQA, image-text retrieval.”

• cites [208] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In multimodal pretraining scenarios, Transformer models work based on the self-attention variants discussed in Section 3.3.2.

“Essentially, in multimodal pretraining scenarios, Transformer models work based on those self-attention variants that are discussed in Section 3.3.2.”

Probably not a substantial claim: may not pass S3 (load-bearing).

From supervision, pretexts can be sorted into unsupervised/self-supervised and supervised.

“the pretexts can be sorted into unsupervised/self-supervised (e.g., masked language modelling (MLM) [7], [137]) and supervised (e.g., image-text matching (ITM) [188] [102], [103], [104], [106], [209]), etc.”

• cites [7]Not checked yet

  Not checked in the free run.

• cites [137]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The core of the Transformer ecosystem is self-attention.

“The core of Transformer ecosystem is self-attention”

May combine two claims.

Multi-task training (by adding auxiliary loss) and adversarial training improve multimodal pretraining Transformers to further boost performance.

“Some practices demonstrate that multi-task training (by adding auxiliary loss) [111], [137] and adversarial training [210] improve multimodal pretraining Transformers to further boost the performance.”

• cites [111] sourceNot checked yet

  Not checked in the free run.

• cites [137]Not checked yet

  Not checked in the free run.

• cites [210] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Weakly aligned cross-modal samples are popular in real-life scenarios; enormous weakly aligned multimodal data samples are emerging in e-commerce due to fine-grained categories, complex combinations, and fuzzy correspondence.

“However, weaklyaligned cross-modal samples are popular in the real-life scenarios, for instance, enormous weakly aligned multimodal data samples are emerging in e-commerce [137], due to fine-grained categories, complex combinations, and fuzzy correspondence.”

• cites [137]Not checked yet

  Not checked in the free run.

May combine two claims.

Down-stream task-agnostic pretraining is optional, not necessary, and down-stream task specific pretraining is widely studied.

“In practices of multimodal Transformers, the aforementioned down-stream task -agnostic pretraining is optional, not necessary, and down-stream task specific pretraining is also widely studied [150], [190], [208], [211].”

• cites [150]Not checked yet

  Not checked in the free run.

• cites [190] sourceNot checked yet

  Not checked in the free run.

• cites [208] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Morgado et al. propose to perform contrastive audio-visual spatial alignment of 360° video and spatial audio to learn from spatial cues in 360° video.

“To learn from the spatial cues in 360 • video, Morgado et al. [145] propose to perform contrastive audio-visual spatial alignment of 360 • video and spatial audio.”

• cites [145] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Transformers contribute to various multimodal generative tasks, including single-modality to single-modality, multimodality to single modality, and multimodality to multimodality.

“Meanwhile, Transformers also contribute to various multimodal generative tasks, including single-modality to single-modality (e.g., raw audio to 3D mesh sequence [39], RGB to 3D scene [40], single image to 3D human texture estimation [223], RGB to scene graph [19], [224], [225], [226], graph to graph [33], knowledge graph to text [227], video to scene graph [228], video to caption [229], [230], [”

May combine two claims.

There are nonnegligible gaps among various down-stream applications (e.g., task logic, data form), making it difficult to transfer from pretraining to down-stream applications.

“There are nonnegligible gaps among various down-stream applications, e.g., task logic, data form, making it difficult to transfer from pretraining to down-stream applications.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Cross-modal alignment is the key to a number of real-world multimodal applications.

“Cross-modal alignment is the key to a number of real-world multimodal applications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If aim to learn more explicit object concepts, more complex pretext losses will be used.

“In general, if aim to learn more explicit object concepts, more complex pretext losses will be used [204].”

• cites [204] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

This makes sense considering the motivations of learning stronger multimodal contextual representations and great advance of computing power.

“This makes sense considering the motivations of learning stronger multimodal contextual representations and great advance of computing power.”

May combine two claims.

XGPT is tailor-made for image captioning, to overcome the limitation that BERT-based cross-modal pretrained models fail to be applied to generative tasks directly.

“XGPT [208] is tailor-made for image captioning, to overcome the limitation that BERT-based cross-modal pretrained models fail to be applied to generative tasks directly.”

• cites [208] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Med-BERT is a contextualized embedding model pretrained on a structured electronic health record dataset of two million patients.

“Med-BERT [184] is a contextualized embedding model pretrained on a structured electronic health record dataset of two million patients.”

• cites [184] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Common early fusion based MML Transformer models are also known as one-stream architecture, allowing adoption of the merits of BERT due to minimal architectural modification.

“Common early fusion based MML Transformer models [7], [104], [108] are also known as one-stream architecture, allowing the adoption of the merits of BERT due to minimal architectural modification.”

• cites [7]Not checked yet

  Not checked in the free run.

• cites [104] sourceNot checked yet

  Not checked in the free run.

• cites [108]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

GraphCodeBERT uses two structure-aware pretext tasks for programming source code.

“GraphCodeBERT [44] uses two structure-aware pretext tasks (i.e., predict where a variable is identified from, data flow edge prediction between variables) for programming source code.”

• cites [44]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The main difference between one-stream models is the usage of problem-specific modalities with variant masking techniques.

“The main difference between these one-stream models is the usage of problem-specific modalities with variant masking techniques.”

May combine two claims.

Models based on this idea are often enormous in size and expensive to optimize from millions or billions of training data.

“The models based on this idea are often enormous in size and expensive to optimize from millions or billions of training data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Kaleido-BERT is a VLP Transformer model tailor-made for the fashion domain.

“Kaleido-BERT [212] is a VLP Transformer model tailor-made for the fashion domain.”

• cites [212] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Image classification is conventionally regarded as a unimodal learning problem and zero-shot classification remains an unsolved challenge despite extensive research.

“given that image classification is conventionally regarded as a unimodal learning problem and zero-shot classification remains an unsolved challenge despite extensive research [268].”

• cites [268]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

ERNIE-ViLG is designed for bidirectional image-text generation with Transformers.

“ERNIE-ViLG [211] is designed for bidirectional image-text generation with Transformers.”

• cites [211]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Murahari et al. present a visual dialogue oriented approach to leverage pretraining on general vision-language datasets.

“Murahari et al. [190] present a visual dialogue oriented approach to leverage pretraining on general vision-language datasets.”

• cites [190] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Several ideas introduced recently include random sampling, learning concept dictionary, uniform masking, patch projection, joint learning of a region detector, and representation aligning before mask prediction.

“Several ideas introduced recently include random sampling [113], learning concept dictionary [203], uniform masking [270], patch projection [192], joint learning of a region detector [271], and representation aligning before mask prediction [263].”

• cites [113] sourceNot checked yet

  Not checked in the free run.

• cites [203] sourceNot checked yet

  Not checked in the free run.

• cites [270]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

This extends the taxonomy introduced in [1] to tackle higher diversity and wider scopes of existing Transformer based MML works in recent years.

“This further extends the taxonomy introduced in [1] to tackle the higher diversity and wider scopes of existing Transformer based MML works in recent years.”

• cites [1]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Transformer based multimodal learning has seven challenges: fusion, alignment, transferability, efficiency, robustness, universalness, and interpretability.

“We discuss seven challenges of Transformer based multimodal learning, including fusion, alignment, transferability, efficiency, robustness, universalness, and interpretability.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

DropToken can be treated as an implementation of dropout or adversarial training.

“DropToken can be treated as an implementation of dropout or adversarial training.”

May combine two claims.

One solution is using knowledge distillation, e.g., distilling from multimodal to uni-modal attention in Transformers, distilling from multiple uni-modal Transformer teachers to a shared Transformer encoder.

“One solution is using knowledge distillation, e.g., distilling from multimodal to uni-modal attention in Transformers [275], distilling from multiple uni-modal Transformer teachers to a shared Transformer encoder [276].”

• cites [275] sourceNot checked yet

  Not checked in the free run.

• cites [276] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Transformer-LS is an efficient Transformer for both language and vision long sequence, with linear computational and memory complexity.

“Transformer-LS [287] is an efficient Transformer for both language and vision long sequence, with linear computational and memory complexity.”

• cites [287]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

DropToken reduces training complexity via random dropping a portion of video and audio tokens from input sequence during training.

“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”

• cites [174] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Kervadec et al. explore how transferable reasoning patterns are in VQA, and demonstrate that for LXMERT/BERT-like reasoning patterns can be partially transferred from an ideal dataset to a real dataset.

“For instance, Kervadec et al. [272], [273] explore how transferable reasoning patterns are in VQA, and demonstrate that for LXMERT [103]/BERT-like reasoning patterns can be partially transferred from an ideal dataset to a real dataset.”

• cites [272]Not checked yet

  Not checked in the free run.

• cites [273]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

GilBERT is a generative VLP model for a discriminative task, i.e., image-text retrieval.

“Recently, more and more attempts study this issue further, e.g., GilBERT [222] is a generative VLP models for a discriminative task, i.e., image-text retrieval.”

• cites [222]Not checked yet

  Not checked in the free run.

May combine two claims.

VILLA is a two-stage strategy (task-agnostic adversarial pretraining followed by task-specific adversarial finetuning) that improves VLP Transformers.

“VILLA [210] is a two-stage strategy (taskagnostic adversarial pretraining, followed by task-specific adversarial finetuning) that improves VLP Transformers.”

• cites [210] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Lee et al. propose a novel parameter sharing scheme based on low-rank approximation.

“Lee et al. [280] propose a novel parameter sharing scheme based on lowrank approximation.”

• cites [280]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Supervised data samples (well-labelled, well-aligned) are costly in practical applications.

“For instance, supervised data samples (well-labelled, well-aligned) are costly in practical applications”

May combine two claims.

CLIP is an inspiring solution that transfers knowledge across modalities by learning a shared multimodal embedding space, enabling zero-shot transfer to downstream tasks.

“CLIP [9] is an inspiring solution that transfers knowledge across modalities by learning a shared multimodal embedding space, enabling zero-shot transfer of the model to down-stream tasks.”

• cites [9]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Knowledge distillation distills knowledge from trained larger Transformers to smaller Transformers.

“Distill the knowledge from the trained larger Transformers to smaller Transformers [93].”

• cites [93]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Cross-lingual gap should be considered for the transferability of Transformer based multimodal learning.

“Cross-lingual gap also should be considered for the transferability of Transformer based multimodal learning, e.g., universal cross-lingual generalization from English to non-English multimodal contexts [206], [277].”

• cites [206]Not checked yet

  Not checked in the free run.

• cites [277] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

In multimodal contexts, calculation explosion will become worse due to jointly high dimension representations.

“In multimodal contexts, calculation explosion will become worse due to jointly high dimension representations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two-stage pipeline is costly as they need object detector.

“Taking the VLP Transformer models as an example, two-stage pipeline is costly as they need object detector.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Wen et al. present a weight-sharing Transformer on top of visual and textual encoders to align text and image.

“Wen et al. [279] present a weight-sharing Transformer on top of the visual and textual encoders to align text and image.”

• cites [279] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Universalness is an important problem for multimodal Transformer models because of the high diversity of tasks and modalities of multimodal learning.

“Due to the highly diversity of tasks and modalities of multimodal learning, universalness is an important problem for multimodal Transformer models.”

May combine two claims.

Existing multimodal Transformer models are superior only for specific MML tasks, as they are designed specifically for only a subset of specific tasks.

“existing multimodal Transformer models [9], [199], [263] are superior only for specific MML tasks, as they are designed specifically for only a subset of specific tasks [137], [142], [212], [249], [260], [261], [265], [266].”

• cites [9]Not checked yet

  Not checked in the free run.

• cites [199] sourceNot checked yet

  Not checked in the free run.

• cites [263] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Unifying the pipelines of different modalities and tasks generally causes larger or more complicated model configuration, whereas for a specific modality or task some components are redundant.

“Unifying the pipelines of different modalities and tasks generally cause larger or more complicated model configuration, whereas for a specific modality or task, some components are redundant.”

May combine two claims.

The interactions and synergy across different modalities are intrinsically complex and even potentially inconsistent over tasks.

“the interactions and synergy across different modalities are intrinsically complex and even potentially inconsistent over tasks [249].”

• cites [249] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Multimodal Transformers pretrained on large-scale corpora achieve state-of-the-art for various multimodal applications, while their robustness is still unclear and understudied.

“Multimodal Transformers pretrained on large-scale corpora achieve the state-of-the-art for various multimodal applications, while their robustness is still unclear and understudied.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cao et al. design a set of probing tasks on UNITER and LXMERT to evaluate what patterns are learned in pretraining.

“Cao et al. [299] design a set of probing tasks on UNITER [106] and LXMERT [103], to evaluate what patterns are learned in pretraining.”

• cites [299]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

ViL-BERT fails to exploit linguistic structure.

“Akula et al. [292] empirically demonstrate that ViL-BERT fails to exploit linguistic structure”

• cites [292] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

VILLA is a generic adversarial training framework that can be applied to various multimodal Transformers.

“VILLA [210] is a generic adversarial training framework that can be applied to various multimodal Transformers.”

• cites [210] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

CLIP converts zero-shot recognition to retrieval, thus reducing the costs of modifying the model.

“CLIP [9] converts zero-shot recognition to retrieval, thus reduces the costs of modifying the model.”

• cites [9]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multi-task loss functions increase the complexity of training.

“Multi-task loss functions increase the complexity of training.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Why and how Transformers perform so well in multimodal learning has been investigated.

“Why and how Transformers perform so well in multimodal learning has been investigated [106], [299], [300], [301], [302], [303], [304], [305], [306].”

• cites [106]Not checked yet

  Not checked in the free run.

• cites [299]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

For image-language Transformers, verb understanding is harder than subject or object understanding.

“Hendricks et al. [301] probe the image-language Transformers by fine-grained image-sentence pairs, and find that verb understanding is harder than subject or object understanding.”

• cites [301] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Chen et al. examine the optimal combination of pretraining tasks via ablation study to compare how different pretexts contribute to the Transformers.

“Chen et al. [106] examine the optimal combination of pretraining tasks via ablation study, to compare how different pretexts contribute to the Transformers.”

• cites [106]Not checked yet

  Not checked in the free run.

May combine two claims.

FSN's fusion-bottleneck strategy improves fusion performance and reduces computational cost.

“This strategy uses the fusion bottleneck as a bridge, and not only improves fusion performance, but also reduces computational cost.”

• cites [175] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

For multimodal Transformer pipelines, understanding and discriminative tasks require Transformer encoders only, while generation/generative tasks require both Transformer encoders and decoders.

“understanding and discriminative tasks require Transformer encoders only, while generation/generative tasks require both Transformer encoders and decoders.”

Probably not a substantial claim: may not pass S2 (checkable).

A clear gap remains between the state-of-the-art and the ultimate goal of universal MML models.

“Furthermore, a clear gap remains between the state-of-the-art and this ultimate goal.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Yan et al.'s sequential adjacent-view fusion solution is intrinsically a greedy strategy.

“This is intrinsically a greedy strategy.”

May combine two claims.

FSN passes messages through a small number of bottleneck latents, requiring the model to purify the most necessary information from each modality for crossmodal sharing.

“FSN passes on the messages through a small number of bottleneck latents, thus requiring the model to purify the most necessary information from each modality for crossmodal sharing.”

• cites [175] sourceNot checked yet

  Not checked in the free run.