Separating Style and Content with Bilinear Models

“Owing to excellent nonlinear feature learning ability of deep learning [11, 12, 14, 19], deep-learning-based cross-media common semantic representation learning methods have attracted broad attentions [13, 40].”

“Disentan-gling the factors of variation is a desirable property of learned representations [5], which has been investigated for a long time [51, 52, 13, 15].”

“These approaches which are dubbed as subspace learning models include unsupervised approaches [7][8][9], supervised approaches [10][11], deep neural network based approaches [12][13], etc.”

“Compact Bilinear Pooling Second-order pooling methods (Tenenbaum and Freeman, 2000) have been shown to be effective at encoding local spatial information for fine-grained visual recognition tasks using CNN models (Lin et al., 2015; Yu et al., 2018).”

“These include essentially additive factor models as used in principal component analysis[9], independent component analysis[3], and bilinear models[11].”

“Like our model, Tenenbaum and Freeman’s [14] and Grimes et al.’s [8] bilinear models and Vasilescu and Terzopoulos’ multilinear model [16] also represented variations in illumination and facial expression, but illumination and appearance are not independent in their models.”

“This technique is known as transfer learning [8], and it has been widely used in applications like handwriting recognition [2], [9], face pose classification [10] etc.”

“Contentstyle separation is addressed via bilinear models in [Tenenbaum and Freeman 2000].”

“In the language of multilinear analysis [24], the features f correspond to the content factor and z the style parameters, which describe the object’s properties.”

“Because it is difficult to divide the observation data into two independent feature spaces in many applications, such as speech recognition, asymmetric bilinear models are usually employed [8].”

“The history of two-stream learning can be traced back to over a decade ago when a “bilinear model” was proposed by [37] to separate the “style” and “content” of an image.”

“factorizations between multiple causes of stimulus appearance, such as 'content' and 'form' (like alphabetic characters and the font in which they are displayed) [9][10].”

“(iv) Bilinear Model (BLM) [23].”

“The Compact Bilinear pooling model [3] is a technique to compress the high-dimension output of a traditional bilinear pooling model [10]; MCB is a multimodal version of this technique.”

“To tackle the above mentioned issues, in this paper, we propose a new MTMV learning method based on asymmetric bilinear factor analyzers (aBFA) [15] and rank constraints to model task-view interactions.”

“Aspect-level sentiment analysis (ASA) aims to predict the sentiment polarity of specific aspects of the input text at a fine-grained level, rather than the sentiment polarity of the whole sentence [34].”

“In order to improve the generative performance of a deep generative model, early attempts were the proposal of restricted Boltzman machines [12] and the usage of bilinear models [13].”

“d value can be interpreted as the generation of “transform coefficients.” Whether decomposing signals into low frequency a nd high frequency [27]; predictable and unpredictable [28]; style and content [29]; object and texture [30]; or dictionary and pattern [31], divideand-conquer approaches have been used to good adva…”

“From C, we can decompose the target type factor ik by using the asymmetrical bilinear model to perform tensor decomposition as in [ 19 ].”

“Therefore, in spite of the large complexity of the model, texture generation can be done in reasonable time using GPUs and performance-optimised toolboxes for training deep neural networks [24].”

“work has tackled this problem through the use of bilinear factorization models (Freeman and Tenenbaum, 1997; Tenenbaum and Freeman, 2000), while others have used discriminative neural models (Zhang et al.”

“Third, we would like to consider Tucker decomposition problems and revisit Tenenbaum and Freeman (2000) and Memisevic and Hinton (2010) through the lens of Resonating Circuits.”

“Early attempts including bilinear model [30], autoencoder [11], manifold learning [8], Restricted Boltzmann Machine [26] and so on.”

“The mechanism of the popular generic style transfer methods at this stage can be seen as a natural evolution of the bilinear model [26] where, we have, ( ) ( ) cs s f f Therefore, the adaptive instance normalization in AdaIN is an unbiased style transfer module.”

“Using the bilinear pooling fusion approach, reference [19, 20] fused two different modalities into a joint representation space by computing the outer product of visual feature vectors and text feature vectors.”

“Compared with traditional linear models, bilinear models can capture richer feature relationships [31].”

“Bilinear models [8] are usually used to model systems with two factors and each factor itself is linear given the other factor fixed.”

“Because of this key difference the model of [1] also requires multiple observations of the same content with different styles, i.e., paired data, which our model does not and can operate on unpaired data collections.”

“2) Bilinear Composition: A bilinear model [59] is employed to merge h m and i a and it can be formulated as the following equation: W ∈ R t × v × o is the bilinear tensor that inspires the interaction between two modalities to make better semantic complement.”

“The bilinear model provides richer representations than the traditional linear model [29], which is successfully applied to visual recognition [30], text generation [31] and visual question answer [32] and so on.”

“Given an image embedding v ∈ R m and a question embedding q ∈ R n , the fully parameterized bilinear pooling (FBP) [10] is simply about applying the outer product operation between the two vectors. technique tends to quadratic expand the feature space dimensionality, producing a bilinear vector z ∈…”

“[68] make use of the Bilinear Model [69] to encode identity and expression in independent controlled parameters, and thus perform joint expression-invariant FR and identity-invariant FER.”

“In [8], a bilinear factorisation of the model coefficients of each sample is proposed to compute a statistical model for reconstruction of the heart at discrete points in the cardiac cycle, inspired by the work on separating style and content [9].”

“Discriminative feature learning BCNN inspired by the Tanenbaum and Freemans bilinear models [42] is effective on fine-grained image recognition.”

“They show on several datasets that the constrained Locally Linear Embedding gives better reconstruction errors than supervised algorithms, factor analysis and a model similar to the bilinear model proposed by Tenenbaum and Freeman to separate style form content [21].”

“They displayed two-mode examples in extrapolating fonts for unseen letters and translating face postures to new illuminations [8].”

“To borrow the terminology of [14] ,we branch over either the style variables or the content ones, and it has been proven in [7] that this branching strategy results in a convergent algorithm.”

“Thus, the bilinear model [9], which is first proposed by Tenenbaum and Freeman to separate the independent “content” and “style” factors of the observations in perceptual systems, is used in this work.”

“Multilinear tensor analysis is an extension of PCA (one orthogonal factor), and bilinear models (two orthogonal factors).(42) Tensor decomposition can be achieved by higher-order singular value decomposition (HOSVD),2544 which is a generalization of SVD.”

“Style consistency is important information and has been widely used in text recognition [4,33,36].”

“Multilinear factor analyzers (MLFA) [23] are further generalization of bilinear models so that they model multiplicative interactions of N different latent variables, corresponding to N groups of factors.”

“Early Fusion consists of feature level fusion techniques like feature concatenation, state-of-the-art subspace learning methods; for example Canonical Correlation Analysis (CCA), Kernel Canonical Correlation Analysis (KCCA) [1], Generalized Canonical Correlation Analysis (GCCA) [16] [29], Regularized Generalized Canon…”

“Bilinear units have been previously proposed and developed [12], [31], [39].”

“The bilinear pooling operation [35], constructs a global feature vector for a given input image, using the outer product of two vectors x ∈ Rn1 and y ∈ Rn2 :”

“Bilinear models were first introduced by Tenenbaum and Freeman [22] to separate style and content.”

“Before the development of deep learning, some researchers [12] proposed bilinear models.”

“Early work such as bilinear models by Tenen-baum and Freeman [36] factorizes style and content using matrix decomposition.”

“Considering the label information, Tenenbaum and Freeman [27] proposed bilinear models (BLMs), which extends an unsupervised CCA to generalized multiview discriminant analysis.”

“Tenenbaum et al. [14] first established a universal framework for solving dual-factor tasks using a bilinear model and applied it to classification, extrapolation, and translation tasks on facial datasets.”

“Besides, there are Partial Least Squares (PLS) [9] methods and Bilinear Model (BLM) [10] methods.”

“It classifies various object images by processing with a bilinear model [27].”

“able to organize transformations and the representations at the same time [4], usually employing some form of slowness.”

“In the bilinear factorization models proposed by Tenenbaum and Freeman [1], two factors which influence the observation data are called “style” and “content” respectively.”

“Examples include principal components analysis (PCA), active appearance/shape models (AAM/ASM) [9, 4], 3D morphable models (3DMM) [3], multi-linear models (MLM) [12, 13], non-linear manifolds [7], among others.”

“The bilinear framework proposed in [12] focuses on splitting up ‘style’ and ‘content’, and is designed to be applied to any two-factor task.”

“In Bilinear layers [44, 6, 30, 53] feature maps of different networks get bilinearly combined together to capture pairwise interactions.”

“Multi-modal Correlation Learning between images and texts is a hot research issue in computer vision and natural language processing in recent years, such as CCA [4], which study the general cross-modal correlation between images and text information and Bi-linear model [12].”

“Subsequently, bilinear pooling, a sophisticated technique for fusing multimodal data [16], is applied to synthesize the collective and individual modal representations.”

“BP models Tenenbaum and Freeman [32] have proven advantageous for fine-grained classification, particularly in tasks that solely involve vision Lin et al. [33].”

“Veeramachaneni and Nagy [19] have proposed a model by assuming Gaussian field class conditional distribution for field classification.”

“Bilinear and multilinear models [46,13] are multi-dimensional generalizations of singular value decomposition, where two different types of information can be represented in a bilinear basis through two sets of coefficients.”

“6 ISOMAP training: (a) the embedding of character features on a 3D manifold; (b) the dimensionality residual of dimension from 1 to 40 drawn character shape should be provided, and its pose should be similar to that of the source character, i.e., Nezha in this example.”

“The key observation is that since models wi, i = 1, 2, ...,M are, in most cases, highly related to each other, e.g., detectors of visually similar objects, it makes sense to model this correlation to produce more compact representations for these models, and to evaluate them more efficiently.”

“1) Data Alignment: We assume a varied set of training data D̄ = {D1, · · · ,DM} obtained from a robot under M conditions.”

“Index Terms—subspace learning; sparse representation; KSVD; rank symmetry I. INTRODUCTION In recent years, subspace-based approaches have been widely studied as a viable solution to the challenging problem of face recognition across lighting conditions, facial poses and facial expressions, etc.”