PhysFlow: Skin tone transfer for remote heart rate estimation through conditional normalizing flows

“We use PyTorch 2.1.2 [41] and train on a single NVIDIA RTX3060 us-ing sequences of 300 frames without overlap and AdamW optimizer with a learning rate of 0.0001.”

“The L perc is a VGG perceptual loss [22] adopted to retain high-frequency details between the reconstructed and original video, where φ j represents the perceptual function which outputs the activation function of the j th layer in the VGG network and C j , H j , W j are the dimensions of the…”

“Normalizing flows (NFs) [46, 50] are a class of generative neural networks that estimate an unknown complex data distribution, p ( z ) , from a known and simpler base probability distribution, p ( u ) , such as a normal distribution N ( µ , σ 2 ) .”

“…of a continuous random variable is equal to the trace of the Jacobin matrix, the total change in log density can be written as: To compute the gradients with respect to the parameters a black-box ordinary differential equation (ODE) solver is applied using an adjoint sensitivity method [ 45].”

“Previous works typically used the Fitzpatrick scale [17] to evaluate or anno-tate skin tone, categorizing it into six levels from I (lightest) to VI (darkest).”

“…generation of synthetic data can be excessively time-consuming, especially in the case of [33], and they may suffer from domain shift, where models trained on synthetic data fail to perform well on real datasets due to the domain gap between the datasets, requiring a domain adaptation stage [58].”

“The normalizing flow can be generalized from a discrete sequence to a continuous transformation [5, 18], parameterizing the dynamics of data transformation over time ∂ u τ ∂τ = g θ ( τ , c , u τ ) , where g θ is a neural network, u τ is the state at time τ , u ∼ N ( µ , σ 2 ) and c is the…”

“In this context, we propose a novel data augmentation approach leveraging normalizing flows [13, 14] to disentangle the skin tone information from the rest of the facial video content.”

“Since Takano et al. [51] and Verkruysse et al. [56] demonstrated remote HR measurement feasibility from facial videos, diverse methods for physiological data recovery have emerged.”

“In this context, we propose a novel data augmentation approach leveraging normalizing flows [13, 14] to disentangle the skin tone information from the rest of the facial video content.”

“Some focus on regions of interest, employing techniques like Blind Source Separation [25, 43, 44], Normalized Least Mean Squares [26], or self-adaptive matrix completion [55].”

“Also, we include some traditional methods such as POS [60] and CHROM [12] as baseline results.”

“Also, we include some traditional methods such as POS [60] and CHROM [12] as baseline results.”

“Implementation details: In all our experiments, we utilize the Mediapipe Face Mesh model [23, 32] to focus our analysis only on facial skin pixels.”

“Unlike previous approaches, some propose end-to-end models [6, 42, 62] to directly extract the rPPG signal from facial videos.”

“…To evaluate the HR estimation performance of the proposed model, we adopt the same metrics used in the literature, such as the mean absolute HR error (MAE), the root mean squared HR error (RMSE), the mean absolute percentage error (MAPE) and Pearson’s correlation coefficients R [26, 29, 30].”

“For example, datasets like VIPL-HR [35] or PURE [49] predominantly consist of Asian or Caucasian populations, respectively.”

“Normalizing flows (NFs) [46, 50] are a class of generative neural networks that estimate an unknown complex data distribution, p ( z ) , from a known and simpler base probability distribution, p ( u ) , such as a normal distribution N ( µ , σ 2 ) .”

“Some combine traditional methods with Convolutional Neural Networks (CNNs) to leverage advanced features [34, 36, 47].”

“Earlier works [2, 16, 59] noted the lowest signal-to-noise ratio and highest error rates for darker skin tones in assessments on private datasets.”

“Deep learning-based methods [24, 31, 39, 42, 48, 63] have surpassed conventional techniques, achieving state-of-the-art performance in estimating vital signs from facial videos.”

“Some focus on regions of interest, employing techniques like Blind Source Separation [25, 43, 44], Normalized Least Mean Squares [26], or self-adaptive matrix completion [55].”

“Since Takano et al. [51] and Verkruysse et al. [56] demonstrated remote HR measurement feasibility from facial videos, diverse methods for physiological data recovery have emerged.”

“Implementation details: In all our experiments, we utilize the Mediapipe Face Mesh model [23, 32] to focus our analysis only on facial skin pixels.”

“Some combine traditional methods with Convolutional Neural Networks (CNNs) to leverage advanced features [34, 36, 47].”

“For example, datasets like VIPL-HR [35] or PURE [49] predominantly consist of Asian or Caucasian populations, respectively.”

“…types (V and VI), certain traditional methods perform comparably or even outperform some deep learning models, indicating skin tone-biased performance Hand-crafted methods Deep learningmethods Method POS [60] CHROM [12] PhysNet[62] EfficientPhys[30] DPMN+TDM [9] Skin attributed to dataset imbalance.”

“Deep learning-based methods [24, 31, 39, 42, 48, 63] have surpassed conventional techniques, achieving state-of-the-art performance in estimating vital signs from facial videos.”

“…types (V and VI), certain traditional methods perform comparably or even outperform some deep learning models, indicating skin tone-biased performance Hand-crafted methods Deep learningmethods Method POS [60] CHROM [12] PhysNet[62] EfficientPhys[30] DPMN+TDM [9] Skin attributed to dataset imbalance.”

“Other recent works [24, 29] explore unsupervised approaches using meta-learning, enhancing generalization in out-of-distribution cases.”

“Some combine traditional methods with Convolutional Neural Networks (CNNs) to leverage advanced features [34, 36, 47].”

“Following [28], during the 3D AE training, we add a small Gaussian noise of 0.05 to the latent space to prevent the latent distribution from collapsing to single delta peaks.”

“Transformer-based models like Physformer [64] and RADI-ANT [19] have gained attention for leveraging long-range spatiotemporal features, although they currently lack optimization for mobile deployment.”

“Subsequently, Dasari et al. [11] conducted an extensive study analyzing estimation biases of camera PPG methods across diverse demographics.”

“In recent years, the interest on camera-based measurement of physiological signals has undergone substantial growth caused by its potential applicability in clinical [21] and human-computer interaction applications [4, 37].”

“For this reason, we followed the same protocol in [52] to exclude from our analysis other factors such as exercise or lighting conditions.”

“Finally, Ba et al. [3] proposed a two-stage optimization using external pre-training [65] to transfer skin tone content, which is an interesting possibility to balance skin-tone distributions as long as external data with the underrepresented skin tones are available for pre-training.”

“The PPG ground truth is pre-processed following [10] to denoise the raw PPG signal, which facilitates a better model convergence during the training.”

“Firstly, across all methods, there is a noticeable decline in heart rate performance with darker skin tones, consistent with findings in previous studies [2, 15, 38].”

“Transformer-based models like Physformer [64] and RADI-ANT [19] have gained attention for leveraging long-range spatiotemporal features, although they currently lack optimization for mobile deployment.”

“Deep learning-based methods [24, 31, 39, 42, 48, 63] have surpassed conventional techniques, achieving state-of-the-art performance in estimating vital signs from facial videos.”

“In recent years, the interest on camera-based measurement of physiological signals has undergone substantial growth caused by its potential applicability in clinical [21] and human-computer interaction applications [4, 37].”

“Primarily, the generation of synthetic data can be excessively time-consuming, especially in the case of [33], and they may suffer from domain shift, where models trained on synthetic data fail to perform well on real datasets due to the domain gap between the datasets, requiring a domain…”

“…To evaluate the HR estimation performance of the proposed model, we adopt the same metrics used in the literature, such as the mean absolute HR error (MAE), the root mean squared HR error (RMSE), the mean absolute percentage error (MAPE) and Pearson’s correlation coefficients R [26, 29, 30].”

“In recent years, the interest on camera-based measurement of physiological signals has undergone substantial growth caused by its potential applicability in clinical [21] and human-computer interaction applications [4, 37].”

“Firstly, Vilesov et al. [57] proposed a method to mitigate bias across skin tones using multi-modal fusion, combining information from an RGB camera and 77 GHz radar.”

“For example, McDuff et al. [33] used Blender to create synthetic avatars aligned with cardiac and respiratory signals across various scenarios, while Wang et al. [61] employed a 3D morphable face model to generate synthetic facial videos.”

“…the minimization of the rPPG signal concerning the ground truth PPG from the original and also from the generated video, which can be written as: To compute L phys , in our experiments, we use the Negative Pearson correlation [62] and TALOS [7] losses depending on the selected rPPG approach.”

“Based on these assumptions, we model our skin tone transfer employing a bi-dimensional representation [53] expressed in CIELAB color space through two components: luminance and hue.”

“Earlier works [2, 16, 59] noted the lowest signal-to-noise ratio and highest error rates for darker skin tones in assessments on private datasets.”

“Firstly, across all methods, there is a noticeable decline in heart rate performance with darker skin tones, consistent with findings in previous studies [2, 15, 38].”

“Recent studies have proposed different data augmentation approaches considering factors like motion [40] or data scarcity [20, 54], but without targeting skin tone bias.”

“Recent studies have proposed different data augmentation approaches considering factors like motion [40] or data scarcity [20, 54], but without targeting skin tone bias.”

“Finally, Ba et al. [3] proposed a two-stage optimization using external pre-training [65] to transfer skin tone content, which is an interesting possibility to balance skin-tone distributions as long as external data with the underrepresented skin tones are available for pre-training.”

“Inspired by several flow-based generative models [27, 28] in different computer vision tasks, we employ a conventional auto-encoder model, which is trained to encode and decode the high-dimensional information into low-dimensional feature embedding, independently of any disentanglement.”

“Recent studies have proposed different data augmentation approaches considering factors like motion [40] or data scarcity [20, 54], but without targeting skin tone bias.”

“To evaluate PhysFlow performance, we use three deep-learning models: PhysNet [62], Effi-cientPhys [30], and DPMN+TDM [9].”

“Finally, the predicted rPPG signal is filtered using a band-pass filter with cut-offs of 0.66-3Hz while the heart rate is calculated using Chirp-Z Transform (CZT) [8].”

“People with darker skin tones might experience poorer heart rate measurement accuracy because of their skin tones potentially skewing data used to develop research findings and clinical recommendations [33].”