CoChat
Sign Up
Cited by 17Anomaly Detection Techniques and ApplicationsIndustrial Vision Systems and Defect DetectionAdversarial Robustness in Machine Learning

Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations

Open access
Rippel, Oliver, Chavan, Arnav, Lei, Chucai, Merhof, Dorit·2022·Proceedings of the 2nd International Conference on Image Processing and Vision Engineering
DOI 10.5220/0011063900003209
Open this paper in CoChat — free account

Abstract

Current state-of-the-art anomaly detection (AD) methods exploit the powerful representations yielded by large-scale ImageNet training. However, catastrophic forgetting prevents the successful fine-tuning of pre-trained representations on new datasets in the semi-supervised setting, and representations are therefore commonly fixed. In our work, we propose a new method to overcome catastrophic forgetting and thus successfully fine-tune pre-trained representations for AD in the transfer learning setting. Specifically, we induce a multivariate Gaussian distribution for the normal class based on the linkage between generative and discriminative modeling, and use the Mahalanobis distance of normal images to the estimated distribution as training objective. We additionally propose to use augmentations commonly employed for vicinal risk minimization in a validation scheme to detect onset of catastrophic forgetting. Extensive evaluations on the public MVTec dataset reveal that a new state of the art is achieved by our method in the AD task while simultaneously achieving anomaly segmentation performance comparable to prior state of the art. Further, ablation studies demonstrate the importance of the induced Gaussian distribution as well as the robustness of the proposed fine-tuning scheme with respect to the choice of augmentations.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes fine-tuning ImageNet-pretrained representations for semi-supervised anomaly detection by inducing a multivariate Gaussian for normal data and minimizing Mahalanobis distance, with early stopping via vicinal-risk-minimization augmentations to detect catastrophic forgetting. On MVTec, this reaches 97.1 ± 0.8 AUROC for anomaly detection, improving over the frozen-feature baseline of 96.3 ± 1.1.

Key findings

  • Fine-tuning with the proposed Gaussian/Mahalanobis objective improves MVTec anomaly detection AUROC from 96.3 ± 1.1 (frozen features) to 97.1 ± 0.8, reported as a new state of the art.
  • Deep SVDD and FCDD objectives are outperformed by the frozen baseline for AD (90.5 ± 2.7 and 87.0 ± 3.6 vs. 96.3 ± 1.1), though fine-tuning with the proposed early stopping improves Deep SVDD from 86.7 ± 3.5 to 90.5 ± 2.7.
  • The proposed early stopping criterion is the only tested method that improves results: vicinity sampling via VRM gives 97.1 ± 0.9 AUROC, versus 96.3 ± 1.2 for fixed epochs, 95.4 ± 1.2 for validation loss, and 96.3 ± 1.1 for no training.
  • For anomaly segmentation, the method reaches 96.5 ± 0.8 pixel-wise AUROC and 88.7 ± 1.7 PRO, comparable in AUROC to prior state of the art but slightly lower in PRO.

Methods

  • The method fits a multivariate Gaussian to normal-class features from a pretrained network, using empirical mean and shrinkage-estimated covariance, and fine-tunes by minimizing the Mahalanobis distance of normal images to this distribution; no outlier exposure is used.
  • It applies the Mahalanobis distance to intermediate feature maps, modeling the mean independently per spatial location and tying covariance across locations, then uses max aggregation for image-level anomaly scores and bilinear upsampling for segmentation.
  • Evaluation uses the public MVTec dataset (15 categories, 5354 images), a 5-fold split of each category’s training set, EfficientNet-B4, batch size 8, Adam with learning rate 1 × 10⁻⁶, frozen BatchNorm statistics, and early stopping based on AUROC between normal and VRM-augmented validation samples.

Limitations

  • The paper states that the approach fails to segment high-level, semantic anomalies such as “cable swap” or “scratches,” and that evaluations are limited to MVTec; it also notes the method cannot handle multimodal normal data distributions.
  • The paper states that validation with subtle augmentations may still introduce bias, though argued to be less than outlier exposure, and that FCDD should be trained without synthetic segmentation masks in future work to fully validate this claim.
  • Evident from the text: the best-performing local Gaussian model has a memory footprint increased by a factor of 10 and achieves only negligible gains from fine-tuning, making it infeasible in practice; anomaly segmentation PRO remains slightly below prior state of the art.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.5220/0011063900003209

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 74 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations · p. 1

    The proposed method achieves anomaly segmentation performance comparable to prior state of the art.

    “Extensive evaluations on the public MVTec dataset reveal that a new state of the art is achieved by our method in the AD task while simultaneously achieving anomaly segmentation performance comparable to prior state of the art.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.
  • Findingown data
    § Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations · p. 1

    The proposed method achieves a new state of the art in AD on the public MVTec dataset.

    “Extensive evaluations on the public MVTec dataset reveal that a new state of the art is achieved by our method in the AD task while simultaneously achieving anomaly segmentation performance comparable to prior state of the art.”
    • Also stated in: § 4.2 Anomaly Detection · p. 6 · plain
      “Analyzing results (Table 1), it can be seen that our proposed transfer learning scheme improves over the baseline (frozen features), increasing AD performance from 96 . 3 ± 1 . 1 to 97 . 1 ± 0 . 8 AUROC and setting a new state-of-the-art in AD on the public MVTec dataset.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.
  • Findingown data
    § Transfer Learning Gaussian Anomaly Detection by Fine-tuning Representations · p. 1

    Ablation studies demonstrate the importance of the induced Gaussian distribution.

    “Further, ablation studies demonstrate the importance of the induced Gaussian distribution as well as the robustness of the proposed fine-tuning scheme with respect to the choice of augmentations.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.
  • Findingown data
    § 5 DISCUSSION · p. 8

    The proposed fine-tuning improves anomaly segmentation (AS) performance.

    “improves AS performance (Table 1, Table 5 and Table 6)”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Table 5

      Not checked in the free run.

    • cites Table 6

      Not checked in the free run.

    • The authors state this as an assumption: Normal data in pre-trained deep features can be modeled by a Gaussian distribution.
  • Findingown data
    § 5 DISCUSSION · p. 8

    The proposed fine-tuning based on the Gaussian assumption achieves a new state-of-the-art in anomaly detection on the public MVTec dataset.

    “Evaluations on the public MVTec dataset have revealed that the proposed fine-tuning based on the Gaussian assumption achieves a new state-of-the-art”
    • Also stated in: § 6 CONCLUSION · p. 10 · boosted
      “In our work, we have demonstrated that fine-tuning of pre-trained feature representations for transfer learning AD is possible using a strong Gaussian prior, achieving a new state-of-the-art in AD on the public MVTec dataset.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Table 1

      Not checked in the free run.

    • The authors state this as an assumption: Normal data in pre-trained deep features can be modeled by a Gaussian distribution.
  • May combine two claims. 9
  • Classified as probably not a substantial claim. 7

  • May combine two claims.

    Findingown data
    § 4.2 Anomaly Detection · p. 6

    The Gaussian assumption is important, since SVDD and FCDD objectives are already outperformed by the baseline where no fine-tuning occurs (90.5 ± 2.7 and 87.0 ± 3.6 vs. 96.3 ± 1.1).

    “Furthermore, it can be seen that the Gaussian assumption is important, since SVDD and FCDD objectives are already outperformed by our baseline where no fine-tuning of feature representations occurs (90 . 5 ± 2 . 7 and 87 . 0 ± 3 . 6 vs. 96 . 3 ± 1 . 1).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.

  • May combine two claims.

    Findingnone given
    § 3 TRANSFER LEARNING GAUSSIAN ANOMALY DETECTION · p. 4

    Fixing the Gaussian parameters to the empirical mean and shrinkage covariance yields a stronger bias and was shown to prevent catastrophic forgetting in prior experiments.

    “This yields a stronger bias and was shown to prevent catastrophic forgetting in prior experiments.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.

  • May combine two claims.

    Findingown data
    § 4.2 Anomaly Detection · p. 6

    The proposed transfer learning scheme improves over the baseline (frozen features), increasing AD performance from 96.3 ± 1.1 to 97.1 ± 0.8 AUROC.

    “Analyzing results (Table 1), it can be seen that our proposed transfer learning scheme improves over the baseline (frozen features), increasing AD performance from 96 . 3 ± 1 . 1 to 97 . 1 ± 0 . 8 AUROC and setting a new state-of-the-art in AD on the public MVTec dataset.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.

  • May combine two claims.

    Findingown data
    § 4.2.1 Ablation studies · p. 7

    The local Gaussian model has the highest base AD performance and only negligible gains are achieved by fine-tuning.

    “Notably, it also has the highest base AD performance and only negligible gains are achieved by fine-tuning.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Table 4

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § 3 TRANSFER LEARNING GAUSSIAN ANOMALY DETECTION · p. 4

    The induced multivariate Gaussian prior imposes a lesser inductive bias than the Deep SVDD objective and allows for more flexible distributions.

    “Our induced multivariate Gaussian prior thus actually imposes a lesser inductive bias than the Deep SVDD objective, and allows for more flexible distributions.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    AD/AS algorithms focus on the semi-supervised setting, where exclusively normal data is used to establish a model of normality.

    “As a consequence, AD/AS algorithms focus on the semi-supervised setting, where exclusively normal data is used to establish a model of normality (Bergmann et al., 2021; Schlegl et al., 2019; Ruff et al., 2018).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Bergmann et al., 2021

      Not checked in the free run.

    • cites Schlegl et al., 2019

      Not checked in the free run.

    • cites Ruff et al., 2018

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 1 INTRODUCTION · p. 1

    AD/AS datasets are heavily imbalanced, often containing only few anomalies for model verification and testing.

    “Together, these characteristics result in AD/AS datasets that are heavily imbalanced, often containing only few anomalies for model verification and testing.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation
    § 3 TRANSFER LEARNING GAUSSIAN ANOMALY DETECTION · p. 4

    Bilinear interpolation is free of hyperparameters and thus more robust while simultaneously offering competitive results.

    “Bilinear interpolation is free of hyperparameters and thus more robust while simultaneously offering competitive results (selection of the Gaussian interpolation kernel was shown to have a strong effect on AS performance in (Liznerski et al., 2021)).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Liznerski et al., 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 Fine-tuning representations · p. 3

    Empirical gains from OE can only be guaranteed when anomalies used for OE differ strongly from the normal class.

    “They further show that empirical gains can only be guaranteed when anomalies are used for OE that differ strongly from the normal class instead.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ye et al., 2021

      Not checked in the free run.

  • May combine two claims.

    Premiseown data
    § 4.2 Anomaly Detection · p. 5

    The feature representations are already discriminative as is and only need to be tuned slightly.

    “The small learning rate is motivated by the fact that the feature representations are already discriminative as is (refer e.g. to the fixed baseline in Table 1), and only need to be tuned slightly.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3 TRANSFER LEARNING GAUSSIAN ANOMALY DETECTION · p. 3

    The same finding holds for the transfer learning setting in AD.

    “Independent investigations into the same phenomenon for AD by Rippel et al. (Rippel et al., 2021c) have revealed the same finding for the transfer learning setting.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rippel et al., 2021c

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 2.2 Fine-tuning representations · p. 3

    Labeled anomalies differing subtly from the normal class have a large impact in OE, possibly degrading performance on unseen anomaly types that are also similar to the normal class.

    “Specifically, Ye et al. (Ye et al., 2021) show that labeled anomalies differing subtly from the normal class have a large impact in OE, possibly degrading performance on unseen anomaly types that are also similar to the normal class.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ye et al., 2021

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § 4.1 Dataset and Evaluation metrics · p. 5

    MVTec is more challenging and indicative of real-world AD/AS performance.

    “Therefore, MVTec is more challenging and indicative of real-world AD/AS performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 INTRODUCTION · p. 1

    Fine-tuning pre-trained representations on the dataset at hand offers the potential of additional performance improvements.

    “Fine-tuning these representations on the dataset at hand now offers the potential of additional performance improvements.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5.1 Limitations · p. 9

    Object detection or segmentation networks generate features that maintain stronger spatial acuity.

    “which have been shown to generate features that maintain stronger spatial acuity (Li et al., 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al., 2019

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5.1 Limitations · p. 9

    Features from object detection or segmentation networks improve AS performance when used as the basis for transfer learning AS.

    “They have furthermore been shown to improve AS performance when used as the basis for transfer learning AS (Rippel and Merhof, 2021).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rippel and Merhof, 2021

      Not checked in the free run.

16 of 16

  • The authors state this as an assumption.

    The normal data distribution in the pre-trained feature space can be modeled as a multivariate Gaussian.

    “In our work, we propose to fine-tune the representations of pre-trained networks based on the Gaussian assumption.”
    “A motivation behind the seemingly overly simplistic Gaussian assumption can be inferred from Lee et al. (Lee et al., 2018).”

  • The paper relies on this without support.

    The normal class can be compactly described in high-dimensional representations.

    “Due to the absence of anomalies, fine-tuning approaches base their learning objective on the concentration assumption instead, i.e. they try to find a compact description of the normal class in highdimensional representations.”

  • The authors state this as an assumption.

    Normal data in pre-trained deep features can be modeled by a Gaussian distribution.

    “proposed fine-tuning based on the Gaussian assumption”
    “using a strong Gaussian prior”

  • The paper relies on this without support.

    Catastrophic forgetting typically hinders transfer learning in anomaly detection.

    “which typically hinders transfer learning in AD”

  • The paper relies on this without support.

    The bias induced by sampling the anomaly distribution for outlier exposure (OE) is likely higher than the bias induced by using subtle augmentations for model validation.

    “avoiding the bias likely induced by sampling the anomaly distribution for OE”
    “argued that this bias would most likely be lower”
5 of 5

Cites

Flagged first, then by citations

1 of the 61 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. Akc ¸ay, S., Atapour-Abarghouei, A., and Breckon, T. P. (2018). GANomaly: Semi-supervised anomaly detection via adversarial training. In Asian Conference on Computer Vision , pages 622-637. Springer.
  2. Ferri, C., Hern´ andez-Orallo, J., and Flach, P. A. (2011). A coherent interpretation of AUC as a measure of aggregated classification performance. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) , pages 657-664.
  3. Li, C.-L., Sohn, K., Yoon, J., and Pfister, T. (2021). Cutpaste: Self-supervised learning for anomaly detection and localization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9659-9669.
  4. 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings .

Cited by

Sorted by citations

Related

Authors
Rippel, O., Chavan, A., Lei, C., & Merhof, D.
Affiliations
  • RWTH Aachen University (DE)
  • Indian Institute of Technology Dhanbad (IN)
Corresponding author
Rippel, Oliver
ORCID
Venue
Proceedings of the 2nd International Conference on Image Processing and Vision Engineering
Publisher
SCITEPRESS - Science and Technology Publications
Published
2022
Pages
45-56
DOI
10.5220/0011063900003209
OpenAlex
W3191740548
Cited by
17
References
61
Citations by year
2025: 3 · 2024: 7 · 2023: 6 · 2022: 1
FWCI
2.35
Type
conference paper
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • International Conference on Image Processing and Vision Engineering (IMPROVE) (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
  • Physical Sciences › Engineering › Industrial and Manufacturing Engineering › Industrial Vision Systems and Defect Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
anomaly detection, catastrophic forgetting, transfer learning, multivariate normal distribution, Mahalanobis distance, semi-supervised learning, data augmentation, anomaly segmentation, MVTec

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review