Cited by 1,011Neural Networks and ApplicationsTopic ModelingImage and Signal Denoising Methods

A Connection Between Score Matching and Denoising Autoencoders

Vincent, Pascal·2011·Neural Computation

Abstract

Denoising autoencoders have been previously shown to be competitive alternatives to restricted Boltzmann machines for unsupervised pretraining of each layer of a deep architecture. We show that a simple denoising autoencoder training criterion is equivalent to matching the score (with respect to the data) of a specific energy-based model to that of a nonparametric Parzen density estimator of the data. This yields several useful insights. It defines a proper probabilistic model for the denoising autoencoder technique, which makes it in principle possible to sample from them or rank examples by their energy. It suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives. It justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models.

Integrity signals

Registry

Crossref · 10.1162/neco_a_00142

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • An energy-based model trained with denoising score matching can effectively purify attacked images within just a few steps.

    “Adaptive Denoising Purification (ADP) [273] demonstrates that an EBM trained with denoising score matching [240] can effectively purify attacked images within just a few steps.”

Cites

Flagged first, then by citations

2 of the 29 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Vincent, P.
Affiliations
Université de Montréal (CA)
Corresponding author
Vincent, Pascal
Venue
Neural Computation
Publisher
MIT Press
Published
2011-07
Volume
23
Issue
7
Pages
1661-1674
ISSN
  • 0899-7667 (ISSN-L)
  • 1530-888X
OpenAlex
W2013035813
Cited by
1,011
References
29
Citations by year
2026: 131 · 2025: 254 · 2024: 198 · 2023: 126 · 2022: 46 · 2021: 59 · 2020: 52 · 2019: 29 · 2018: 20 · 2017: 22 · 2016: 20 · 2015: 14 · 2014: 9 · 2013: 14 · 2012: 9
FWCI
11.49
Type
journal article
Language
en
Open access
closed
Locations
  • Neural Computation (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
Keywords
score matching, denoising autoencoder, energy-based models, unsupervised pre-training, restricted Boltzmann machine, probabilistic models, deep learning architectures
MeSH terms
Computer Simulation, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review