The DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Effectively Unbiased FID and Inception Score and Where to Find Them
Open accessAbstract
This paper shows that two commonly used evaluation metrics for generative models, the Fréchet Inception Distance (FID) and the Inception Score (IS), are biased -- the expected value of the score computed for a finite sample set is not the true value of the score. Worse, the paper shows that the bias term depends on the particular model being evaluated, so model A may get a better score than model B simply because model A's bias term is smaller. This effect cannot be fixed by evaluating at a fixed number of samples. This means all comparisons using FID or IS as currently computed are unreliable. We then show how to extrapolate the score to obtain an effectively bias-free estimate of scores computed with an infinite number of samples, which we term FID Infinity and IS Infinity. In turn, this effectively bias-free estimate requires good estimates of scores with a finite number of samples. We show that using Quasi-Monte Carlo integration notably improves estimates of FID and IS for finite sample sets. Our extrapolated scores are simple, drop-in replacements for the finite sample scores. Additionally, we show that using low discrepancy sequence in GAN training offers small improvements in the resulting generator.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“FID is crucial for gauging the similarity in style and content between two image sets, where a lower FID score signifies greater resemblance.”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717
Spectral Normalization for Generative Adversarial Networks
Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018) · arXiv · doi:10.48550/arxiv.1802.05957 · Cited by 3,438
A Note on the Generation of Random Normal Deviates
Box, G. E. P., & Muller, M. E. (1958) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177706645 · Cited by 3,430
On the distribution of points in a cube and the approximate evaluation of integrals
Sobol', I. M. (1967) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(67)90144-9 · Cited by 2,975
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
Stochastic Simulation: Algorithms and Analysis
Asmussen, S., & Glynn, P. W. (2007) · Applications of mathematics · doi:10.1007/978-0-387-69033-9 · Cited by 1,423
Options: A Monte Carlo approach
Boyle, P. P. (1977) · Journal of Financial Economics · doi:10.1016/0304-405x(77)90005-8 · Cited by 1,341
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2856256 · Cited by 1,267
Demystifying MMD GANsPreprint
Bińkowski, M., Sutherland, D. J., Arbel, M., & Gretton, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01401 · Cited by 1,083
Are GANs Created Equal? A Large-Scale StudyPreprint
Lučić, M., Kurach, K., Michalski, M., Gelly, S., & Bousquet, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10337 · Cited by 865
Barratt, S., & Sharma, R. (2018) · arXiv · doi:10.48550/arxiv.1801.01973 · Cited by 732
Few-Shot Unsupervised Image-to-Image Translation
Liu, M.-Y., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., & Kautz, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01065 · Cited by 687
Global Illumination using Photon Maps
Jensen, H. W. (1996) · Eurographics · doi:10.1007/978-3-7091-7484-5_3 · Cited by 624
High-dimensional integration: The quasi-Monte Carlo way
Dick, J., Kuo, F. Y., & Sloan, I. H. (2013) · Acta Numerica · doi:10.1017/s0962492913000044 · Cited by 615
Algorithm 247: Radical-inverse quasi-random point sequence
Halton, J. H. (1964) · Communications of the ACM · doi:10.1145/355588.365104 · Cited by 609
Wasserstein Auto-EncodersPreprint
Tolstikhin, I., Bousquet, O., Gelly, S., & Schölkopf, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01558 · Cited by 508
Randomly Permuted (t,m,s)-Nets and (t, s)-Sequences
Owen, A. B. (1995) · Lecture notes in statistics · doi:10.1007/978-1-4612-2552-2_19 · Cited by 407
Faster Valuation of Financial Derivatives
Paskov, S. H., & Traub, J. F. (1995) · The Journal of Portfolio Management · doi:10.3905/jpm.1995.409541 · Cited by 369
An empirical study on evaluation metrics of generative adversarial networksPreprint
Xu, Q., Huang, G. M., Yuan, Y., Guo, C., Sun, Y., Wu, F., & Weinberger, K. Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07755 · Cited by 310
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative ModelsPreprint
Zhou, S., Gordon, M., Krishna, R., Narcomey, A., Fei-Fei, L., & Bernstein, M. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01121 · Cited by 135
Few-Shot Unsupervised Image-to-Image TranslationPreprint
Liu, M.-Y., Huang, X. C., Mallya, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.01723 · Cited by 79
Bias Correction of Learned Generative Models using Likelihood-Free Importance WeightingPreprint
Grover, A., Song, J., Agarwal, A., Tran, K., Kapoor, A., Horvitz, E., & Ermon, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09531 · Cited by 75
BoTorch: Programmable Bayesian Optimization in PyTorch.
Balandat, M., Karrer, B., Jiang, D., Daulton, S., Letham, B., Wilson, A. G., & Bakshy, E. (2019) · Cited by 60
Dick, J. (2011) · The Annals of Statistics · doi:10.1214/11-aos880 · Cited by 49
Seeing is Not Necessarily Believing: Limitations of BigGANs for Data Augmentation
Ravuri, S. V., & Vinyals, O. (2019) · Cited by 44
Generating low-discrepancy sequences from the normal distribution: Box–Muller or inverse transform?
Ökten, G., & Göncü, A. (2010) · Mathematical and Computer Modelling · doi:10.1016/j.mcm.2010.12.011 · Cited by 43
Randomization of lattice rules for numerical multiple integration
Joe, S. (1990) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(90)90172-v · Cited by 22
Randomization of lattice rules for numerical multiple integration
JoeS. (1990) · Journal of Computational and Applied Mathematics · doi:10.5555/90089.90104 · Cited by 21
Quasi-Monte Carlo Variational InferencePreprint
Buchholz, A., Wenzel, F., & Mandt, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.01604 · Cited by 13
Flagged first, then by citations
1 of the 43 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- smolyak, 1963, Quadrature and interpolation formulas for tensor products of certain classes of functions, Doklady Akademii Nauk, 148, 1042
- goodfellow, 0, Generative adversarial nets, Advances in neural information processing systems, 2672
- owen, 0, Quasi-monte carlo sampling
- zhou, 2019, Hype Human eye perceptual evaluation of generative models
Cited by
Sorted by citations
Pros and cons of GAN evaluation measures: New developments
Borji, A. (2021) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2021.103329 · Cited by 381
On Aliased Resizing and Surprising Subtleties in GAN Evaluation
Parmar, G., Zhang, R., & Zhu, J.-Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01112 · Cited by 358
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Jayasumana, S., Ramalingam, S., Veit, A., Gläsner, D., Chakrabarti, A., & Kumar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00889 · Cited by 320
Islam, S., Aziz, M. T., Nabil, H. R., Jim, J. R., Mridha, M. F., Kabir, M. M., Asai, N., & Shin, J. (2024) · IEEE Access · doi:10.1109/access.2024.3370848 · Cited by 198
Lumiere: A Space-Time Diffusion Model for Video Generation
Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Liu, G., Raj, A., et al. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687614 · Cited by 215
Metadata
- Authors
- Chong, M. J., & Forsyth, D.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 6069-6078
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2985516053
- Cited by
- 242
- References
- 43
- Citations by year
- 2026: 26 · 2025: 59 · 2024: 69 · 2023: 43 · 2022: 24 · 2021: 18 · 2020: 3
- FWCI
- 6.78
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Keywords
- Fréchet Inception Distance, Inception Score, generative adversarial networks, quasi-Monte Carlo integration, low-discrepancy sequences, GAN training, generative model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
