CoChat
Sign Up
PreprintCited by 1,459Neural Networks and ApplicationsStochastic Gradient Optimization TechniquesMachine Learning and ELM

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Open access
Tancik, Matthew, Srinivasan, Pratul P., Mildenhall, Ben, Fridovich-Keil, Sara, Raghavan, Nithin, Singhal, Utkarsh, Ramamoorthi, Ravi, Barron, Jonathan T. +1 more·2020·arXiv
DOI 10.48550/arxiv.2006.10739arXiv 2006.10739
Open this paper in CoChat — free accountOpen PDF

Abstract

We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.10739

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The CNN-based Diffusion Policy performs poorly when the desired action sequence changes quickly and sharply through time, likely due to the inductive bias of temporal convolutions to prefer low-frequency signals.

    “However, it performs poorly when the desired action sequence changes quickly and sharply through time (such as velocity command action space), likely due to the inductive bias of temporal convolutions to prefer lowfrequency signals [49].”

    From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as [49]

    The Introduction (p. 1) and Background and Notation (p. 3) describe a low-frequency learning bias and difficulty learning high-frequency functions in standard coordinate-based MLPs. They do not establish that temporal convolutions have this bias or that it causes CNN-based Diffusion Policy to perform poorly on rapidly changing action sequences. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The CNN-based Diffusion Policy backbone performs poorly when the desired action sequence changes quickly and sharply through time, such as in velocity command action space.

    “However, it performs poorly when the desired action sequence changes quickly and sharply through time (such as velocity command action space), likely due to the inductive bias of temporal convolutions to prefer low-frequency signals Tancik et al. (2020).”

    From Diffusion policy: Visuomotor policy learning via action diffusion (2025) · cites this paper as Tancik et al. (2020)

    The Introduction (p. 1) and Background (p. 3) describe spectral bias: standard coordinate-based MLPs struggle to learn high-frequency functions. They do not establish that temporal convolutions have this bias or that a CNN-based Diffusion Policy performs poorly on rapidly changing velocity-command sequences. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The poor performance of the CNN-based backbone on high-frequency action changes is likely due to the inductive bias of temporal convolutions to prefer low-frequency signals.

    “However, it performs poorly when the desired action sequence changes quickly and sharply through time (such as velocity command action space), likely due to the inductive bias of temporal convolutions to prefer low-frequency signals Tancik et al. (2020).”

    From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as Tancik et al. (2020)

    The Introduction (p. 1) and Background (p. 3) report that standard coordinate-based MLPs have difficulty learning high-frequency functions because of spectral bias. The paper does not establish that temporal convolutions have the same bias or that it explains a CNN backbone’s performance on rapidly changing action sequences. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The time-series diffusion transformer minimizes over-smoothing effects of typical CNN-based models and achieves state-of-the-art performance on tasks that require high-frequency action changes and velocity control.

    “Time-series diffusion transformer. We propose a new transformer-based diffusion network that minimizes the over-smoothing effects of typical CNN-based models and achieves state-of-the-art performance on tasks that require high-frequency action changes and velocity control.”

    From Diffusion policy: Visuomotor policy learning via action diffusion (2025) · cites this paper as Tancik et al. (2020)

    The Introduction (pp. 1–2) and Section 5 (p. 6) discuss Fourier features helping coordinate-based MLPs learn high-frequency functions. Searches for the time-series diffusion transformer, CNN over-smoothing, action changes, and velocity control found no passage addressing the claimed model or its performance. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [1][1] Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super-resolution: Dataset and study. CVPR Workshops , 2017. -
  2. [2][2] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. ICML , 2019. -
  3. [3][3] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. arXiv preprint arXiv:2003.04560 , 2020. - · arXiv
  4. [4][4] Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. NeurIPS , 2019. -
  5. [5][5] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. NeurIPS , 2019. -
  6. [6][6] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. arXiv preprint arXiv:2002.02561 , 2020. - · arXiv
  7. [7][7] R. N. Bracewell. Strip integration in radio astronomy. Australian Journal of Physics , 1956. -
  8. [9][9] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. CVPR , 2019. -
  9. [10][10] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Neural articulated shape approximation. arXiv preprint arXiv:1912.03207 , 2019. - · arXiv
  10. [11][11] Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. ICLR , 2019. -
  11. [12][12] Kyle Genova, Forrester Cole, Aaron Sarna Daniel Vlasic, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. ICCV , 2019. -
  12. [13][13] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3D shape. CVPR , 2020. -
  13. [14][14] Reinhard Heckel and Mahdi Soltanolkotabi. Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation. arXiv preprint arXiv:2005.03991 , 2020. - · arXiv
  14. [15][15] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Learning a neural 3d texture space from 2d exemplars. CVPR , 2020. -
  15. [16][16] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and generalization in neural networks. NeurIPS , 2018. -
  16. [17][17] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3D scenes. CVPR , 2020. -
  17. [18][18] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321 , 2019. - · arXiv
  18. [19][19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR , 2015. -
  19. [20][20] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. NeurIPS , 2019. -
  20. [22][22] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. CVPR , 2020. -
  21. [23][23] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3D supervision. NeurIPS , 2019. -
  22. [24][24] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. CVPR , 2019. -
  23. [26][26] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. ICCV , 2019. -
  24. [27][27] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934 , 2020. - · arXiv
  25. [28][28] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CVPR , 2015. -
  26. [29][29] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. CVPR , 2020. -
  27. [30][30] Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha SohlDickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in Python. ICLR , 2020. -
  28. [31][31] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. ICCV , 2019. -
  29. [32][32] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. CVPR , 2019. -
  30. [33][33] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. ICML , 2019. -
  31. [34][34] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. NeurIPS , 2007. -
  32. [35][35] Shunsuke Saito, , Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV , 2019. -
  33. [36][36] Lawrence A. Shepp and Benjamin F. Logan. The Fourier reconstruction of a head section. IEEE Transactions on nuclear science , 1974. -
  34. [37][37] Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. NeurIPS , 2019. -
  35. [39][39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017. -
  36. [40][40] Martin J. Wainwright. Reproducing Kernel Hilbert Spaces , page 383-415. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. -
  37. [41][41] Ingo Wald, Sven Woop, Carsten Benthin, Gregory S Johnson, and Manfred Ernst. Embree: a kernel framework for efficient CPU ray tracing. ACM Transactions on Graphics (TOG) , 2014. -
  38. [42][42] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. NeurIPS , 2019. -
  39. [43][43] Greg Yang and Hadi Salman. A fine-grained spectral perspective on neural networks. arXiv preprint arXiv:1907.10599 , 2019. - · arXiv
  40. [44][44] Ellen D. Zhong, Tristan Bepler, Joseph H. Davis, and Bonnie Berger. Reconstructing continuous distributions of 3D protein structure from cryo-EM images. ICLR , 2020.

Cited by

Sorted by citations

Authors
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., & Ng, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.10739
arXiv
2006.10739
OpenAlex
W3036843665
Cited by
1,459
References
1
Citations by year
2026: 23 · 2025: 290 · 2024: 466 · 2023: 383 · 2022: 205 · 2021: 87 · 2020: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and ELM
Keywords
spectral bias, neural tangent kernel, multilayer perceptron, kernel bandwidth, 3D scene representation, neural radiance fields

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review