Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Open accessAbstract
Stochastic neurons and hard non-linearities can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic or non-smooth neurons? I.e., can we "back-propagate" through these stochastic neurons? We examine this question, existing approaches, and compare four families of solutions, applicable in different settings. One of them is the minimum variance unbiased gradient estimator for stochatic binary neurons (a special case of the REINFORCE algorithm). A second approach, introduced here, decomposes the operation of a binary stochastic neuron into a stochastic binary part and a smooth differentiable part, which approximates the expected effect of the pure stochatic binary neuron to first order. A third approach involves the injection of additive or multiplicative noise in a computational graph that is otherwise differentiable. A fourth approach heuristically copies the gradient with respect to the stochastic output directly as an estimator of the gradient with respect to the sigmoid argument (we call this the straight-through estimator). To explore a context where these estimators are useful, we consider a small-scale version of {\em conditional computation}, where sparse stochastic units form a distributed representation of gaters that can turn off in combinatorially many ways large chunks of the computation performed in the rest of the neural network. In this case, it is important that the gating units produce an actual 0 most of the time. The resulting sparsity can be potentially be exploited to greatly reduce the computational cost of large deep networks for which conditional computation would be useful.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A common fix approximates the quantization gradient with the Straight Through Estimator (STE).
“A popular approach to address this is to approximate the gradient of this operator by the so-called Straight Through Estimator (STE) [13].”
From A Survey of Quantization Methods for Efficient Neural Network Inference (2022) · cites this paper as [13]
STE approximates the gradient d w / dw by 1.
“STE approximates the gradient d ¯ w dw by 1.”
From PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation (2021) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 15
- Bengio, Y. (2013). Deep learning of representations: Looking forward. Technical Report arXiv:1305.0445, Universite de Montreal. · arXiv
- Bengio, Y., Courville, A., and Vincent, P. (2013). Unsupervised feature learning and deep learning: A review and new perspectives. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) .
- Dayan, P. (1990). Reinforcement comparison. In Connectionist Models: Proceedings of the 1990 Connectionist Summer School , San Mateo, CA.
- El Hihi, S. and Bengio, Y. (1996). Hierarchical recurrent neural networks for long-term dependencies. In NIPS 8 . MIT Press.
- Fiete, I. R. and Seung, H. S. (2006). Gradient learning in spiking neural networks by dynamic perturbations of conductances. Physical Review Letters , 97 (4).
- Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier neural networks. In AISTATS .
- Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout networks. In ICML'2013 .
- Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580. · arXiv
- Krizhevsky, A., Sutskever, I., and Hinton, G. (2012a). ImageNet classification with deep convolutional neural networks. In NIPS'2012 .
- Krizhevsky, A., Sutskever, I., and Hinton, G. (2012b). ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS'2012) .
- Nair, V. and Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In ICML'10 .
- Salakhutdinov, R. and Hinton, G. (2009). Semantic hashing. In International Journal of Approximate Reasoning .
- Spall, J. C. (1992). Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control , 37 , 332-341.
- Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In ICML 2008 .
- Weaver, L. and Tao, N. (2001). The optimal reward baseline for gradient-based reinforcement learning. In Proc. UAI'2001 , pages 538-545.
Cited by
Sorted by citations
A Survey of Quantization Methods for Efficient Neural Network Inference
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2022) · Low-Power Computer Vision · doi:10.1201/9781003162810-13 · Cited by 1,219 · see citing works
“We should first mention that [13] also proposes a stochastic neuron approach as an alternative to STE (this is briefly discussed in Section III-H).”
Dynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848 · see citing works
Learning Sparse Neural Networks through $L_0$ Regularization
Louizos, C., Welling, M., & Kingma, D. P. (2017) · arXiv · doi:10.48550/arxiv.1712.01312 · Cited by 628 · see citing works
A Lite Distributed Semantic Communication System for Internet of Things
Xie, H., & Qin, Z. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036968 · Cited by 497 · see citing works
SpotTune: Transfer Learning Through Adaptive Fine-Tuning
Guo, Y., Shi, H., Kumar, A., Grauman, K., Rosing, T., & Feris, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00494 · Cited by 506 · see citing works
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6409 · Cited by 437 · see citing works
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2006.16668 · Cited by 412 · see citing works
Low-bit Quantization of Neural Networks for Efficient Inference
Choukroun, Y., Kravchik, E., Yang, F., & Kisilev, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00363 · Cited by 404 · see citing works
Hierarchical Multiscale Recurrent Neural Networks
Chung, J., Ahn, S., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.01704 · Cited by 386 · see citing works
Self-citationLearning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss
Jung, S., Son, C., Lee, S., Son, J., Han, J.-J., Kwak, Y., Hwang, S. J., & Choi, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00448 · Cited by 385 · see citing works
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372 · see citing works
“In the VQ-VAE this is done using the straightthrough estimator [79], i.”
MethodInteger Quantization for Deep Learning Inference: Principles and Empirical Evaluation
Wu, H., Judd, P., Zhang, X., Isaev, M., & Micikevicius, P. (2020) · arXiv · doi:10.48550/arxiv.2004.09602 · Cited by 246 · see citing works
LSQ+: Improving low-bit quantization through learnable offsets and better initialization
Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., & Kwak, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00356 · Cited by 236 · see citing works
Movement pruning: Adaptive sparsity by fine-tuning
Sanh, V., Wolf, T., & Rush, A. M. (2020) · arXiv · doi:10.48550/arxiv.2005.07683 · Cited by 218 · see citing works
Unsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · see citing works
Bringing ai to edge: from deep learning’s perspective
Liu, D., Kong, H., Luo, X., Liu, W., & Subramaniam, R. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.04.141 · Cited by 171 · see citing works
“Some initial works on this topic are called conditional computation [172], [173], where the main objective of conditional computation is to enhance the capacity of a network while not significantly increasing the computational cost.”
BackgroundBinary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
Zhu, S., Dong, X., & Su, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00506 · Cited by 153 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141 · see citing works
Handling Vanishing Gradient Problem Using Artificial Derivative
Hu, Z., Zhang, J., & Ge, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3054915 · Cited by 109 · see citing works
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124 · see citing works
“To estimate returns that consider rewards beyond the prediction horizon, we compute bootstrapped λ -returns 29 that integrate the predicted rewards and the values.”
MethodSpeech Translation and the End-to-End Promise: Taking Stock of Where We Are
Sperber, M., & Paulik, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.661 · Cited by 93 · see citing works
PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation
Kim, J., Chang, S., & Kwak, N. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-248 · Cited by 85 · see citing works
“Dequantization step just brings the quantized value back to the original range by multiplying the step-size: w̄ = ŵ × Sw. (3) These quantization and dequantization processes are nondifferentiable, so we utilize a straight-through estimator (STE) [19] for backpropagation.”
MethodInfluentialBERTese: Learning to Speak to BERT
Haviv, A., Berant, J., & Globerson, A. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.316 · Cited by 84 · see citing works
Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
McKinstry, J. L., Esser, S. K., Appuswamy, R., Bablani, D., Arthur, J. V., Yildiz, I. B., & Modha, D. S. (2018) · arXiv · doi:10.48550/arxiv.1809.04191 · Cited by 74 · see citing works
AddNet: Deep Neural Networks Using FPGA-Optimized Multipliers
Faraone, J., Kumm, M., Hardieck, M., Zipf, P., Liu, X., Boland, D., & Leong, P. H. W. (2020) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2019.2939429 · Cited by 66 · see citing works
Kim, J., Kang, J.-K., & Kim, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3099075 · Cited by 50 · see citing works
PSAQ-ViT V2: Toward Accurate and General Data-Free Quantization for Vision Transformers
Li, Z., Chen, M., Xiao, J., & Gu, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3301007 · Cited by 41 · see citing works
“Math word problem generation with mathematical consistency and problem context constraints,”
Wang, Z., Lan, A., & Baraniuk, R. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.484 · Cited by 39 · see citing works
Design in the DARK: Learning Deep Generative Models for De Novo Protein Design
Moffat, L., Kandathil, S. M., & Jones, D. T. (2022) · doi:10.1101/2022.01.27.478087 · Cited by 25 · see citing works
FLightNNs: Lightweight Quantized Deep Neural Networks for Fast and Accurate Inference
Ding, R., Liu, Z., Chin, T.-W., Marculescu, D., & Blanton, R. D. (. (2019) · Proceedings of the 56th Annual Design Automation Conference 2019 · doi:10.1145/3316781.3317828 · Cited by 19 · see citing works
Mitigating Adversarial Attacks in Federated Learning with Trusted Execution Environments
Queyrut, S., Schiavoni, V., & Felber, P. (2023) · 2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs57875.2023.00069 · Cited by 18 · see citing works
Training Compact Neural Networks with Binary Weights and Low Precision Activations
Zhuang, B., Shen, C., & Reid, I. (2018) · arXiv · doi:10.48550/arxiv.1808.02631 · Cited by 10 · see citing works
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
Raposo, D., Ritter, S., Richards, B., Lillicrap, T., Humphreys, P. C., & Santoro, A. (2024) · arXiv · doi:10.48550/arxiv.2404.02258 · Cited by 8 · see citing works
“This terminology was introduced by Bengio (2013), and the concept was explored further over the next several years (Bengio et al., 2016, 2013; Cho and Bengio, 2014; Graves, 2016; Jernite et al., 2017; Wang et al., 2017).”
MethodColas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6 · see citing works
A Quantized Neural Network Library for Proper Implementation of Hardware Emulation
Kiyama, M., Nakahara, Y., Amagasaki, M., & Iida, M. (2019) · 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW) · doi:10.1109/candarw.2019.00032 · Cited by 5 · see citing works
Oflamaz, C. U., & Yalçın, M. E. (2025) · Electronics · doi:10.3390/electronics14183686 · Cited by 4 · see citing works
Intelligence plays dice: Stochasticity is essential for machine learning
Sabuncu, M. R. (2020) · arXiv · doi:10.48550/arxiv.2008.07496 · Cited by 3 · see citing works
ARB-LLM: Alternating Refined Binarizations for Large Language Models
Li, Z., Yan, X., Zhang, T., Qin, H., Xie, D., Tian, J., shi, Z., Kong, L., Zhang, Y., & Yang, X. (2024) · arXiv · doi:10.48550/arxiv.2410.03129 · Cited by 1 · see citing works
Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging
Zhang, J.-P., Pu, C.-F., Guo, M.-H., Cao, Y.-P., & Hu, S.-M. (2026) · arXiv · doi:10.48550/arxiv.2602.04805
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Lee, B., Kim, D., You, Y., & Kim, Y. (2025) · arXiv · doi:10.48550/arxiv.2506.13771
“For backpropagating through the non-differentiable sign( x ) function, we compared the Straight-Through Estimator (STE) [31] with our proposed SmoothSign technique.”
Influential
- Authors
- Bengio, Y., Léonard, N., & Courville, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2013
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1308.3432
- OpenAlex
- W2242818861
- Cited by
- 1,910
- References
- 0
- Citations by year
- 2026: 9 · 2025: 97 · 2024: 237 · 2023: 322 · 2022: 216 · 2021: 370 · 2020: 262 · 2019: 224 · 2018: 101 · 2017: 37 · 2016: 17 · 2015: 11 · 2014: 4 · 2013: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- stochastic neurons, gradient estimation, backpropagation, conditional computation, REINFORCE, straight-through estimator, noise injection, differentiable approximation, sparse gating, computational complexity reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
