The DOI matches its OpenAlex record; 68 of the 68 references it cites were found in a registry or an index.
R-Drop: Regularized Dropout for Neural Networks
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Bigger models always tend to have better performance, especially for various large-scale pre-trained models, e.g., Vision Transformer [11], Swin Transformer [40], GPT families [55, 56, 6], BERT [9], BART [35], Switch Transformers [14], etc.”
BackgroundDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Wu, Y., & He, K. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01198-w · Cited by 3,601
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4009 · Cited by 2,344
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · arXiv · doi:10.48550/arxiv.1804.08771 · Cited by 2,111
(2013) · Cited by 2,056
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844
Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00454 · Cited by 1,752
(2018) · International Conference on Learning Representations · Cited by 1,621
(1992) · Cited by 1,405
A theoretically grounded application of dropout in recurrent neural networks
Gal, Y., & Ghahramani, Z. (2016) · Cambridge University Engineering Department Publications Database · Cited by 1,089
Big Transfer (BiT): General Visual Representation Learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2019) · arXiv · doi:10.48550/arxiv.1912.11370 · Cited by 1,039
Big Transfer (BiT): General Visual Representation Learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_29 · Cited by 1,039 · cited 1×
“We utilize the same data preprocessing strategies with [11], where the details are given in [30].”
MethodSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00381 · Cited by 954
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Salimans, T., & Kingma, D. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07868 · Cited by 883
Scaling Neural Machine Translation
Ott, M., Edunov, S., Grangier, D., & Auli, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6301 · Cited by 544
The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training
Erhan, D., Manzagol, P.-A., Bengio, Y., Samy Bengio, A., & Vincent, P. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 525
(2021) · International Conference on Learning Representations · Cited by 449
Pointer Sentinel Mixture Models
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07843 · Cited by 427
Adaptive dropout for training deep neural networks
Ba, J. L., & Frey, B. J. (2013) · neural information processing systems · Cited by 419
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04770 · Cited by 407
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., & Zhou, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.217 · Cited by 374
(2013) · Cited by 326
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. X. (2020) · International Conference on Machine Learning · Cited by 325
Towards dropout training for convolutional neural networks
Wu, H., & Gu, X. (2015) · Neural Networks · doi:10.1016/j.neunet.2015.07.007 · Cited by 314
Variational Dropout Sparsifies Deep Neural Networks
Molchanov, D., Ashukha, A., & Vetrov, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.05369 · Cited by 311
Self-Distillation Amplifies Regularization in Hilbert SpacePreprint
Mobahi, H., Farajtabar, M., & Bartlett, P. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05715 · Cited by 307
A survey of regularization strategies for deep models
Moradi, R., Berangi, R., & Minaei, B. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09784-7 · Cited by 297
Survey of Dropout Methods for Deep Neural NetworksPreprint
Labach, A., Salehinejad, H., & Valaee, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.13310 · Cited by 287
Structured Bayesian Pruning via Log-Normal Multiplicative NoisePreprint
Neklyudov, K., Molchanov, D., Ashukha, A., & Vetrov, D. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1705.07283 · Cited by 268
Manual and automatic evaluation of summaries
Lin, C.-Y., & Hovy, E. (2002) · Proceedings of the ACL-02 Workshop on Automatic Summarization - · doi:10.3115/1118162.1118168 · Cited by 265
Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff PerspectivePreprint
Zhou, H., Song, L., Chen, J., Zhou, Y., Wang, G., Yuan, J., & Zhang, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.00650 · Cited by 234
Incorporating BERT into Neural Machine Translation
Zhu, J., Xia, Y., Wu, L., He, D., Qin, T., Zhou, W., Li, H., & Liu, T.-Y. (2020) · arXiv · doi:10.48550/arxiv.2002.06823 · Cited by 228
Self-citationHuang, L., Liu, X., Lang, B., Yu, A., Wang, Y., & Li, B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11768 · Cited by 179
Self-citationTowards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Allen-Zhu, Z., & Li, Y. (2020) · arXiv · doi:10.48550/arxiv.2012.09816 · Cited by 178
Shen, D., Zheng, M., Shen, Y., Qu, Y., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13818 · Cited by 162
Self-citationVery Deep Transformers for Neural Machine TranslationPreprint
Liu, X., Duh, K., Liu, L., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07772 · Cited by 148
Analyzing noise in autoencoders and deep networksPreprint
Poole, B., Sohl‐Dickstein, J., & Ganguli, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.1831 · Cited by 136
Self-Knowledge Distillation in Natural Language Processing
Hahn, S., & Choi, H. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_050 · Cited by 98
SIMPLIFYING NEURAL NETS BY DISCOVERING FLAT MINIMA
Hochreiter, S., & Schmidhuber, J. (1994) · Cited by 92
DropAttention: A Regularization Method for Fully-Connected Self-Attention NetworksPreprint
Lin, Z., Liu, P., Huang, L., Chen, J., Qiu, X., & Huang, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.11065 · Cited by 68
MUSE: Parallel Multi-Scale Attention for Sequence to Sequence LearningPreprint
Zhao, G., Sun, X., Xu, J., Zhang, Z., & Luo, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09483 · Cited by 66
Scheduled DropHead: A Regularization Method for Transformer Models
Zhou, W., Ge, T., Wei, F., Zhou, M., & Xu, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.178 · Cited by 56
Shakeout: A New Regularized Deep Neural Network Training Scheme
Kang, G., Li, J., & Tao, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10202 · Cited by 52
Self-citationDepth Growing for Neural Machine Translation
Wu, L., Wang, Y.-R., Xia, Y., Tian, F., Gao, F., Qin, T., Lai, J., & Liu, T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1558 · Cited by 50
Self-citationDropout with Expectation-linear RegularizationPreprint
Ma, X., Gao, Y., Hu, Z., Yu, Y., Deng, Y., & Hovy, E. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08017 · Cited by 44
Not All Attention Is All You NeedPreprint
Wu, H., Zhao, H., & Zhang, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04692 · Cited by 14
Self-citation
Cited by
ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation
Yang, L., Zhuo, W., Qi, L., Shi, Y., & Gao, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00423 · Cited by 518
Metadata
- Authors
- Liang, X., Wu, L., Li, J., Wang, Y., Meng, Q., Qin, T., Chen, W., Zhang, M., & Liu, T.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2021-06-28
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2106.14448
- OpenAlex
- W3174864715
- Cited by
- 297
- References
- 68
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.