Maxout Networks
Open accessAbstract
We consider the problem of designing models to leverage a recently introduced approximate model averaging technique called dropout. We define a simple new model called maxout (so named because its output is the max of a set of inputs, and because it is a natural companion to dropout) designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. We empirically verify that the model successfully accomplishes both of these tasks. We use maxout and dropout to demonstrate state of the art classification performance on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Implementing the arithmetic mean in the case of early fusion and non-linear shared layers is not straightforward.
“Unfortunately, implementing the arithmetic mean in the case of early fusion and non-linear shared layers is not straightforward [51].”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [51]
Maxout Networks (p. 2) says it is not obvious how to compute the arithmetic mean of predictions from the exponentially many models trained by dropout. Section 7 (pp. 6–7) discusses approximate model averaging in deep networks, but does not address early fusion or establish the claim specifically for non-linear shared layers. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 23
- Bastien, Fr´ ed´ eric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
- Bergstra, James, Breuleux, Olivier, Bastien, Fr´ ed´ eric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy) , June 2010. Oral Presentation.
- Breiman, Leo. Bagging predictors. Machine Learning , 24 (2):123-140, 1994.
- Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier neural networks. In JMLR W&CP: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2011) , April 2011.
- Goodfellow, Ian J., Courville, Aaron, and Bengio, Yoshua. Joint training of deep Boltzmann machines for classification. In International Conference on Learning Representations: Workshops Track , 2013.
- Hahnloser, Richard H. R. On the piecewise analysis of networks of linear threshold neurons. Neural Networks , 11(4):691-697, 1998.
- Hinton, Geoffrey E., Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580, 2012. · arXiv
- Jarrett, Kevin, Kavukcuoglu, Koray, Ranzato, Marc'Aurelio, and LeCun, Yann. What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV'09) , pp. 2146-2153. IEEE, 2009.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS'2012) . 2012.
- LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, November 1998.
- Malinowski, Mateusz and Fritz, Mario. Learnable pooling regions for image classification. In International Conference on Learning Representations: Workshop track , 2013.
- Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. Deep Learning and Unsupervised Feature Learning Workshop, NIPS, 2011.
- Rifai, Salah, Dauphin, Yann, Vincent, Pascal, Bengio, Yoshua, and Muller, Xavier. The manifold tangent classifier. In NIPS'2011 , 2011. Student paper award.
- Salakhutdinov, R. and Hinton, G.E. Deep Boltzmann machines. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009) , volume 8, 2009.
- Salinas, E. and Abbott, L. F. A model of multiplicative neural responses in parietal cortex. Proc Natl Acad Sci U S A , 93(21):11956-11961, October 1996.
- Sermanet, Pierre, Chintala, Soumith, and LeCun, Yann. Convolutional neural networks applied to house numbers digit classification. CoRR , abs/1204.3968, 2012a. · arXiv
- Sermanet, Pierre, Chintala, Soumith, and LeCun, Yann. Convolutional neural networks applied to house numbers digit classification. In International Conference on Pattern Recognition (ICPR 2012) , 2012b.
- Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan Prescott. Practical bayesian optimization of machine learning algorithms. In Neural Information Processing Systems , 2012.
- Srebro, Nathan and Shraibman, Adi. Rank, trace-norm and max-norm. In Proceedings of the 18th Annual Conference on Learning Theory , pp. 545-560. SpringerVerlag, 2005.
- Srivastava, Nitish. Improving neural networks with dropout. Master's thesis, U. Toronto, 2013.
- Wang, Shuning. General constructive representations for continuous piecewise-linear functions. IEEE Trans. Circuits Systems , 51(9):1889-1896, 2004.
- Yu, Dong and Deng, Li. Deep convex net: A scalable architecture for speech pattern classification. In INTERSPEECH , pp. 2285-2288, 2011.
- Zeiler, Matthew D. and Fergus, Rob. Stochastic pooling for regularization of deep convolutional neural networks. In International Conference on Learning Representations , 2013.
Cited by
Sorted by citations
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
“techniques [13, 30, 10, 36], aggressive data augmentation [18, 14, 29, 33], and large-scale data [4, 26].”
BackgroundFaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
“The kernel is specified as rows× cols, stride and the maxout [6] pooling size as p = 2.”
MethodRecent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Geometric Deep Learning: Going beyond Euclidean data
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., & Vandergheynst, P. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2693418 · Cited by 3,661 · see citing works
Brain tumor segmentation with Deep Neural Networks
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., & Larochelle, H. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.05.004 · Cited by 3,278 · see citing works
Self-citationA systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088 · see citing works
“There are exactly 5000 training and 1000 test examples in each class. We do not use any data augmentation but follow standard preprocessing comprising global contrast normalization and ZCA whitening [58]. For CIFAR-10 experiments we use one of the best performing type of CNN model on this dataset, i.e. All-CNN [59]. T…”
MethodHD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition
Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., & Yu, Y. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.314 · Cited by 936 · see citing works
Data augmentation: A comprehensive survey of modern approaches
Mumuni, A., & Mumuni, F. (2022) · Array · doi:10.1016/j.array.2022.100258 · Cited by 891 · see citing works
Krauss, C., Do, X. A., & Huck, N. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2016.10.031 · Cited by 750 · see citing works
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · see citing works
“…representation for all inputs. x pool = max( X | y ); mean( X | y ); min( X | y ); x self (15) We feed this joined representation through a three-layer, batch-normalized [Ioffe and Szegedy, 2015] maxout network [Goodfellow et al., 2013] to produce a probability distribution over possible classes.”
MethodModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420 · see citing works
MegDet: A Large Mini-Batch Object Detector
Peng, C., Xiao, T., Li, Z., Jiang, Y., Zhang, X., Jia, K., Yu, G., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00647 · Cited by 311 · see citing works
Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
Ioannou, Y., Robertson, D., Cipolla, R., & Criminisi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.633 · Cited by 321 · see citing works
Neural network with unbounded activation functions is universal approximator
Sonoda, S., & Murata, N. (2017) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2015.12.005 · Cited by 303 · see citing works
Golas, S. B., Shibahara, T., Agboola, S., Otaki, H., Sato, J., Nakae, T., Hisamitsu, T., Kojima, G., Felsted, J., Kakarmath, S., et al. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0620-z · Cited by 287 · see citing works
Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning
Roy, D., Panda, P., & Roy, K. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.09.010 · Cited by 257 · see citing works
MFAS: Multimodal Fusion Architecture Search
Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193 · see citing works
Yu, Z., Tan, E.-L., Ni, D., Qin, J., Chen, S., Li, S., Lei, B., & Wang, T. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2705031 · Cited by 109 · see citing works
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification
Li, C., Stevens, A., Chen, C., Pu, Y., Gan, Z., & Carin, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.611 · Cited by 39 · see citing works
- Authors
- Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2013
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1302.4389
- OpenAlex
- W7099244549
- Cited by
- 935
- References
- 0
- Citations by year
- 2025: 19 · 2024: 49 · 2023: 59 · 2022: 58 · 2021: 97 · 2020: 128 · 2019: 108 · 2018: 122 · 2017: 113 · 2016: 86 · 2015: 69 · 2014: 25 · 2013: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- dropout, neural network architecture, image classification, training optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
