Knowledge Matters: Importance of Prior Information for Optimization
Open accessThe DOI matches its OpenAlex record; 13 of the 13 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Self-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationImproving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · cited 1×
“Hence one would expect that as sparsity of the gradients (obtained because of sparsity of the activations) increases, training would become more efficient, as if we were training a smaller sub-network for each example, with shared weights across examples, as in dropouts (Hinton et al., 2012).”
BackgroundLarge-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · cited 1×
“However, online SGD directly optimizes the expected risk, because the examples are drawn iid from the ground-truth distribution (Bottou, 2010).”
BackgroundBengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 5×
“Bengio et al. (2009b) introduce a computational hypothesis related to a presumed issue with effective local minima when directly learning the target task: the good solutions correspond to hard-to-find-by-chance effective local minima, and intermediate tasks prepare the learner’s internal configuration…”
BackgroundResultInfluentialSelf-citationADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · cited 1×
“Greedy layerwise unsupervised training procedure is used to train the deep auto-encoder architecture (Bengio et al., 2007).”
MethodSelf-citation(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 2×
“(Jarrett et al., 2009a) used a local contrast normalization layer in their architecture which performs subtractive and divisive normalization.”
MethodIdentifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827 · cited 1×
“work suggests that it would generally not be because of local minima (Dauphin et al., 2014; Choromanska et al., 2015).”
BackgroundSelf-citationThe Loss Surfaces of Multilayer Networks
Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.0233 · Cited by 733 · cited 1×
“work suggests that it would generally not be because of local minima (Dauphin et al., 2014; Choromanska et al., 2015).”
BackgroundTrends in Cranial Capacity and Cranial Index in Subsaharan Africa During the Holocene
Henneberg, M., & Steyn, M. (1993) · American Journal of Human Biology · doi:10.1002/ajhb.1310050411 · Cited by 70
Cited by
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
“On the contrary, (Gülçehre & Bengio, 2013) apply the standardization layer to theoutputof the nonlinearity, which results in sparser activations.”
BackgroundMethodFitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Self-citationOn Training Targets for Supervised Speech Separation
Wang, Y., Narayanan, A., & Wang, D. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2352935 · Cited by 1,067
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853
A deep learning system for detecting diabetic retinopathy across the disease spectrum
Dai, L., Wu, L., Li, H., Cai, C., Wu, Q., Kong, H., Liu, R., Wang, X., Hou, X., Liu, Y., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-23458-5 · Cited by 620
Metadata
- Authors
- Gülçehre, Ç., & Bengio, Y.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2013-01-17
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1301.4083
- OpenAlex
- W1814328102
- Cited by
- 128
- References
- 13
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.