Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 301–400 of 5,240
Label Propagation via Teaching-to-Learn and Learning-to-Teach
Gong, C., Tao, D., Liu, W., Liu, L., & Yang, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2514360 · Cited by 138
“[19] proposed curriculum learning, which treats”
Dynamic Computational Time for Visual Attention
Li, Z., Yang, Y., Liu, X., Zhou, F., Wen, S., & Xu, W. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.145 · Cited by 130
“We address this problem with curriculum learning [13].”
BackgroundA theoretical understanding of self-paced learning
Meng, D., Zhao, Q., & Jiang, L. (2017) · Information Sciences · doi:10.1016/j.ins.2017.05.043 · Cited by 134
Guide Subspace Learning for Unsupervised Domain Adaptation
Zhang, L., Fu, J., Wang, S., Zhang, D. D., Dong, Z. Y., & Chen, C. L. P. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2944455 · Cited by 132
“[26] tends to train the model by including the samples from simple to complex progressively.”
BackgroundKnowledge Matters: Importance of Prior Information for Optimization
Gülçehre, Ç., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.4083 · Cited by 128
“Bengio et al. (2009b) introduce a computational hypothesis related to a presumed issue with effective local minima when directly learning the target task: the good solutions correspond to hard-to-find-by-chance effective local minima, and intermediate tasks prepare the learner’s internal configuration…”
BackgroundResultInfluentialSelf-citationPhysics-Based Rendering for Improving Robustness to Rain
Halder, S., Lalonde, J.-F., & Charette, R. D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01030 · Cited by 130
“Instead, we refine our untuned models using curriculum learning [4] on rain intensity in ascending order (25, then 50, then 75 and finally 100mm/hr rain).”
MethodDeep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges
Bhagat, S., Banerjee, H., Tse, Z. T. H., & Ren, H. (2019) · Robotics · doi:10.3390/robotics8010004 · Cited by 160
Towards Human-Machine Cooperation: Self-Supervised Sample Mining for Object Detection
Wang, K., Yan, X., Zhang, D., Zhang, L., & Lin, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00173 · Cited by 127
“In our framework, inspired by the recently proposed techniques: Curriculum Learning (CL) [2] and Selfpaced Learning (SPL) [20][18], we formulate the combining of the SSM and the AL as a concise optimization problem.”
BackgroundMethodQu, X., Gan, W., Song, D., & Zhou, L. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.114016 · Cited by 155
FigureQA: An Annotated Figure Dataset for Visual Reasoning
Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.07300 · Cited by 122
“s not considered during generation. The FigureQA corpus can be extended iteratively, each time raising the task complexity, as model performance increases. This is reminiscent of curriculum learning (Bengio et al., 2009) allowing iterative pretraining on increasingly challenging versions of the data. By releasing the…”
BackgroundSelf-citationHierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129
Improving Spatiotemporal Self-supervision by Deep Reinforcement Learning
Büchler, U., Brattoli, B., & Ommer, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_47 · Cited by 123
“The fine-tuning has been performed using the standard CaffeNet, without batch normalization and groups 2 for conv[2,4,5].”
BackgroundMethodPretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 450
Rapid locomotion via reinforcement learning
Margolis, G. B., Yang, G., Paigwar, K., Chen, T., & Agrawal, P. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649231224053 · Cited by 231
“Training can be made easier by initially providing simple tasks to the agent and then slowly increasing their complexity using a curriculum [4].”
BackgroundTeaching Robots to Predict Human Motion
Gui, L.-Y., Zhang, K., Wang, Y.-X., Liang, X., Moura, J. M. F., & Veloso, M. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594452 · Cited by 123
“This problem is alleviated by a noise scheduling scheme [31] that adds noise to the input during”
BackgroundActive Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
Chang, H.-S., Miller, E. G., & McCallum, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.07433 · Cited by 125
“Inspired by this observation, curriculum learning [5] has shown that learning from easier instances first can also improve neural network training.”
Improving Medical Images Classification With Label Noise Using Dual-Uncertainty Estimation
Ju, L., Wang, X., Wang, L., Mahapatra, D., Zhao, X., Zhou, Q., Liu, T., & Ge, Z. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3141425 · Cited by 128
Robot Learning From Randomized Simulations: A Review
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.799893 · Cited by 128
“Widely known in supervised learning (Bengio et al., 2009; Kumar et al., 2010), curriculum learning has been applied to RL (Asada et al.”
BackgroundTraining Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning
Wu, Y., & Tian, Y. (2017) · International Conference on Learning Representations · Cited by 136
Yang, J., Shen, C., & Huang, N. (2020) · Frontiers in Pharmacology · doi:10.3389/fphar.2020.00069 · Cited by 124
“This training strategy is similar to the “ curriculum learning ” strategy (Bengio et al., 2009) because it reduces the dif fi culty of learning via training on the organized data.”
ResultRain Rendering for Evaluating and Improving Robustness to Bad Weather
Tremblay, M., Halder, S. S., de Charette, R., & Lalonde, J. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01366-3 · Cited by 170
“Finally, we also use the latter to refine algorithms using curriculum learning [5], and demonstrate improved robustness on real rainy images.”
MethodReinforced Multi-Teacher Selection for Knowledge Distillation
Yuan, F., Shou, L., Pei, J., Lin, W., Gong, M., Fu, Y. H., & Jiang, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17680 · Cited by 187
“This is similar to the observation in curriculum learning (Bengio et al. 2009).”
ResultMultimodal Sparse Transformer Network for Audio-Visual Speech Recognition
Song, Q., Sun, B., & Li, S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3163771 · Cited by 145
“Thereby, a curriculum learning [44] strategy is used.”
MethodTraining Deep Convolutional Neural Networks to Play Go
Clark, C., & Storkey, A. (2015) · Cited by 131
A Survey on Deep Active Learning: Recent Advances and New Frontiers
Li, D., Wang, Z. R., Chen, Y., Jiang, R., Ding, W., & Okumura, M. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3396463 · Cited by 265
“As a methodology for selecting or generating a subset of training data in data-centric AI, DAL is closely related to learning settings and practical techniques, including curriculum learning [10], transfer learning [11], data augmentation or pruning [12], [13], and dataset distillation [14].”
DreamCoder: growing generalizable, interpretable knowledge with wake–sleep Bayesian program learning
Ellis, K., Wong, L., Nye, M., Sablé-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A., & Tenenbaum, J. B. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0050 · Cited by 125
“DreamCoder’s knowledge grows gradually, with dynamics related to but different from earlier developmental proposals for ‘curriculum learning’ [49] and ‘starting small’ [50].”
BackgroundCurriculum Learning and Minibatch Bucketing in Neural Machine Translation
Charles University, F. O. M. A. P., Kocmi, T., & Bojar, O. (2017) · RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning · doi:10.26615/978-954-452-049-6_050 · Cited by 110
“very quick overfitting as we discuss in Section 4.5. If the network was presented only with the more difficult examples, its performance on theeasy ones would drop. Some mixing strategy is thus needed. Bengio et al. (2009) propose a relatively simple strategy. They organize all training data into bins ofsimilarcomplexit…”
BackgroundInfluentialMassively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters
Pratap, V., Sriram, A., Tomasello, P., Hannun, A., Liptchinsky, V., Synnaeve, G., & Collobert, R. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2831 · Cited by 117
“For these cases, we introduced a curriculum training [17] based approach, which incrementally adds each language after the model has been trained for a fixed number of iterations or the Character Error Rate (CER) goes below 50% for the previously added language.”
MethodSelf-citationZamir, A., Wu, T.-L., Sun, L., Shen, W. B., Shi, B. E., Malik, J., & Savarese, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.196 · Cited by 179
“We provided a study on feedback based learning, arguing it is a worthwhile alternative to commonly employed feedforward paradigm with several basic advantages: early prediction, taxonomy compliance, and Episodic Curriculum Learning.”
BackgroundMethodInfluentialBattiti, R., & Passerini, A. (2010) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2010.2058118 · Cited by 136
“Concept learning [47], curriculum learning [48], or deep learning [49] techniques can be tested here.”
BackgroundTrack and Segment: An Iterative Unsupervised Approach for Video Object Proposals
Xiao, F., & Lee, Y. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.107 · Cited by 113
“In particular, the tracking approaches of [48, 24] use the framework of self-paced learning [4, 29] to carefully choose which frames to learn and update a tracking model.”
MethodAytar, Y., Castrejon, L., Vondrick, C., Pirsiavash, H., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2753232 · Cited by 116
“Our approach is a form of curriculum learning [5].”
BackgroundGoing Deeper with Deep Knowledge Tracing.
Xiong, X., Zhao, S., Inwegen, E. G. V., & Beck, J. E. (2016) · Educational Data Mining · Cited by 155
Self-Paced Prioritized Curriculum Learning With Coverage Penalty in Deep Reinforcement Learning
Ren, Z., Dong, D., Li, H., & Chen, C. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2790981 · Cited by 151
“Curriculum learning is motivated by a student learning strategy in which students get benefit from a meaningful ordered curriculum [22].”
Compound Probabilistic Context-Free Grammars for Grammar Induction
Kim, Y., Dyer, C., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1228 · Cited by 117
“We employ a curriculum learning strategy (Bengio et al., 2009) where we train only on sentences of length up to 30 in the first epoch, and increase this length limit by 1 each epoch.”
MethodThe Eighty Five Percent Rule for optimal learning
Wilson, R. C., Shenhav, A., Straccia, M., & Cohen, J. D. (2019) · Nature Communications · doi:10.1038/s41467-019-12552-4 · Cited by 119
“More generally, our work closely to the Region of Proximal Learning and Desirable Difficulty frameworks in education [25–27] and Curriculum Learning and Self-Paced Learning [7, 8] in computer science.”
BackgroundMethodInfluentialSpatio-Contextual Deep Network-Based Multimodal Pedestrian Detection for Autonomous Driving
Dasgupta, K., Das, A., Das, S. K., Bhattacharya, U., & Yogamani, S. K. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3146575 · Cited by 153
“Curriculum Learning: Curriculum Learning [45] is a training method that aims to gradually increase the complexity of the data fed to the input neural network.”
BackgroundCompetence-based Multimodal Curriculum Learning for Medical Report Generation
Liu, F., Ge, S., & Wu, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.234 · Cited by 122
“In recent years, curriculum learning (Bengio et al., 2009), which enables the models to gradually proceed from easy samples to more complex ones in training (Elman, 1993), has received growing research interests in natural language processing field, e.g., neural machine translation (Platanios et…”
BackgroundMundt, M., Hong, Y., Pliushch, I., & Ramesh, V. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.014 · Cited by 132
“Bengio et al. (2009) have similarly found that sorting data in a curriculum that introduces classes into the training process according to their difficulty improves the obtained accuracy.”
BackgroundLearning action-oriented models through active inference
Tschantz, A., Seth, A. K., & Buckley, C. L. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007805 · Cited by 121
How Do Humans Teach: On Curriculum Learning and Teaching Dimension
Khan, F. S., Mutlu, B., & Zhu, X. (2011) · neural information processing systems · Cited by 175
Curriculum Learning for Domain Adaptation in Neural Machine Translation
Zhang, X., Shapiro, P., Kumar, G., McNamee, P., Carpuat, M., & Duh, K. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1189 · Cited by 109
“In contrast, the probabilistic curriculum (Bengio et al., 2009) works by dividing the training procedure into distinct phases.”
BackgroundMethodNeural Interactive Collaborative Filtering
Zou, L., Xia, L., Gu, Y., Zhao, X., Liu, W., Huang, J. X., & Yin, D. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401181 · Cited by 149
“Since the larger γ means planning in longer future horizons for RL, the increasing { γ e } can be treated as an increasingly difficult curriculum [2], which gradually guides the learning agent from 1-horizon (greedy solution), 2-horizon, . . . , to overall optimal solutions.”
BackgroundRapid Locomotion via Reinforcement Learning
Margolis, G. B., Yang, G., Paigwar, K., Chen, T., & Agrawal, P. (2022) · Robotics: Science and Systems · doi:10.15607/rss.2022.xviii.022 · Cited by 219
LLaRA: Large Language-Recommendation Assistant
Liao, J., Li, S., Yang, Z., Wu, J., Yuan, Y., Wang, X., & He, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657690 · Cited by 420
Synthesizing Supervision for Learning Deep Saliency Network without Human Annotation
Zhang, D., Han, J., Zhang, Y., & Xu, D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2900649 · Cited by 118
“Self-paced learning (SPL) [40] and curriculum learning (CL) [41] are two types of learning strategies that can improve the learning process by selecting from easy examples to hard ones, in which the key is to select reliable and informative training samples at each learning stage.”
BackgroundSelf-Paced Clustering Ensemble
Zhou, P., Du, L., Liu, X., Shen, Y.-D., Fan, M., & Li, X. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2984814 · Cited by 119
“[10], which incrementally involves instances (from easy to difficult ones) into learning.”
BackgroundLearning From Synthetic Animals
Mu, J., Qiu, W., Hager, G. D., & Yuille, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01240 · Cited by 161
“To tackle the problem, we introduce a curriculum learning strategy [2, 13, 16] to progressively increase pseudo-labels and train models in iterations.”
MethodWeakly Supervised Temporal Sentence Grounding with Gaussian-based Contrastive Proposal Learning
Zheng, M., Huang, Y., Chen, Q., Peng, Y., & Liu, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01511 · Cited by 127
“The curriculum learning method emulates the learning behavior of humans [1, 25].”
MethodToward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119
“The potential bootstrapping mechanisms that we will consider operate in the context of unsupervised and reinforcement learning, and go well beyond the types of curriculum learning ideas used in today’s machine learning (Bengio et al., 2009).”
BackgroundUnderwater object detection using Invert Multi-Class Adaboost with deep learning
Chen, L., Liu, Z., Tong, L., Jiang, Z., Wang, S., Dong, J., & Zhou, H. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207506 · Cited by 160
“For Faster RCNN, we use three backbone networks including VGG16, ResNet50 [34] and ResNet101 [34].”
BackgroundMethodInfluentialMulti-scale convolutional neural network for automated AMD classification using retinal OCT images
Sotoudeh‐Paima, S., Jodeiri, A., Hajizadeh, F., & Soltanian-Zadeh, H. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105368 · Cited by 138
“This is similar to the idea of curriculum learning [68], where the model starts with learning simpler concepts first (e.”
MethodQuery-Efficient Imitation Learning for End-to-End Autonomous Driving
Zhang, J., & Cho, K. (2016) · arXiv · doi:10.48550/arxiv.1605.06450 · Cited by 118
“This has an effect of automated curriculum learning [2] with a mix strategy [20], where the safety policy selects training examples of appropriate difficulty at each iteration.”
BackgroundHow Hard Can It Be? Estimating the Difficulty of Visual Search in an Image
Ionescu, R. T., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, D. P., & Ferrari, V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.237 · Cited by 115
“Curriculum learning [4] can help to optimize the training of deep learning models.”
BackgroundGraph Distillation for Action Detection with Privileged Modalities
Luo, Z., Hsieh, J.-T., Jiang, L., Niebles, J. C., & Fei-Fei, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_11 · Cited by 111
“Since it is nontrivial to jointly train on multiple modalities from scratch, we employ curriculum learning [1] to train the distillation graph.”
MethodMulti-agent reinforcement learning for autonomous vehicles: a survey
Dinneweth, J., Boubezoul, A., Mandiau, R., & Espié, S. (2022) · Autonomous Intelligent Systems · doi:10.1007/s43684-022-00045-z · Cited by 171
“Curriculum learning [32] refers to a learning method that gradually increases the difficulty.”
Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019) · arXiv
“This kind of incremental curriculum is intuitive and variants of it appear in the literature when a task is too hard to learn directly [40, 67–70].”
BackgroundCharacterizing possible failure modes in physics-informed neural networksPreprint
Krishnapriyan, A. S., Gholami, A., Zhe, S., Kirby, R. M., & Mahoney, M. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01050 · Cited by 446
“This is somewhat analogous to curriculum learning in ML [1], but applied by progressively making the PDE/ODE harder to solve.”
MethodAdversarially Robust Policy Learning: Active construction of physically-plausible perturbations
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., & Savarese, S. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206245 · Cited by 116
“We used a curriculum learning approach [4] in order to train our agents on increasing φ Algorithm 1: Adversarially Robust Policy Learning”
MethodSelf-Supervised Representation Learning From Videos for Facial Action Unit Detection
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01118 · Cited by 137
“We adopted the curriculum learning [2] strategy to train the model with progressively difficult samples, because the randomly sampled image pairs may contain large deviations, which are too challenging to learn.”
MethodAdaptive Consistency Regularization for Semi-Supervised Transfer Learning
Abuduweili, A., Li, X., Shi, H., Xu, C., & Dou, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00685 · Cited by 153
“Which can be regarded as a kind of curriculum learning[1].”
BackgroundSLNL: A novel method for gene selection and phenotype classification
Huang, H., Wu, N., Liang, Y. Q., Peng, X., & Shu, J. (2022) · International Journal of Intelligent Systems · doi:10.1002/int.22844 · Cited by 119
“Inspired by human learning mechanisms, Bengio et al. propose the so ‐ called curriculum learning or SPL. 24,37 As described by Kumar et al., 24 the SPL can be a concise optimization model with a penalty term being introduced.”
BackgroundGeoMVSNet: Learning Multi-View Stereo with Geometry Perception
Zhang, Z., Peng, R., Hu, Y., & Wang, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02060 · Cited by 127
“Based on the observation, we refer to the idea of curriculum learning [2] to embed coarse geometric priors into finer stages from easy to difficult.”
Contrastive Self-supervised Learning in Recommender Systems: A Survey
Jing, M., Zhu, Y., Zang, T., & Wang, K. (2023) · ACM Transactions on Information Systems · doi:10.1145/3627158 · Cited by 183
“It leverages curriculum learning [4] to pre-train the model in an elementary-to-advanced process.”
BackgroundAdversarial Generation of Natural Language
Subramanian, S., Rajeswar, S., Dutil, F., Pal, C., & Courville, A. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2629 · Cited by 112
“As a way to provide training signals of incremental difficulty, we use curriculum learning (Bengio et al., 2009) and train our generator to produce sequences of gradually increasing lengths as training progresses.”
MethodSemantic Models for the First-Stage Retrieval: A Comprehensive Review
Guo, J., Cai, Y., Fan, Y., Sun, F., Zhang, R., & Cheng, X. (2022) · ACM Transactions on Information Systems · doi:10.1145/3486250 · Cited by 113
“We believe it would be interesting and valuable to study more complex training strategies, such as curriculum learning [18], to help the model optimization for the first-stage retrieval.”
BackgroundSelf Paced Deep Learning for Weakly Supervised Object Detection
Sangineto, E., Nabi, M., Culibrk, D., & Sebe, N. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2804907 · Cited by 114
“A pioneering work in this direction is the curriculum learning approach proposed in [11].”
BackgroundIlluminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
Justesen, N., Torrado, R. R., Bontrager, P., Khalifa, A., Togelius, J., & Risi, S. (2018) · OAR@UM (University of Malta) · doi:10.48550/arxiv.1806.10729 · Cited by 109
“Essentially the same idea has later been independently invented as curriculum learning [4].”
BackgroundLaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models
Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Yang, C., Yang, C., He, Y., Yu, J., Yang, P., et al. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02295-1 · Cited by 647
“To facilitate the fine-tuning process, we employ curriculum learning (Bengio et al., 2009).”
A PAC-Bayesian bound for Lifelong Learning
Pentina, A., & Lampert, C. H. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.2838 · Cited by 111
“into the direction of learning tasks of continuously improving difficulty (Bengio et al., 2009).”
BackgroundMemory-aware curriculum federated learning for breast cancer classification
Jiménez-Sánchez, A., Tardy, M., González Ballester, M. A., Mateus, D., & Piella, G. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107318 · Cited by 114
“Contributions: In this work, we investigate for the first time the use of Curriculum Learning (CL) [7] in FL to boost the classification performance while improving domain alignment.”
BackgroundMethodFedHAR: Semi-Supervised Online Learning for Personalized Federated Human Activity Recognition
Yu, H., Chen, Z., Zhang, X., Chen, X., Zhuang, F., Xiong, H., & Cheng, X. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3136853 · Cited by 113
“…of optimizing over both supervised and unsupervised objectives simultaneously in the early training stage, we develop a simple yet effective strategy to gradually enlarge unsupervised gradients and anneal the supervised gradients across training iterations inspired by curriculum learning [33].”
BackgroundDeep Learning of Representations: Looking Forward
Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1305.0445 · Cited by 99
“Another option, following the idea of curriculum learning (Bengio et al., 2009), is to provide guidance ourselves to learning machines (as exemplified in the toy example of Gulcehre and Bengio (2013)), by “teaching them” gradually more complex concepts to help them understand the world around us (keeping in mind that…”
BackgroundSelf-citationVideo Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition
Ahsan, U., Madhok, R., & Essa, I. (2019) · 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2019.00025 · Cited by 106
“Progressive Training Approach We borrow principles from curriculum learning [6] to train our video jigsaw network with an easy jigsaw puzzle task first and then train it for a harder task.”
MethodTransferable Curriculum for Weakly-Supervised Domain Adaptation
Shu, Y., Cao, Z., Long, M., & Wang, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014951 · Cited by 105
“This paper presents a Transferable Curriculum Learning ( TCL ) approach to weakly-supervised domain adaptation, extending from curriculum learning (Bengio et al. 2009) and adversarial learning (Goodfellow et al. 2014).”
BackgroundMethodMessage Passing for Hyper-Relational Knowledge Graphs
Galkin, M., Trivedi, P., Maheshwari, G., Usbeck, R., & Lehmann, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.596 · Cited by 194
“Additionally, HINGE has to be trained sequentially in a curriculum learning (Bengio et al., 2009) fashion requiring sorting all facts in a KG in an ascending order of the amount of qualifiers per fact which might be prohibitively expensive for large-scale graphs.”
Curriculum Loss: Robust Learning and Generalization against Label CorruptionPreprint
Lyu, Y., & Tsang, I. W.-H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10045 · Cited by 190
“This kind of works are inspired by curriculum learning (Bengio et al., 2009).”
BackgroundMethodInfluentialSelf-supervised learning of a facial attribute embedding from video
Wiles, O., Koepke, A. S., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1808.06882 · Cited by 111
“The training of the network is divided into stages, so that knowledge can be built up over time as the examples given become progressively more difficult, as inspired by [5, 27].”
BackgroundSelf-Paced Balance Learning for Clinical Skin Disease Recognition
Yang, J., Wu, X., Liang, J., Sun, X., Cheng, M., Rosin, P. L., & Wang, L. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2917524 · Cited by 125
“[67] formalize a general training strategy termed curriculum learning (CL).”
BackgroundRobust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks
Diaz-Guerra, D., Miguel, A., & Beltran, J. R. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3040031 · Cited by 162
“Therefore, we followed a curriculum learning strategy [42] using batches of 5 trajectories with SNR= 30 dB for the first 20 epochs and for the following epochs we employed the full range of SNRs, increased the batch size to 10 trajectories, and reduced the learning rate from 1e-4 to 1e-5.”
Distributed Heuristic Multi-Agent Path Finding with Communication
Ma, Z., Luo, Y., & Ma, H. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560748 · Cited by 162
“Instead, we use curriculum learning method by gradually introducing more difficult tasks to agents [20].”
Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following
Ma, D., Chen, X., Ma, W., Zheng, H., & Qu, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3282681 · Cited by 126
“Therefore, it is trained by learning from easy samples at the beginning and gradually progressing to more complex samples and knowledge [31].”
BackgroundWu, J., Tenenbaum, J. B., & Kohli, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.744 · Cited by 104
“supervision required for training the encoder through curriculum learning [2].”
BackgroundA Developmental Approach to Machine Learning?
Smith, L. B., & Slone, L. K. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.02124 · Cited by 107
“There are current approaches to machine learning (curriculum learning, and iterative teaching, for example) that seek to optimize learning through ordered and structured training sets (e.g., Bengio et al., 2009; Krueger and Dayan, 2009).”
BackgroundBalanced Self-Paced Learning for Generative Adversarial Clustering Network
Dizaji, K. G., Wang, X., Deng, C. D., & Huang, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00452 · Cited by 127
“Inspired by the human learning principle, curriculum learning starts learning with easier examples, and then gradually takes more complex examples into consideration [2].”
BackgroundDeep Deformation Network for Object Landmark Localization
Yu, X., Zhou, F., & Chandraker, M. K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_4 · Cited by 120
“This is a shared motivation with curriculum learning [51] and avoids the difficulties of training the whole network from scratch.”
BackgroundGlancing Transformer for Non-Autoregressive Neural Machine Translation
Qian, L., Zhou, H., Bao, Y., Wang, M., Qiu, L., Zhang, W., Yu, Y., & Li, L. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.155 · Cited by 108
“With a similar idea with curriculum learning (Bengio et al., 2009), we propose a curriculum sampling strategy to decide the number that aims at reducing more difficulty from the start and gradually approaching the simultaneous generation setting as the model is trained better.”
BackgroundMethodZamir, A., Wu, T.-L., Sun, L., Shen, W., Shi, B. E., Malik, J., & Savarese, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · Cited by 118
Meta-Transfer Learning Through Hard Tasks
Sun, Q., Liu, Y., Chen, Z., Chua, T.-S., & Schiele, B. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3018506 · Cited by 150
“Curriculum learning [34] and hard negative mining [35] both suggest that faster convergence and stronger performance can be achieved by better arrangement of”
BackgroundLearning-Based Multi-Robot Formation Control With Obstacle Avoidance
Bai, C., Peng, Y., Pan, W., & Guo, J. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3107336 · Cited by 121
“2) Training Stage: Inspired by the curriculum learning paradigm [32], we propose a two-stage training process.”
MethodReinforcement Learning Neural Turing Machines - Revised
Zaremba, W., & Sutskever, I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00521 · Cited by 93
“Thus, it is plausible that curriculum learning [4], which has not b een helpful for DNNs because their training objectives are too easy, will be useful for NTMs sin ce their objectives are harder.”
BackgroundPretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06467 · Cited by 101
“ed with neural rankers [Dehghani et al., 2017, MacAvaney et al., 2019b]. Also hoping to improve training, MacAvaney et al. [2020d] investigated whether monoBERT can benefit from a training curriculum [Bengio et al., 2009] in which the model is presented with progressively more difficult training examples as training pro…”
BackgroundCrowd Counting via Segmentation Guided Attention Networks and Curriculum Loss
Wang, Q., & Breckon, T. P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3138896 · Cited by 114
“Curriculum learning is a strategy of model training (e.g., neural networks) in machine learning and was proposed by Bengio et al. [45].”
ALLSTEPS: Curriculum‐driven Learning of Stepping Stone Skills
Xie, Z., Ling, H. Y., Kim, N. H., & van de Panne, M. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14115 · Cited by 159
“Curriculum learning is the learning process where task difficulty is increased overtime during training [BLCW09].”
BackgroundProgressive Transformer-Based Generation of Radiology Reports
Nooralahzadeh, F., Perez Gonzalez, N., Frauenfelder, T., Fujimoto, K., & Krauthammer, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.241 · Cited by 107
“, image-to-text-to-text) generation framework (inspired by Curriculum Learning (Bengio et al., 2009) and the work of Tan et al.”
BackgroundMethodComputer Vision: A Reference Guide
Scaramuzza, D. (2014) · Springer eBooks · doi:10.1007/978-0-387-31439-6 · Cited by 116
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5538 · Cited by 98
“This is nicely related to the curriculum learning idea [24] that it may be much easier to learn simpler concepts first and then build”
BackgroundSelf-citationReplay in Deep Learning: Current Approaches and Missing Biological Elements
Hayes, T. L., Krishnan, G. P., Bazhenov, M., Siegelmann, H. T., Sejnowski, T. J., & Kanan, C. (2021) · Neural Computation · doi:10.1162/neco_a_01433 · Cited by 106
“More interestingly, hindsight experience replay can serve as a form of curriculum learning (Bengio, Louradour, Collobert, & Weston, 2009) by structuring the rewards such that they start off simple and grow increasingly more complex during training.”
A Survey on Document-level Neural Machine Translation: Methods and Evaluation
Maruf, S., Saleh, F., & Haffari, G. (2021) · ACM Computing Surveys · doi:10.1145/3441691 · Cited by 253
“16 Stojanovski and Fraser [118] introduced a curriculum learning approach [9] that leveraged oracle information to promote anaphora resolution while training a context-aware NMT model.”
BackgroundLi, C., Zheng, P., Yin, Y., Pang, Y. M., & Huo, S. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102471 · Cited by 108
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.