Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 1,001–1,100 of 5,240
Neural Text-to-Speech Adaptation from Low Quality Public Recordings
Hu, Q., Marchi, E., Winarsky, D., Stylianou, Y., Naik, D., & Kajarekar, S. (2019) · 10th ISCA Workshop on Speech Synthesis (SSW 10) · doi:10.21437/ssw.2019-5 · Cited by 31
“The parameters of the speaker encoder are optimized via a curriculum learning procedure [30] to improve robustness under various acoustic conditions and text variability.”
MethodPersonalization of Deep Learning
Schneider, J., & Vlachos, M. (2021) · Springer eBooks · doi:10.1007/978-3-658-32182-6_14 · Cited by 63
“lum learning [31].”
BackgroundInfluentialChirping up the Right Tree: Incorporating Biological Taxonomies into Deep Bioacoustic Classifiers
Cramer, A. L., Lostanlen, V., Farnsworth, A., Salamon, J., & Bello, J. P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052908 · Cited by 44
“Lastly, the complex optimization landscape of multitask training alone can make model fitting difficult; but methods such as using a hierarchical form of curriculum learning [37] might help to alleviate these issues in the near future.”
BackgroundLearning a family of motor skills from a single motion clip
Lee, S., Lee, S., Lee, Y., & Lee, J. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459774 · Cited by 56
Semi-Supervised Hierarchical Graph Classification
Li, J., Huang, Y., Chang, H., & Rong, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3203703 · Cited by 44
“To this end, we make use of the idea of cautious iteration [37] and curriculum learning [43], and cautiously exploit a subset of (Et;G) to update the parameters of IC in each iteration.”
MethodSelf-paced Adaptive Bipartite Graph Learning for Consensus Clustering
Zhou, P., Liu, X., Du, L., & Li, X. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3564701 · Cited by 36
“For example, in [5, 6], it was used to tackle the local optimum problem in non-convex optimization; in [24, 42], it was plugged in the multi-task learning; Jiang et al.”
MethodWang, X., Man, Z., You, M., & Shen, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03124 · Cited by 56
“1(b) in the manner of curriculum learning paradigm [4].”
BackgroundUnicorn: Continual Learning with a Universal, Off-policy Agent
Mankowitz, D. J., Žídek, A., Barreto, A. D. M. S., Horgan, D., Hessel, M., Quan, J., Oh, J., van Hasselt, H. P., Silver, D., & Schaul, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08294 · Cited by 34
“In particular, an automatic curriculum [22, 14, 15, 40, 36] with progressively more emphasis on harder tasks can accelerate learning via more efficient exploration and better generalization [7, 13, 14].”
BackgroundStoyanova, I., Museux, N., Nguyen, S. M., & Filliat, D. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.time.2024.5 · Cited by 1
Orthogonal Gradient Descent for Continual LearningPreprint
Farajtabar, M., Azizan, N., Mott, A., & Li, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07104 · Cited by 365
“Multi-task learning (Sorokin and Burtsev, 2019), curriculum learning (Bengio et al., 2009), and transfer learning (Pan and Yang, 2009; Li et al., 2019a) are other related areas helpful to develop better continual learning machines that do not catas-trophically forget previous experiences.”
BackgroundPre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
Ma, X., Guo, J., Zhang, R., Fan, Y., & Cheng, X. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531772 · Cited by 35
“Other learning techniques like curriculum learning [1] are left for further studies to alleviate this problem.”
BackgroundLearning to Adapt Across Dual Discrepancy for Cross-Domain Person Re-Identification
Luo, C., Song, C., & Zhang, Z. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3167053 · Cited by 39
“Self-paced learning [75] (SPL) is inspired by the cognitive process of human beings, where samples are involved in training from easy to complex [76].”
BackgroundTopology Aware Deep Learning for Wireless Network Optimization
Zhang, S., Yin, B., Zhang, W., & Cheng, Y. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3179352 · Cited by 53
“We take a curriculum training [51] approach to organize the samples and conduct training.”
MethodSelf-Paced Cross-Modal Subspace Matching
Liang, J., Li, Z., Cao, D., He, R., & Wang, J. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2911527 · Cited by 36
“Curriculum learning [4] and self-paced learning [20] have start with learning easier aspects of a learning task, and then gradually take more complex examples into consideration [25].”
BackgroundLearning Bayesian network structures under incremental construction curricula
Zhao, Y., Chen, Y., Tu, K., & Tian, J. W. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.01.092 · Cited by 32
“[4] proposed that a well chosen curriculum strategy can act as a continuation method [33], which first optimizes a highly smoothed objective 150”
BackgroundMethodExploring Commonality and Individuality for Multi-Modal Curriculum Learning
Gong, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10774 · Cited by 34
“The concept of curriculum learning was originally proposed by Bengio et al. (Bengio et al. 2009).”
BackgroundInfluentialBerger, L. U., Hyde, E., Cardoso, M. J., & Sebastien Ourselin, A. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-95921-4_26 · Cited by 39
“Curriculum learning [7] and derivative methods like self-paced learning [8] build on the intuition that, rather than considering all samples simultaneously, the algorithm should be presented with the training data in a meaningful order that facilitates learning.”
MethodEnsemble Teaching for Hybrid Label Propagation
Gong, C., Tao, D., Chang, X., & Yang, J. J. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2773562 · Cited by 34
Self-Paced Kernel Estimation for Robust Blind Image Deblurring
Gong, D., Tan, M., Zhang, Y., van den Hengel, A., & Shi, Q. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.184 · Cited by 34
“Inspired by the learning process of humans [3, 18], self-paced learning (SPL) was originally proposed to learn a latent variable model by focussing first the easy training data, where a set of samples is easy if it admits a good fit in the model space [18].”
MethodCurriculum learning based approach for noise robust language identification using DNN with attention
Vuddagiri, R. K., Vydana, H. K., & Vuppala, A. K. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.06.004 · Cited by 33
Data-driven autocompletion for keyframe animation
Zhang, X., & van de Panne, M. (2018) · Motion in Games · doi:10.1145/3274247.3274502 · Cited by 48
“Consequently, we introduce ω, the Key Importance Weight, which we anneal during the training process as part of a curriculum learning method [Bengio et al. 2009] to help the network learn better during training.”
MethodWhen to Learn What: Deep Cognitive Subspace Clustering
Jiang, Y., Yang, Z., Xu, Q., Cao, X., & Huang, Q. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240582 · Cited by 35
“Self-paced learning (SPL) derives from curriculum learning [3], a human-like learning paradigm where the easier parts of the task are learned first, and the difficulty level gradually increases during the learning process.”
BackgroundA Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks
Sanh, V., Wolf, T., & Ruder, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016949 · Cited by 203
“More recently, Mccann et al. (2018) explored various batch-level sampling strategies and showed that an anti-curriculum learning strategy (Bengio et al. 2009) is most effective.”
BackgroundElastic Boundary Projection for 3D Medical Image Segmentation
Ni, T., Xie, L., Zheng, H., Fishman, E. K., & Yuille, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00221 · Cited by 39
“To alleviate this issue, in the training stage, we gradually replace the added term in Eqn (4) with prediction, following the idea of curriculum learning [1].”
MethodZhou, Y., Nelakurthi, A. R., & He, J. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219952 · Cited by 32
Learning to Train with Synthetic Humans
Hoffmann, D. T., Tzionas, D., Black, M. J., & Tang, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33676-9_43 · Cited by 35
“We vary lighting conditions similarly as [18, 10] using spherical harmonics [6] with randomly drawn coefficients.”
BackgroundMethodChen, Z., Rosenberg, A., Zhang, Y., Wang, G., Ramabhadran, B., & Moreno, P. J. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1475 · Cited by 44
“We use a curriculum training [29] strategy to balance the contribution of written and spoken source material.”
MethodGlobal Context-enhanced Graph Convolutional Networks for Document-level Relation Extraction
Zhou, H., Xu, Y., Yao, W., Liu, Z., Lang, C., & Jiang, H. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.461 · Cited by 40
“To reduce the influence of the noisy data in the distantly supervised dataset, the curriculum learning strategy (Bengio et al., 2009) is applied to get a better pre-trained model.”
A too-good-to-be-true prior to reduce shortcut reliance
Dagaev, N., Roads, B. D., Luo, X., Barry, D. N., Patil, K. R., & Love, B. C. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.12.010 · Cited by 61
“In contrast, in other research directions, such as curriculum learning [3, 13, 40] and self-paced learning [24, 28], it is recommended to stress easy examples early in training.”
Robotic Table Tennis with Model-Free Reinforcement Learning
Gao, W., Graesser, L., Choromański, K., Song, X., Lazic, N., Sanketi, P., Sindhwani, V., & Jaitly, N. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341191 · Cited by 57
“We can address these issues using curriculum learning [3] by carefully adjusting two aspects of the problem:”
MethodResource Management in Cloud Computing Using Machine Learning: A Survey
Goodarzy, S., Nazari, M., Han, R. Y., Keller, E. R., & Rozner, E. (2020) · International Conference on Machine Learning and Applications · doi:10.1109/icmla51294.2020.00132 · Cited by 50
“To deal with continuous streaming of job arrivals, they terminate the training episodes early in the beginning and gradually grow their length, making policy to learn how to handle short and straightforward job sequences and then learn more lengthy challenging sequences [52].”
BackgroundBeneficial and harmful explanatory machine learning
Ai, L., Muggleton, S. H., Hocquette, C., Gromowski, M., & Schmid, U. (2021) · Machine Learning · doi:10.1007/s10994-020-05941-0 · Cited by 34
“Therefore, learning might be organised incrementally, guided by a curriculum [6,52].”
BackgroundNL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Elgohary, A., Meek, C., Richardson, M., Fourney, A., Ramos, G., & Awadallah, A. H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.444 · Cited by 66
“, curriculum learning (Bengio et al., 2009)), (3) optimizing the correction model for better user experience using the progress measure we introduce, and (4) using the SQL edits scheme in other related tasks such as conversational text-to-SQL parsing.”
MethodSpatially Self-Paced Convolutional Networks for Change Detection in Heterogeneous Images
Li, H., Gong, M., Zhang, M., et al. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3078437 · Cited by 34
“SPL can trace back to curriculum learning (CL) [23].”
BackgroundDai, Y., Li, H., Li, Y., Sun, J. Q., Huang, F., Si, L., & Zhu, X. (2021) · doi:10.18653/v1/2021.acl-short.111 · Cited by 38
“This paradigm is called curriculum learning (CL) (Bengio et al., 2009) and has been shown useful in various other problems (Wang et al.”
BackgroundMethodWang, W., Zhang, W., Liu, S., Liu, Q., Zhang, B., Lin, L., & Zha, H. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3111436 · Cited by 39
“This is well motivated by curriculum learning [4] which first learns easier aspects of a task and then gradually moves to difficult ones.”
BackgroundA Generative Adversarial Imitation Learning Approach for Realistic Aircraft Taxi-Speed Modeling
Pham, D.-T., Tran, T.-N., Alam, S., & Dương, V. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3119073 · Cited by 36
Semi-Supervised Action Recognition From Temporal Augmentation Using Curriculum Learning
Tong, A., Tang, C., & Wang, W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3210271 · Cited by 40
“According to the learning principles of a curriculum, they are divided into: (1) pre-defined CL (PCL) [59], [60], which qualitatively evaluates the difficulty of the problem by experts, utilizes a training scheduler to adjust the training data subsets, and prioritizes the subsets that contribute to…”
MethodKnowledge Distillation via Route Constrained OptimizationPreprint
Xiao, J. J., Peng, B., Wu, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09149 · Cited by 34
“In curriculum learning [1] the sequence of learning is generated by spliting the X in to several “lessons” with different hardness depending on a predefined criterion.”
BackgroundInfluentialAdaptive Self-training for Few-shot Neural Sequence LabelingPreprint
Wang, Y., Mukherjee, S., Chu, H., Tu, Y., Wu, M., Gao, J., & Awadallah, A. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03680 · Cited by 43
Zhang, B., Xu, Z., Zhang, J., & Wu, G. (2021) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.12757 · Cited by 34
“For this reason, the training strategy of curriculum learning (Bengio et al., 2009) is adopted, where the learning is from simple samples to complex samples.”
BackgroundRaimondi, D., Corso, M., Fariselli, P., & Moreau, Y. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab1099 · Cited by 38
Physics-informed reinforcement learning for motion control of a fish-like swimming robot
Rodwell, C., & Tallapragada, P. (2023) · Scientific Reports · doi:10.1038/s41598-023-36399-4 · Cited by 43
“…that in reinforcement learning using the Deterministic Policy Gradient (DPG) algorithm for complex tasks, much of computational time is spent going from the initialized random policy to an intermediate policy that, while sub optimal, has qualitative similarities to the optimal policy 30–32 .”
BackgroundMethodImage Deblurring With Image Blurring
Li, Z., Gao, Z. Y., Han, Y., Fu, Y., & Chen, B. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3321515 · Cited by 38
Neural combinatorial optimization with reinforcement learning in industrial engineering: a survey
Chung, K. T., Lee, C. K. M., & Tsang, Y. P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11045-1 · Cited by 65
“…ways to alleviate these problems are, e.g., to soft-constrain the action space with a penalty in the early stage of training to allow exploration (Farazi et al. 2022), by curriculum learning (Bengio et al. 2009 ), different schemes to generate and apply action masks (Huang and Ontañón 2020).”
Pagliaro, A. (2025) · Electronics · doi:10.3390/electronics14091721 · Cited by 65
Gradual DropIn of Layers to Train Very Deep Neural Networks
Smith, L. N., Hand, E., & Doster, T. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.515 · Cited by 34
“DropIn follows the philosophy embedded within curriculum learning [2].”
BackgroundZero-Shot Learning Across Heterogeneous Overlapping Domains
Kumar, A., Muddireddy, P. R., Dreyer, M., & Hoffmeister, B. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-516 · Cited by 34
“Consistent with prior work [2], we find that this training strategy significantly speeds up convergence in training compared to purely random sampling, though sampling from the normalized output distribution adds a fixed time cost.”
BackgroundSelf-Paced Learning-Based Probability Subspace Projection for Hyperspectral Image Classification
Yang, S., Feng, Z., Wang, M., et al. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2841009 · Cited by 33
“It stems from Bengio’s curriculum learning [25], in which each sample is considered to be simple or complex.”
NetTailor: Tuning the Architecture, Not Just the Weights
Morgado, P. M., & Vasconcelos, N. M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00316 · Cited by 38
“This can be done by controlling the order in which examples are introduced [3].”
BackgroundCategory-Sensitive Domain Adaptation for Land Cover Mapping in Aerial Scenes
Fang, B., Kou, R., Li, P., & Chen, P. (2019) · Remote Sensing · doi:10.3390/rs11222631 · Cited by 36
“Adopting curriculum learning [20], Zhang et al.”
Learning a Multi-Domain Curriculum for Neural Machine Translation
Wang, W., Tian, Y., Ngiam, J., Yang, Y., Caswell, I. R., & Parekh, Z. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.689 · Cited by 50
“Curriculum learning (CL) (Bengio et al., 2009) has been used as a formulation for dynamic data selection.”
MethodLifelong Language Knowledge Distillation
Chuang, Y.-S., Su, S., & Chen, Y.-N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.233 · Cited by 66
“In the right-to-left order, the baseline LAMOL can easily outperform multi-task models due to its easiest-to-hardest order, which helps the model to better transfer the knowledge gradually in these NLG tasks, similar to curriculum learning (Bengio et al., 2009).”
MethodSelf-paced resistance learning against overfitting on noisy labels
Shi, X., Guo, Z., Li, K., Liang, Y., & Zhu, X. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109080 · Cited by 40
Open Hierarchical Relation Extraction
Zhang, K., Yao, Y., Xie, R., Han, X., Liu, Z., Lin, F., Lin, L., & Sun, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.452 · Cited by 44
“We propose a hierarchical training paradigm, which is a curriculum learning strategy (Bengio et al., 2009) that enables progressive training.”
MethodCertainty driven consistency loss on multi-teacher networks for semi-supervised learning
Liu, L., & Tan, R. T. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108140 · Cited by 58
“Our method adapts a progressive learning strategy [14] to gradually learn from certain targets to less certain targets in the consistency regularization.”
BackgroundMethodRelative Distributed Formation and Obstacle Avoidance with Multi-agent Reinforcement Learning
Yan, Y., Li, X., Qiu, X., Qiu, J., Wang, J., Wang, Y., & Shen, Y. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812263 · Cited by 42
“reduce the difficulty of training and speed up convergence, we use the idea of predefined curriculum learning [34] and set different levels of difficulties for obstacle avoidance.”
MethodMultimodal Self-Paced Locality-Preserving Learning for Diagnosis of Alzheimer’s Disease
Hao, X., Wang, R., Guo, Y., Xiao, Y., Yu, M., Wang, M., Chen, W., & Zhang, D. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3189701 · Cited by 34
“Observing that people usually first learn simple concepts and then learn complex knowledge, Bengio et al. [32] proposed curriculum learning (CL) to imitate this learning mechanism of humans.”
BackgroundWhere to Look Next: Learning Viewpoint Recommendations for Informative Trajectory Planning
Lodel, M., Brito, B., Serra-Gómez, Á., Ferranti, L., Babuška, R., & Alonso–Mora, J. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812190 · Cited by 49
“We employ curriculum learning during training [37], increasing the number of obstacles from one to three.”
MethodDeep ToA Mask Based Recursive Radar Pulse Deinterleaving
Xiang, H., Shen, F., & Zhao, J. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3193948 · Cited by 36
“Then, the precision P and recall R are as follows: F1 is the harmonic average of precision and recall where P , R , and F1 evaluate the pros and cons of the DTM al-gorithm more accurately, comprehensively, and intuitively.”
Multi-Objective Self-Paced Learning
Li, H., Gong, M., Meng, D., & Miao, Q. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10255 · Cited by 34
“In the field of machine learning and artificial intelligence, curriculum learning (CL) (Bengio et al. 2009) and self-paced learning (SPL) (Kumar, Packer, and Koller 2010) have been attracting increasing attention to deal with the difficulty of training in the presence of non-convex training criteria…”
BackgroundLearning to Discover Efficient Mathematical IdentitiesPreprint
Zaremba, W., Kurach, K., & Fergus, R. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.1584 · Cited by 46
“• We learn using a curriculum [1], starting with the simplest expressions of low degree and slowly increasing k.”
MethodA Fully Progressive Approach to Single-Image Super-ResolutionPreprint
Wang, Y., Perazzi, F., McWilliams, B., Sorkine‐Hornung, A., Sorkine‐Hornung, O., & Schroers, C. (2018) · arXiv (Cornell University) · doi:10.3929/ethz-b-000321437 · Cited by 54
Learning Deployable Navigation Policies at Kilometer Scale from a Single TraversalPreprint
Bruce, J., Sünderhauf, N., Mirowski, P., Hadsell, R. T., & Milford, M. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05211 · Cited by 39
“We mitigate this problem by training the agent on a curriculum of gradually increasing difficulty [30, 31, 17].”
Dynamic Curriculum Learning for Imbalanced Data ClassificationPreprint
Wang, Y., Gan, W., Yang, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06783 · Cited by 33
“The idea of curriculum learning was originally proposed in [1], it demonstrates that the strategy of learning from easy to hard significantly improves the generalization of the deep model.”
BackgroundAccelerating Deep Learning by Focusing on the Biggest LosersPreprint
Jiang, A. H., Wong, D. L. K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M. A., Lipton, Z. C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00762 · Cited by 181
“These techniques, however, do not target and analyze training speedup (Bengio et al., 2009; Chang et al., 2017; Katharopoulos & Fleuret, 2017; Kumar et al., 2010), often adding overhead to the training process by, e.g., training an additional DNN (Katharopoulos & Fleuret, 2017; Jiang et al., 2018;…”
BackgroundDiverseDepth: Affine-invariant Depth Prediction Using Diverse DataPreprint
Yin, W., Wang, X., Shen, C., Liu, Y., Tian, Z., Xu, S., Sun, C., & Renyin, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00569 · Cited by 96
“Several works have demonstrated that curriculum learning [2,15,29] can boost the performance of deep learning methods.”
BackgroundBERT-of-Theseus: Compressing BERT by Progressive Module ReplacingPreprint
Xu, C., Zhou, W., Ge, T., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02925 · Cited by 33
“Although setting a constant replacement rate p can meet the need for compressing a model, we further highlight a Curriculum Learning [1] driven replacement scheduler, which helps progressively substitute the modules in a model.”
BackgroundMethodCLNode: Curriculum Learning for Node Classification
Wei, X., Gong, X., Zhan, Y., Du, B., Luo, Y., & Hu, W. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570385 · Cited by 40
“Previous studies [1, 32, 33] have shown that curriculum learning improves generalization capacity and guides the modelto-wards a better parameter space.”
BackgroundMethodInfluentialMadani, M., Motameni, H., & Roshani, R. (2023) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622023500347 · Cited by 40
Utilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports
Zhu, Q., Mathai, T. S., Mukherjee, P., Peng, Y., Summers, R. M., & Lu, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_19 · Cited by 77
“One possible way to address this issue is to use active learning [21] or curriculum learning [22] methods to differentiate different types of samples and better train the machine learning models.”
Robust and Scalable Vehicle Re-Identification via Self-Supervision
Khorramshahi, P., Shenoy, V., & Chellappa, R. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00558 · Cited by 43
“To address this issue, a variational auto-encoder model was developed in [19, 20, 36] to generate coarse vehicle images and obtain self-supervised saliency maps which highlight identity-dependant information to either adjust images directly or excite intermediate feature maps of an underlying DNN, inspired by the idea…”
MethodAdaptive SVM+: Learning with Privileged Information for Domain Adaptation
Sarafianos, N., Vrigkas, M., & Kakadiaris, I. A. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.313 · Cited by 38
“Curriculum learning [4, 22, 26], which is the learning strategy that implements this paradigm, employs the prior knowledge learned from the first “easier” tasks to improve the performance on “harder” ones that are learned at a later stage.”
BackgroundJoint Recognition of Handwritten Text and Named Entities with a Neural End-to-End Model
Carbonell, M., Villegas, M., Fornés, A., & Lladós, J. (2018) · IAPR International Workshop on Document Analysis Systems (DAS) · doi:10.1109/das.2018.52 · Cited by 33
“Finally, another concept that can help to improve the quality of our models’ prediction is curriculum learning [13].”
BackgroundWeakly- and Semi-supervised Faster R-CNN with Curriculum Learning
Wang, J., Wang, X., & Liu, W. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8546088 · Cited by 33
“The concept of curriculum learning is proposed by Bengio et al. [10], indicating that learning from easy to hard examples can be beneficial.”
BackgroundSlim DensePose: Thrifty Learning From Sparse Annotations and Motion Cues
Neverova, N., Thewlis, J., Güler, R. A., Kokkinos, I., & Vedaldi, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01117 · Cited by 32
“We assume that this is due to both the larger variability of images in the COCO training set, as well as the cleaner nature of COCO images (blur-free, larger resolution), which is known to assist training [1].”
BackgroundLalor, J. P., Wu, H., & Yu, H. Q. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1434 · Cited by 88
“Curriculum learning (CL) is a training procedure where models are trained to learn simple concepts before more complex concepts are introduced (Bengio et al., 2009).”
BackgroundSchilling, M., Konen, K., Ohl, F. W., & Korthals, T. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341754 · Cited by 42
“In the future, we want to extend training to more diverse and demanding tasks, similar to curriculum learning [40].”
BackgroundA Quadruple Diffusion Convolutional Recurrent Network for Human Motion Prediction
Men, Q., Ho, E. S. L., Shum, H. P. H., & Leung, H. W. H. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3038145 · Cited by 32
“To detect the error, they suggested curriculum learning [33] to increasingly perturb input to mimic the distribution of the noisy prediction.”
BackgroundLi, X., Lei, L., Sun, Y., & Kuang, G. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3048332 · Cited by 46
“Finally, inspired by Jin et al. [34], we consider the network training from the perspective of curriculum learning [35].”
MethodProgressive Multi-Granularity Training for Non-Autoregressive Translation
Ding, L., Wang, L., Liu, X., Wong, D. F., Tao, D., & Tu, Z. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.247 · Cited by 39
Dialogue Response Selection with Hierarchical Curriculum Learning
Su, Y., Cai, D., Zhou, Q. S., Lin, Z., Baker, S., Cao, Y., Shi, S., Collier, N., & Wang, Y. (2021) · doi:10.18653/v1/2021.acl-long.137 · Cited by 43
Deep reinforcement learning with credit assignment for combinatorial optimization
Yan, D., Weng, J., Huang, S., Li, C., Zhou, Y., Su, H., & Zhu, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108466 · Cited by 37
Unsupervised domain adaptation via progressive positioning of target-class prototypes
Du, Y., Zhou, Y., Xie, Y., Zhou, D., Shi, J., & Lei, Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110586 · Cited by 31
Park, B., Cho, Y., Lee, G., Lee, S. M., Cho, Y.-H., Lee, E. S., Lee, K. H., Seo, J. B., & Kim, N. (2019) · Scientific Reports · doi:10.1038/s41598-019-51832-3 · Cited by 31
Image Caption Generation with Text-Conditional Semantic Attention.Preprint
Zhou, L., Xu, C., Koch, P. A., & Corso, J. J. (2016) · arXiv (Cornell University) · Cited by 34
A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari GamesPreprint
Leibfried, F., Kushman, N., & Hofmann, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07078 · Cited by 43
“Following (Oh et al., 2015) and (Michalski et al., 2014), we apply a curriculum learning (Bengio et al., 2009) scheme by successively increasing K in the course of training such that the network initially learns to predict over a short time horizon and becomes fine-tuned on longer-term predictions…”
MethodMeta-Transfer Learning for Few-Shot LearningPreprint
Sun, Q., Liu, Y., Chua, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02391 · Cited by 32
“Curriculum learning [3] and hard negative mining [46] both suggest that faster convergence and stronger performance can be achieved by a better arrangement of training data.”
BackgroundRouting Networks and the Challenges of Modular and Compositional Computation
Rosenbaum, C., Cases, I., Riemer, M., & Klinger, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.12774 · Cited by 32
“Another strategy that can be applied is curriculum learning [Bengio et al., 2009] as is done by Chang et al.”
BackgroundCurriculum Audiovisual LearningPreprint
Hu, D., Wang, Z., Xiong, H., Wang, D., Nie, F., & Dou, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09414 · Cited by 44
“Curriculum learning Compared with a disordered arrangement, starting from easier samples or tasks then gradually increasing the difficulty level could contribute to better learning performance [9, 6], which is called curriculum learning and has been widely applied in image classification [14], natural language process…”
BackgroundMa, S., Song, K., Niu, M., Tian, H., & Yan, Y. (2024) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-022-02051-7 · Cited by 31
“The proposal of the transferable curriculum is inspired by curriculum learning (Bengio et al., 2009),whichpromotesnetworkconvergencebyreorganizing the training order of samples through an easy-to-hard principle.”
MELT: Mutual Enhancement of Long-Tailed User and Item for Sequential Recommendation
Kim, K., Hyun, D., Yun, S., & Park, C. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591725 · Cited by 49
“The CL [2] strategy mimics the human’s learning process and can eventually enhance the knowledge obtained from head users and head items by stabilizing the knowledgetransfer process.”
BackgroundYi, X., Fu, Y., Yu, J., Liu, R., Zhang, H., & Hua, R. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3507073 · Cited by 40
“The most relevant approach to our proposed LHR-RFL is Curriculum Learning [41], [42], [43], which trains deep learning models in a predetermined sequence, starting from simpler samples and gradually progressing towards more challenging ones.”
Self-Supervised Learning of Pose Embeddings from Spatiotemporal Relations in Videos
Sümer, Ö., Dencker, T., & Ommer, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.461 · Cited by 29
“Moreover, the use of curriculum learning [7] and repetition mining refine and guide our self-supervised tasks to learn stronger temporal features.”
BackgroundLearning Object Detectors With Semi-Annotated Weak Labels
Zhang, D., Han, J., Guo, G., & Zhao, L. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2884173 · Cited by 31
Semisupervised Hotspot Detection With Self-Paced Multitask Learning
Chen, Y., Lin, Y., Gai, T., Su, Y., Wei, Y., & Pan, D. Z. (2019) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2019.2912948 · Cited by 32
GANai: Standardizing CT Images using Generative Adversarial Network with Alternative Improvement
Liang, G., Fouladvand, S., Zhang, J., Brooks, M., Jacobs, N., & Chen, J. (2019) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2019.8904763 · Cited by 41
“Since the components of GANai are trained separately, one idea motivated by the curriculum learning is to increase model training efficiency by training G and D using different data [46].”
BackgroundVisual Tracking by Means of Deep Reinforcement Learning and an Expert Demonstrator
Dunnhofer, M., Martinel, N., Foresti, G. L., & Micheloni, C. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00282 · Cited by 41
“In addition to the guiding process done by the imitating learners using the expert demonstrations, we designed a curriculum learning strategy [1] to further facilitate the training.”
MethodTsarfaty, R., Bareket, D., Klein, S., & Seker, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.660 · Cited by 56
“Interleving information from all layers may be done by all at once (e.g., via Multi-Task Learning (Caruana, 1997)) or by gradually adding complexity (e.g., via Curriculum Learning (Bengio et al., 2009)).”
BackgroundThe Return of Lexical Dependencies: Neural Lexicalized PCFGs
Zhu, H., Bisk, Y., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00337 · Cited by 28
“Curriculum Learning We also apply curriculum learning (Bengio et al., 2009; Spitkovsky et al., 2010) to learn the grammar gradually.”
Method
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.