Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
“Choosing which triplets to use turns out to be very important for achieving good performance and, inspired by curriculum learning [1], we present a novel online negative exemplar mining strategy which ensures consistently increasing difficulty of triplets as the network trains.”
BackgroundA survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
“This is nicely related to the curriculum learning idea [24] that it may be much easier to learn simpler concepts first and then build”
BackgroundSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationAn overview of gradient descent optimization algorithms
Ruder, S. (2016) · arXiv · doi:10.48550/arxiv.1609.04747 · Cited by 4,962
“supplying the training examples in a meaningful order may actually lead to improved performance and better convergence. The method for establishing this meaningful order is called Curriculum Learning [3]. Zaremba and Sutskever [21] were only able to train LSTMs to evaluate simple programs using Curriculum Learning and…”
MethodNatural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
“(Bengio et al., 2009) provides a more detailed discussion of this, the (as yet, poorly understood) “curriculum” process.”
BackgroundMethodSelf-citationGraph Convolutional Neural Networks for Web-Scale Recommender Systems
Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W. L., & Leskovec, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219890 · Cited by 3,733
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
“The strategy of training machine learning models on a series of gradually increasing tasks is known as curriculum learning [5].”
BackgroundContinual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342
“Authors in [63] formalised training strategies in the context of machine learning that show that improvements in generalisation and in speed of convergence of the training.”
BackgroundPractical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
“The hypothesis proposed in (Bengio et al., 2009) is that curriculum learning can act similarly to a continuation method, i.”
BackgroundInfluentialSelf-citationDeep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
“In order to make training more stable, we experiment with a training curriculum (Bengio et al., 2009; Zaremba & Sutskever, 2014), which accelerates training and results in better generalization as well.”
MethodWhy Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010) · Journal of Machine Learning Research · Cited by 2,631
Self-citationDeep Learning for Image Super-Resolution: A Survey
Wang, Z., Chen, J., & Hoi, S. C. H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2982166 · Cited by 1,969
“Curriculum learning [148] refers to starting from an easier task and gradually increasing the difficulty.”
Background(2010) · Cited by 1,906
Self-citationA Continual Learning Survey: Defying Forgetting in Classification Tasks
Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., & Tuytelaars, T. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3057446 · Cited by 1,647
“Our hypothesis resembles curriculum learning [95], implying knowledge is better captured starting with the general easier tasks followed by harder specific tasks.”
BackgroundA continual learning survey: Defying forgetting in classification tasks
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., & Tuytelaars, T. (2019) · arXiv · doi:10.48550/arxiv.1909.08383 · Cited by 1,647
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621
“Here, we propose a curriculum learning approach [7] to gently bridge the gap between training and inference for sequence prediction tasks using recurrent neural networks.”
MethodLearning agile and dynamic motor skills for legged robots
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., & Hutter, M. (2019) · Science Robotics · doi:10.1126/scirobotics.aau5872 · Cited by 1,482
“This technique is related to curriculum learning introduced by Bengio et al. [59], which incrementally introduces samples of more difficulties.”
MethodThe “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471
“This approach can be viewed as “curriculum learning” [2], where simple concepts are taught first, and more complicated concepts are added progressively over time.”
BackgroundHybrid computing using a neural network with dynamic external memory
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412
“After training with curriculum learning 28,29 using graphs and queries with gradually increasing complexity, the networks were tested (with no retraining) on two specific graphs as a test of generalization to realistic data: a symbolic map of the London Underground and an invented family tree.”
MethodSuper-convergence: very fast training of neural networks using large learning rates
Smith, L. N., & Topin, N. (2019) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.2520589 · Cited by 1,335
“Please note that the philosophy behind CLR is a combination of curriculum learning [Bengio et al., 2009] and simulated annealing [Aarts and Korst, 1988], both of which have a long history of use in deep learning.”
BackgroundMeta-Transfer Learning for Few-Shot Learning
Sun, Q., Liu, Y., Chua, T.-S., & Schiele, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00049 · Cited by 1,264
“Curriculum learning [3] and hard negative mining [46] both suggest that faster convergence and stronger performance can be achieved by a better arrangement of training data.”
BackgroundTransformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,296
“Inspired by curriculum learning [375], Nooralahzadeh et al.”
MethodConvolutional Neural Network Architectures for Matching Natural Language Sentences
Hu, B., Lu, Z., Li, H., & Chen, Q. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03244 · Cited by 1,098
“This therefore raises the interesting question about how to balance the representation of matching and the representations of objects, and whether we can guide the learning process through something like curriculum learning [4].”
BackgroundEquivalent-accuracy accelerated neural-network training using analogue memory
Ambrogio, S., Narayanan, P., Tsai, H., Shelby, R. M., Boybat, I., di Nolfo, C., Sidler, S., Giordano, M., Bodini, M., Farinha, N. C. P., et al. (2018) · Nature · doi:10.1038/s41586-018-0180-5 · Cited by 1,151
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133
“This simulates curriculum learning [2] and helps in keeping the forecasted motion close to the manifold of human motion.”
BackgroundDeep Reinforcement Learning for Dialogue Generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1127 · Cited by 1,204
“A curriculum learning strategy is adopted (Bengio et al., 2009) as in Ranzato et al.”
MethodSelf-Paced Learning for Latent Variable Models
Kumar, M., Packer, B., & Koller, D. (2010) · neural information processing systems · Cited by 1,708
Deep Residual Network for Steganalysis of Digital Images
Boroumand, M., Chen, M., & Fridrich, J. (2019) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2871749 · Cited by 1,032
“The detectors for all remaining payloads were built via curriculum training [5] with 50–100k iterations (57–114 epochs) with learning rate r1 and an additional 50k iterations (57 epochs) with the smaller learning rate r2.”
MethodOn Human Motion Prediction Using Recurrent Neural Networks
Martinez, J., Black, M. J., & Romero, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.497 · Cited by 984
“Therefore, they propose to gradually add noise to the input during training (as is common in curriculum learning [6]), which forces the network to be more robust to prediction errors.”
BackgroundHuman-in-the-loop machine learning: a state of the art
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10246-w · Cited by 1,025
“As an outstanding feature of human learning, curriculum, or learning in a meaningful order, has been exploited and transferred to machine learning, which forms the subdiscipline named curriculum learning (CL) (Bengio et al. 2009).”
BackgroundInfluentialThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“In addition, techniques like curriculum learning [240] can enhance the agent’s understanding of relationships between simple and complex tools, such as how complex tools are built on simpler ones and allow agents to effectively discern nuances across various application scenarios and transfer…”
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
“Curriculum learning schemes can also cause improved generalization and faster convergence of agent policies (Bengio et al. 2009).”
BackgroundCooperative Multi-agent Control Using Deep Reinforcement Learning
Gupta, J. K., Egorov, M., & Kochenderfer, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-71682-4_5 · Cited by 874
“This idea is known as curriculum learning [4], and it can be extended to reinforcement learning problems with multiple cooperating agents.”
BackgroundMeta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
“Instead of defining a curriculum by hand [150], meta-learning can automate the”
BackgroundMeta-Learning in Neural Networks: A Survey
Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05439 · Cited by 886
“Instead of defining a curriculum by hand [150], meta-learning can automate the”
Feedback Network for Image Super-Resolution
Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., & Wu, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00399 · Cited by 883
“Curriculum learning[2], which gradually increases the difficulty of the learned target, is well known as an efficient strategy to improve the training procedure.”
BackgroundKolbaek, M., Yu, D., Tan, Z.-H., & Jensen, J. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2726762 · Cited by 820
Learning to Reweight Examples for Robust Deep Learning
Ren, M., Zeng, W., Yang, B., & Urtasun, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09050 · Cited by 740
“Reweighting examples is also related to curriculum learning (Bengio et al., 2009), where the model reweights among many available tasks.”
BackgroundDeep Learning Hierarchical Representations for Image Steganalysis
Ye, J., Ni, J., & Yi, Y. (2017) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2017.2710946 · Cited by 823
“In our work, however, the issue above can be well solved by adopting a curriculum learning [26] or transfer learning strategy [27].”
BackgroundNeural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821
Fadlullah, Z. M., Tang, F., Mao, B., Kato, N., Akashi, O., Inoue, T., & Mizutani, K. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2707140 · Cited by 810
“In other words, in each level, a new transformation of the previously learned features is performed that is served as the input to the next level [28]–[31].”
BackgroundDynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
Socher, R., Huang, E., Pennin, J., Manning, C. D., & Ng, A. Y. (2011) · neural information processing systems · Cited by 958
Smith, L. N. (2018) · arXiv · doi:10.48550/arxiv.1803.09820 · Cited by 790
“It is interesting to note that the 1cycle learning rate policy is a combination of curriculum learning (Bengio et al., 2009) and simulated annealing (Aarts & Korst, 1988), both of which have a long history of use in deep learning.”
BackgroundDeep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
“…Bertsekas and Tsitsiklis, 1996; Puterman, 2005; Littman, 2015; Kaelbling et al., 1996); deep learning (LeCun et al., 2015; Goodfellow et al., 2016; Bengio, 2009; Deng and Dong, 2014); machine learning (Jordan and Mitchell, 2015; Hastie et al., 2009; Bishop, 2011; Murphy, 2012; James et al.,…”
Wang, L., & Yoon, K.-J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3055564 · Cited by 752
“[131] generalize the selection of distillation positions by employing variational information maximization [18], curriculum learning [20], adversarial learning [65], similarity-presentation in representation learning [91], muti-task learning [43], reinforcement learning [163].”
MethodClass-Incremental Learning: Survey and Performance Evaluation on Image Classification
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., & van de Weijer, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3213473 · Cited by 740
“This setting is different from the one proposed in Curriculum Learning [131], since the objective here is not to find the best order to learn tasks efficiently, but rather to analyze incremental learning settings (in which the order is not known in advance) and analyze the robustness of methods under…”
MethodDeep learning with noisy labels: Exploring techniques and remedies in medical image analysis
Karimi, D., Dou, H., Warfield, S. K., & Gholipour, A. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101759 · Cited by 706
“Curriculum learning, first proposed by [148], is based on training a model with examples of increasing complexity or difficulty.”
BackgroundLearning scheduling algorithms for data processing clusters
Mao, H., Schwarzkopf, M., Venkatakrishnan, S. B., Meng, Z., & Alizadeh, M. (2019) · Proceedings of the ACM Special Interest Group on Data Communication · doi:10.1145/3341302.3342080 · Cited by 723
“The concept of gradually increasing job sequence length—and therefore, problem complexity—during training is a realization of curriculum learning [8] for cluster scheduling.”
BackgroundTaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation
Luo, Y., & Mesgarani, N. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462116 · Cited by 686
“We apply curriculum training strategy [24] in a similar fashion with [3, 5].”
MethodA Survey on Curriculum Learning
Wang, X., Chen, Y., & Zhu, W. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3069908 · Cited by 692
“Specifically, CL helps supervised setting mainly by guiding when (i) the task is hard [20], [83], (ii) parts of the training data are difficult to learn [6], [41], (iii) the target distribution heavily shifts from training distribution [103], [121].”
BackgroundMethodInfluentialCost-Effective Active Learning for Deep Image Classification
Wang, K., Zhang, D., Li, Y., Zhang, R., & Lin, L. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2589879 · Cited by 663
“For the problem of keep the model stable in the training stage, many works [34], [35] are proposed in recent years inspired by the learning process of humans that gradually include samples into training from easy to complex.”
BackgroundA review: Deep learning for medical image segmentation using multi-modality fusion
Zhou, T., Ruan, S., & Canu, S. (2019) · Array · doi:10.1016/j.array.2019.100004 · Cited by 656
“Based on curriculum learning [4], which means gradually increasing the difficulty of training tasks, they applied an effective strategy to improve the convergence quality of the model by training the first task only until the loss curve tends to flatten, then the first data and the second data are concatenated along t…”
MethodUnsupervised Person Re-identification: Clustering and Fine-tuning
Fan, H., Zheng, L., Yan, C., & Yang, Y. (2018) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3243316 · Cited by 636
“Inspired by this process, the theory of Curriculum Learning (CL) [41] is proposed to accelerate the speed of convergence of the training process to a minimum.”
MethodDynamic R-CNN: Towards High Quality Object Detection via Dynamic Training
Zhang, H., Chang, H., Ma, B., Wang, N., & Chen, X. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58555-6_16 · Cited by 651
“Besides, Curriculum Learning [1] and Self-paced Learning [20] focus on improving the training order of the examples.”
BackgroundEmergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657
“The idea of curricula is long-standing in the machine learning literature (e.g. [30]).”
BackgroundAdvanced Deep-Learning Techniques for Salient and Category-Specific Object Detection: A Survey
Han, J., Zhang, D., Cheng, G., Liu, N., & Xu, D. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2749125 · Cited by 621
“To address the noise issue in human annotation, further research could design weighting-based learning mechanisms (such as the models based on self-paced learning [94], [96] and curriculum learning [97], [98]) to further improve the learning robustness when the human annotations of the learned object categories are no…”
MethodUnity: A General Platform for Intelligent Agents
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.02627 · Cited by 626
“This enables evaluation of an agent on a train/test split of environment variations and facilitates the creation of curriculum learning scenarios (Bengio et al., 2009).”
BackgroundLi, Y., Liu, S., Yang, J., & Yang, M.-H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.624 · Cited by 645
“To effectively train our network, we use the curriculum strategy [3] by gradually increasing the difficulty level and network scale.”
MethodCurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition
Huang, Y., Wang, Y., Tai, Y., Liu, X., Shen, P., Li, S., Li, J., & Huang, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00594 · Cited by 580
“Motivated by the nature of human learning that easy cases are learned first and then come the hard ones [2], our CurricularFace incorporates the idea of Curriculum Learning (CL) into face recognition in an adaptive manner, which differs from the traditional CL in two aspects.”
BackgroundMethodDense reinforcement learning for safety validation of autonomous vehicles
Feng, S., Sun, H., Yan, X., Zhu, H., Zou, Z., Shen, S., & Liu, H. X. (2023) · Nature · doi:10.1038/s41586-023-05732-2 · Cited by 587
“The connections can even be added in the form of curriculum learning 46 , which can guide the information propagation gradually.”
BackgroundAdaFace: Quality Adaptive Margin for Face Recognition
Kim, M., Jain, A. K., & Liu, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01819 · Cited by 584
“Typically, assigning different importance to different difficulties of samples is done by looking at the training progression (curriculum learning) [1,13].”
BackgroundVSE++: Improving Visual-Semantic Embeddings with Hard Negatives
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05612 · Cited by 543
Generating Diverse and Natural 3D Human Motions from Text
Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., & Cheng, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00509 · Cited by 575
“To address the variable length sequence-to-sequence generation task, our training process utilizes both curriculum learning [3] and scheduled sampling [2] strategies, as follows.”
MethodEdge Computing with Artificial Intelligence: A Machine Learning Perspective
Hua, H., Li, Y., Wang, T., Dong, N., Li, W., & Cao, J. (2023) · ACM Computing Surveys · doi:10.1145/3555802 · Cited by 563
“DRL combined with curriculum learning [111] has been used to realize a bottom-up energy management scheme [110].”
MethodCurriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Zhang, Y., David, P., & Gong, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.223 · Cited by 494
“Accordingly, we propose a curriculum-style [4] domain adaptation approach.”
MethodDeep Adaptive Image Clustering
Chang, J., Wang, L., Meng, G., Xiang, S., & Pan, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.626 · Cited by 535
“Inspired by curriculum learning [2], we attempt to control the clustering procedure such that the samples are increasingly selected.”
BackgroundMethodMask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes
Lyu, P., Liao, M., Yao, C., Wu, W., & Bai, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_5 · Cited by 522
Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning
Long, P., Fanl, T., Liao, X., Liu, W., Zhang, H., & Pan, J. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461113 · Cited by 538
“Inspired by the curriculum learning paradigm [27], we propose a two-stage training process, which accelerates the policy to converge to a satisfying solution, and gets higher rewards than the policy trained from scratch with the same number of epoch (as shown in Figure 5).”
MethodBilingual Word Embeddings for Phrase-Based Machine Translation
Zou, W. Y., Socher, R., Cer, D., & Manning, C. D. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1141 · Cited by 518
“Through large-scale curriculum training (Bengio et al., 2009), we obtain bilingual distributed representations which lie in the same feature space.”
BackgroundMethodDeep Learning of Representations: Looking Forward
Bengio, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39593-2_1 · Cited by 513
“Another option, following the idea of curriculum learning (Bengio et al., 2009), is to provide guidance ourselves to learning machines (as exemplified in the toy example of Gulcehre and Bengio (2013)), by “teaching them” gradually more complex concepts to help them understand the world around us (keeping in mind that…”
BackgroundSelf-citationCo-Saliency Detection via a Self-Paced Multiple-Instance Learning Framework
Zhang, D., Meng, D., & Han, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2567393 · Cited by 508
“In addition, this regularizer conducts either 1 or 0 (i.e., selected in training or not) for the weight v ðkÞ i imposed on instance x ðkÞ i , by judging whether its loss value is smaller than the pace parameter or not [17], [38].”
BackgroundMethodInfluentialModel-based Reinforcement Learning: A Survey
M. Moerland, T., Broekens, J., Plaat, A., & M. Jonker, C. (2023) · Foundations and Trends in Machine Learning · doi:10.1561/2200000086 · Cited by 524
Strategies for training large scale neural network language models
Mikolov, T., Deoras, A., Povey, D., Burget, L., & Cernocky, J. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163930 · Cited by 509
“For example, in [15] H = 100 is used when the amount of the training data is over 600M words.”
BackgroundMethodDroNet: Learning to Fly by Driving
Loquercio, A., Maqueda, A. I., del-Blanco, C. R., & Scaramuzza, D. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2795643 · Cited by 514
“This can be seen as a particular form of curriculum learning [19].”
BackgroundMitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., et al. (2018) · Communications of the ACM · doi:10.1145/3191513 · Cited by 507
“Work on curriculum learning (Bengio et al. 2009) explores potential synergies across sets or sequences of learning tasks.”
BackgroundST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation
Yang, L., Zhuo, W., Qi, L., Shi, Y., & Gao, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00423 · Cited by 518
“Moreover, both aforementioned works adopt the plain training pipeline, whereas we further propose the ST++ to safely exploit unlabeled images in a curriculum learning manner [5].”
BackgroundSelf-Paced Curriculum Learning
Jiang, L., Meng, D., Zhao, Q., Shan, S., & Hauptmann, A. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9608 · Cited by 480
Time-series Generative Adversarial Networks
Yoon, J., Jarrett, D., & van der Schaar, M. (2019) · neural information processing systems · Cited by 2,070
Deep joint demosaicking and denoising
Gharbi, M., Chaurasia, G., Paris, S., & Durand, F. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982399 · Cited by 473
RadioUNet: Fast Radio Map Estimation With Convolutional Neural Networks
Levie, R., Yapar, Ç., Kutyniok, G., & Caire, G. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3054977 · Cited by 488
“One approach for achieving this is called curriculum learning [50].”
BackgroundPTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11259 · Cited by 478
FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., & Shinozaki, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08263 · Cited by 466
“To address these issues, we propose Curriculum Pseudo Labeling (CPL), a curriculum learning [15] strategy to take into account the learning status of each class for semi-supervised learning.”
BackgroundMethodEmbracing Change: Continual Learning in Deep Neural Networks
Hadsell, R., Rao, D., Rusu, A. A., & Pascanu, R. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.004 · Cited by 467
“Compared with curriculum learning [12,13], the learner does not control the task ordering.”
BackgroundAbdelrahman, G., Wang, Q., & Nunes, B. (2023) · ACM Computing Surveys · doi:10.1145/3569576 · Cited by 468
Weakly Supervised Object Localization with Multi-Fold Multiple Instance Learning
Cinbis, R. G., Verbeek, J., & Schmid, C. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2535231 · Cited by 420
“For example, curriculum learning (CL) [5], which is a conceptual framework, suggests that training can be improved by initializing a model with easy examples, and then, gradually utilizing more complex ones.”
BackgroundThe Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448
“C L ] 2 0 Ju n 20 vanced attention mechanisms [Xiong et al., 2017], attention networks [Vaswani et al., 2017], question answering [Seo et al., 2017, Xiong et al., 2018, Yu et al., 2016, Weissenborn et al., 2017], and curriculum learning [Bengio et al., 2009].”
BackgroundMancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-Identification
Wang, C., Zhang, Q., Huang, C., Liu, W., & Wang, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_23 · Cited by 410
“Inspired by curriculum learning [5], we propose a new sampling way named curriculum sampling.”
BackgroundMethodAction-Conditional Video Prediction using Deep Networks in Atari Games
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.08750 · Cited by 419
“We found that this curriculum learning [6] approach is necessary to stabilize the training.”
BackgroundSoviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01611-x · Cited by 431
“[17] are the first to formalize the easy-to-hard training strategies in the context of machine learning, proposing the curriculum learning (CL) paradigm.”
BackgroundMethodInfluentialMulti-Task Learning with Deep Neural Networks: A Survey
Crawshaw, M. (2020) · arXiv · doi:10.48550/arxiv.2009.09796 · Cited by 428
“Despite the fact that most methods use these baseline task schedulers, it is a well known fact that optimized task scheduling can significantly improve model performance (Bengio et al., 2009).”
MethodLight Gated Recurrent Units for Speech Recognition
Ravanelli, M., Brakel, P., Omologo, M., & Bengio, Y. (2018) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2017.2762739 · Cited by 411
“Moreover, the sorting approach exploits a curriculum learning strategy [62] that has been shown to slightly improve”
BackgroundSelf-citation12-in-1: Multi-Task Vision and Language Representation Learning
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01045 · Cited by 410
“Inspired by prior multi-task literature [4] [31], we experimented with both curriculum and anti-curriculum strategies based on task difficulty.”
MethodSelf-supervised learning for recommender systems: A survey
Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3282907 · Cited by 413
“Particularly, collaborative methods, where the generative/predictive task serves the contrastive task by dynamically generating samples, offer significant advantages in training effectiveness over static counterparts [64], [106], [134].”
BackgroundExploit the Unknown Gradually: One-Shot Video-Based Person Re-identification by Stepwise Learning
Wu, Y., Lin, Y., Dong, X., Yan, Y., Ouyang, W., & Yang, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00543 · Cited by 383
“Curriculum Learning (CL) is proposed in [2], which progressively obtains knowledge from easy to hard samples in a pre-defined scheme.”
BackgroundRingeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., Messner, E.-M., et al. (2019) · Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop · doi:10.1145/3347320.3357688 · Cited by 409
“respective GRU outputs. The model is implemented using a Pytorchframework and is trained with an Adamoptimiser. As previous studies have shown the benefits of training a network following a curriculum [7, 37], whereinstances aregraduallypresentedinincreasing levelofdiffi- culty, we implemented this approach using the fol…”
MethodDeep Reinforcement Learning for Dialogue GenerationPreprint
Li, J., Monroe, W., Ritter, A., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01541 · Cited by 405
“A curriculum learning strategy is adopted (Bengio et al., 2009) as in Ranzato et al.”
MethodMulti-Agent Reinforcement Learning: A Review of Challenges and Applications
Canese, L., Cardarilli, G. C., Di Nunzio, L., Fazzolari, R., Giardino, D., Re, M., & Spanò, S. (2021) · Applied Sciences · doi:10.3390/app11114948 · Cited by 403
“In addition, the scalability of the PS-TRPO algorithm was addressed using curriculum learning [38], in which the agents had to solve a number of sub-tasks that increased in difficulty in order to coordinate with an increasing number of agents.”
Method
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.