Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 201–300 of 5,240
DMT: Dynamic mutual training for semi-supervised learning
Feng, Z., Zhou, Q., Gu, Q., Tan, X., Cheng, G., Lu, X., Shi, J., & Ma, L. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108777 · Cited by 190
“Furthermore, inspired by curriculum learning, or easy to hard [14], we apply DMT iteratively, each time considering more unlabeled data, to gradually increase performance.”
BackgroundMethodST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection
Yang, J., Shi, S., Wang, Z., Li, H., & Qi, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01023 · Cited by 198
“[39, 9, 8] employ curriculum learning [2] and separate cases by their difficulties to realize local sample-level curriculum.”
BackgroundAutomated cattle counting using Mask R-CNN in quadcopter vision system
Xu, B., Wang, W., Falzon, G., Kwan, P., Guo, L., Chen, G., Tait, A., & Schneider, D. (2020) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2020.105300 · Cited by 196
End-to-End Relation Extraction using LSTMs on Sequences and Tree StructuresPreprint
Miwa, M., & Bansal, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.00770 · Cited by 194
“Our model also incorporates curriculum learning (Bengio et al., 2009) and scheduled sampling (Bengio et al.”
MethodImproved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning
Shaked, A., & Wolf, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.730 · Cited by 185
“We believe that the reflective loss can be extended to other problems and applications such as the gradual learning schemes self-paced [19] and curriculum [1] learning.”
BackgroundEidetic 3D LSTM: A Model for Video Prediction and Beyond
Wang, Y., Jiang, L., Yang, M. H., Li, L.-J., Long, M., & Fei-Fei, L. (2019) · International Conference on Learning Representations · Cited by 371
Graph-based semi-supervised learning: A review
Chong, Y., Ding, Y., Yan, Q., & Pan, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.12.130 · Cited by 194
Unsupervised Tracklet Person Re-Identification
Li, M., Zhu, X., & Gong, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2903058 · Cited by 185
“Hence, the CCTA loss is in a curriculum leaning spirit [58].”
BackgroundWhat Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
Mayer, N., Ilg, E., Fischer, P., Hazirbas, C., Cremers, D., Dosovitskiy, A., & Brox, T. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1082-6 · Cited by 191
“This is strongly connected to curriculum learning, which has previously been applied to shape recognition (Bengio et al, 2009; Elman, 1993) and other (non-vision) tasks.”
MethodAction-conditional video prediction using deep networks in Atari games
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · neural information processing systems · Cited by 561
Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Castrejon, L., Aytar, Y., Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.321 · Cited by 188
“Our approach is a form of curriculum learning [4].”
Robust Optimization for Deep Regression
Belagiannis, V., Rupprecht, C., Carneiro, G., & Navab, N. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.324 · Cited by 179
“Furthermore, weighting training samples also conforms with the idea of curriculum [5] and self-paced learning [22], where each training sample has different contribution to the minimization depending on its error.”
BackgroundFew-Example Object Detection with Model Communication
Dong, X., Zheng, L., Ma, F., Yang, Y., & Meng, D. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2844853 · Cited by 167
“[48] first propose a learning paradigm in which organizing the examples in a meaningful order significantly improves the performance.”
BackgroundMethodSakaridis, C., Dai, D., & Van Gool, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045882 · Cited by 186
“This is in line with the concept of curriculum learning [69].”
ResultAdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles
Wang, J., Pun, A., Tu, J., Manivasagam, S., Sadat, A., Casas, S., Ren, M., & Urtasun, R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00978 · Cited by 186
“First, we propose a curriculum learning (CL) [5] baseline where we first train on standard examples till convergence (easy examples), and then train on real challenging scenarios selected based on reachable actors (Sec 4.”
BackgroundA robot arm digital twin utilising reinforcement learning
Matulis, M., & Harvey, C. (2021) · Computers & Graphics · doi:10.1016/j.cag.2021.01.011 · Cited by 183
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184
Knowledge-guided machine learning can improve carbon cycle quantification in agroecosystems
Liu, L., Zhou, W., Guan, K., Peng, B., Xu, S., Tang, J., Zhu, Q., Till, J., Jia, X., Jiang, C., et al. (2024) · Nature Communications · doi:10.1038/s41467-023-43860-5 · Cited by 189
Task-sequencing meta learning for intelligent few-shot fault diagnosis with limited data
Hu, Y., Liu, R., Li, X., Chen, D., & Hu, Q. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3112504 · Cited by 180
“Consideringthehumanlearningprocessgenerally followstheorderfromeasytodifficult,curriculumlearning[34] advocatesthatthemodelshouldstartlearningfromeasysamples and gradually transition to complex samples.”
BackgroundStarCraft Micromanagement With Reinforcement Learning and Curriculum Transfer Learning
Shao, K., Zhu, Y., & Zhao, D. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2018.2823329 · Cited by 178
“The initial tasks are used to guide the learner so that it can perform better on the final task [53].”
MethodDecentralized Multi-Agent Pursuit Using Deep Reinforcement Learning
de Souza, C., Newbury, R., Cosgun, A., Castillo, P., Vidolov, B., & Kuli, D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3068952 · Cited by 181
“Curriculum learning [28] is a learning paradigm to help improve speed of convergence and reduce local minima by gradually increasing the complexity of training data.”
BackgroundDeep clustering and conventional networks for music separation: Stronger together
Luo, Y., Chen, Z., Hershey, J. R., Le Roux, J., & Mesgarani, N. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952118 · Cited by 168
“Similarly to [13], we also applied a curriculum learning strategy [21], where we first train the network on segments of 100 frames, then train on segments of 500 frames.”
MethodSpatial Memory for Context Reasoning in Object Detection
Chen, X., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.440 · Cited by 167
“For efficiency, we also follow a curriculum learning [8] strategy: bootstrap a SMN of more iterations (e.”
MethodDeep reinforcement learning for drone navigation using sensor data
Hodge, V. J., Hawkins, R., & Alexander, R. (2021) · Neural Computing and Applications · doi:10.1007/s00521-020-05097-x · Cited by 178
“There are several techniques including: transfer learning, multitask learning and curriculum learning [8].”
MethodPRIMAL$_2$: Pathfinding Via Reinforcement and Imitation Multi-Agent Learning - Lifelong
Damani, M., Luo, Z., Wenzel, E., & Sartoretti, G. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3062803 · Cited by 177
“between 3 and 21. We find that uniformly sampling these parameters works well in practice. Curriculum learning with increasing difficulty of environments has also been shown to be effective in practice [31]. However, our experiments with implementations of curriculum learning did not yield significant performance improve…”
BackgroundKnowledge Distillation via Route Constrained Optimization
Jin, X., Peng, B., Wu, Y., Liu, Y., Liu, J., Liang, D., Yan, J., & Hu, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00143 · Cited by 187
“In curriculum learning [1] the sequence of learning is generated by spliting the X in to several “lessons” with different hardness depending on a predefined criterion.”
BackgroundInfluentialLearning the easy things first: Self-paced visual category discovery
Lee, Y. J., & Grauman, K. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995523 · Cited by 168
“Recent work explores how a curriculum that sorts training data into easier and harder groups can lead to better local minima when training a classifier with a non-convex criterion [3].”
BackgroundHyperparameter optimization in learning systems
Andonie, R. (2019) · Journal of Membrane Computing · doi:10.1007/s41965-019-00023-0 · Cited by 175
“According to [8], these ordering strategies can be seen as a special form of transfer learning where the initial tasks are used to guide the learner so that it will perform better on the final task.”
BackgroundSupervision by Fusion: Towards Unsupervised Learning of Deep Salient Object Detector
Zhang, D., Han, J., & Zhang, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.436 · Cited by 169
“The learning confidence maps are used to provide the superpixel-wise learning weight for the learning process, which naturally compiles to the modern self-paced learning (SPL) and curriculum learning (CL) regimes [14, 42, 9, 2].”
MethodDense Captioning with Joint Inference and Visual Context
Yang, L., Tang, K., Yang, J., & Li, L.-J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.214 · Cited by 167
Charagram: Embedding Words and Sentences via Character n-grams
Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1157 · Cited by 168
On the Convergence and Robustness of Adversarial Training
Wang, Y., Ma, X., Bailey, J., Yi, J., Zhou, B., & Gu, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.08304 · Cited by 159
“Curriculum learning (Bengio et al., 2009) is a learning paradigm in which a model learns from easy examples first then gradually learns from more and more difficult examples.”
BackgroundA survey on multi-agent reinforcement learning and its application
Ning, Z., & Xie, L. (2024) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2024.02.003 · Cited by 173
Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
Forestier, S., Portelas, R., Mollard, Y., & Oudeyer, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02190 · Cited by 166
Marucci, M., Di Flumeri, G., Borghini, G., Sciaraffa, N., Scandola, M., Pavone, E. F., Babiloni, F., Betti, V., & Aricò, P. (2021) · Scientific Reports · doi:10.1038/s41598-021-84196-8 · Cited by 170
Dynamic-Structured Semantic Propagation Network
Liang, X., Xing, E., & Zhou, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00085 · Cited by 157
“In the spirit of curriculum learning [2] that gradually increases the target difficulty levels and exploits previously learned knowledge for learning new fine-grained”
BackgroundBERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Xu, C., Zhou, W., Ge, T., Wei, F., & Zhou, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.633 · Cited by 168
“Although setting a constant replacement rate p can meet the need for compressing a model, we further highlight a Curriculum Learning [1] driven replacement scheduler, which helps progressively substitute the modules in a model.”
BackgroundMethodYao, X., Feng, X., Han, J., Cheng, G., & Guo, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2991407 · Cited by 169
A comprehensive review of deep learning-based single image super-resolution
Bashir, S. M. A., Wang, Y., Khan, M., & Niu, Y. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.621 · Cited by 172
“Curriculum learning In the Curriculum learning technique (Bengio et al., 2009), the method adapts itself to the variable difficulty of tasks, i.”
MethodRetractedDeep Audio-visual Learning: A Survey
Zhu, H., Luo, M.-D., Wang, R., Zheng, A.-H., & He, R. (2021) · International Journal of Automation and Computing · doi:10.1007/s11633-021-1293-0 · Cited by 174
“The researchers applied the curriculum learning strategy [118] to this task and divided the samples into four categories: positives (the corresponding audio-video pairs), easy negatives (audio and video clips originating from different videos), difficult negatives (audio and video clips originating from the same video…”
MethodCount-ception: Counting by Fully Convolutional Redundant Counting
Cohen, J. P., Boucher, G., Glastonbury, C. A., Lo, H. Z., & Bengio, Y. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.9 · Cited by 165
“This yields a total of 44 images containing 126 ± 33 cells (identified nuclei).”
BackgroundSelf-citationEfficient Reinforcement Learning using Gaussian Processes
Deisenroth, M. P. (2010) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ksp/1000019799 · Cited by 163
Semantic-based regularization for learning and inference
Diligenti, M., Gori, M., & Saccà, C. (2017) · Artificial Intelligence · doi:10.1016/j.artint.2015.08.011 · Cited by 162
An Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networks
Tang, R., Du, M., Liu, N., Yang, F., & Hu, X. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403064 · Cited by 165
“1) Inspired by the curriculum learning[5], which gradually increases the complexity of inputs to benefit model training.”
BackgroundOnline Batch Selection for Faster Training of Neural Networks
Loshchilov, I., & Hutter, F. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06343 · Cited by 150
“Curriculum learning is another relevant concept where the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones Bengio et al. (2009).”
BackgroundReview on Deep Neural Networks Applied to Low-Frequency NILM
Huber, P., Calatroni, A., Rumsch, A., & Paice, A. (2021) · Energies · doi:10.3390/en14092390 · Cited by 165
Zhang, D., Han, J., Zhao, L., & Meng, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1112-4 · Cited by 164
“Specifically, the proposed C-SPCL model incorporates the component of curriculum learning (CL) proposed by Bengio et al. (2009) to introduce a pre-defined curriculum to guide the learning procedure of a certain learning task.”
MethodCurriculum Learning for Natural Language Understanding
Xu, B., Zhang, L., Mao, Z., Wang, Q., Xie, H., & Zhang, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.542 · Cited by 158
“Curriculum Learning (CL) is first proposed by (Bengio et al., 2009) in machine learning area, where the definition of easy examples is established ahead, and an easy-to-difficult curriculum is arranged accordingly for the learning procedure.”
BackgroundMethodInfluentialLearning to find eye region landmarks for remote gaze estimation in unconstrained settings
Park, S., Zhang, X., Bulling, A., & Hilliges, O. (2018) · Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications · doi:10.1145/3204493.3204545 · Cited by 161
“Our final training scheme applies curriculum learning, increasing noise as training progresses [Bengio et al. 2009].”
MethodCompositional Matrix-Space Models for Sentiment Analysis
Yessenalina, A., & Cardie, C. (2011) · Cited by 161
Active Self-Paced Learning for Cost-Effective and Progressive Face Identification
Lin, L., Wang, K., Meng, D., Zuo, W., & Zhang, L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2652459 · Cited by 159
“On the other hand, SPL is a recently proposed learning regime to mimic the learning process of humans/animals that gradually incorporates easy to more complex samples into training [17], [18], where an easy sample is actual the one of high classification confidence by the currently trained model.”
BackgroundLearning symmetric and low-energy locomotion
Yu, W., Turk, G., & Liu, C. K. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201397 · Cited by 155
“…in machine learning have shown that CL can improve performance and efficiency of learning problems such as question-answering [Graves et al. 2017], classification [Bengio et al. 2009; Pentina et al. 2015], navigation [Held et al. 2017; Matiisen et al. 2017] and game playing [Narvekar et al. 2016].”
BackgroundAva Active Speaker: An Audio-Visual Dataset for Active Speaker Detection
Roth, J., Chaudhuri, S., Klejch, O., Marvin, R., Gallagher, A., Kaver, L., Ramaswamy, S., Stopczynski, A., Schmid, C., Xi, Z., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053900 · Cited by 152
“In addition, it could be augmented by utilizing the rest of the AVA dataset movies to provide large amounts of unsupervised or semi-supervised data in conjunction with curriculum learning [4, 23] or reinforcement learning [30, 31].”
BackgroundTransferring Cross-Domain Knowledge for Video Sign Language Recognition
Li, D., Yu, X., Xu, C., Petersson, L., & Li, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00624 · Cited by 158
“In particular, [21] combines curriculum learning [1] and selfpace learning [19] to learn a concept detector.”
BackgroundTextWorld: A Learning Environment for Text-Based Games
Côté, M.-A., Kádár, Á., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M., El Asri, L., Adada, M., et al. (2019) · Communications in Computer and Information Science · doi:10.1007/978-3-030-24337-1_3 · Cited by 209
“The CommAI framework [Baroni et al., 2017] emphasizes the ability to generate curricula [Bengio et al., 2009], so that agents may learn incrementally from environments of increasing complexity.”
MethodTang, Y., Wang, X., Harrison, A. P., Lü, L., Xiao, J. Z., & Summers, R. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00919-9_29 · Cited by 165
“In AGCL, we use the disease severity level to group the data samples as a means to build the curriculum for curriculum learning [1].”
BackgroundMethodBengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1709.08568 · Cited by 155
“…hypothesize that language may actually help humans build sharper internal representations (which are better disentangled) as well as facilitate learning – see the arguments around curriculum learning [Bengio et al., 2009] and cultural learning [Bengio, 2014] – and enable collaborative task-solving.”
BackgroundSelf-citationReconstruction-Based Disentanglement for Pose-Invariant Face Recognition
Peng, X., Yu, X., Sohn, K., Metaxas, D., & Chandraker, M. K. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.180 · Cited by 179
“We believe that our MSMT learning can be viewed as a form of curriculum learning [1] since multiple objectives introduced by multi-source and multi-task learning are at different levels of difficulty (e.”
BackgroundSelf-Paced Joint Sparse Representation for the Classification of Hyperspectral Images
Peng, J., Sun, W., & Du, Q. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2865102 · Cited by 151
“Self-paced learning (SPL) is a newly proposed learning mechanism that mimics the learning scheme of humans and animals which learns easy samples at first and then gradually adds more complex samples into training [33], [34].”
BackgroundDeep reinforcement learning for modeling human locomotion control in neuromechanical simulation
Song, S., Kidziński, Ł., Peng, X. B., Ong, C., Hicks, J., Levine, S., Atkeson, C. G., & Delp, S. L. (2021) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-021-00919-y · Cited by 150
“Curriculum learning [144] was also used by the top teams.”
MethodInformation Retrieval: Recent Advances and Beyond
Hambarde, K., & Proença, H. P. M. C. (2023) · IEEE Access · doi:10.1109/access.2023.3295776 · Cited by 284
“directions, such as contrastive learning [253], [254].”
Self-Paced Learning for Matrix Factorization
Zhao, Q., Meng, D., Jiang, L., Xie, Q., Xu, Z., & Hauptmann, A. G. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9584 · Cited by 168
“Inspired by the intrinsic learning principle of humans/animals, Bengio et al. (2009) proposed a new machine learning framework, called curriculum learning.”
BackgroundA Multi-scale CNN and Curriculum Learning Strategy for Mammogram Classification
Lotter, W., Sorensen, G., & Cox, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67558-9_20 · Cited by 147
“To address this challenge, we have developed a two-stage, curriculum learning-based [2] approach.”
MethodA Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
Zhang, Y., David, P., Foroosh, H., & Gong, B. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2903401 · Cited by 162
“Accordingly, we propose a curriculum-style [33] domain adaptation approach.”
MethodScene-Specific Pedestrian Detection for Static Video Surveillance
Wang, X., Wang, M., & Li, W. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.124 · Cited by 144
“methods [51], [52] could be adopted to obtain a better local”
MethodPixel-in-Pixel Net: Towards Efficient Facial Landmark Detection in the Wild
Jin, H., Liao, S., & Shao, L. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01521-4 · Cited by 206
“Different from original curriculum learning (Bengio et al. 2009), which presents training examples progressively for a specific task, our strategy applies curriculum learning at a task level.”
BackgroundIntrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., & Fergus, R. (2017) · arXiv · doi:10.48550/arxiv.1703.05407 · Cited by 149
“Curriculum learning (Bengio et al., 2009) is widely used in many machine learning approaches.”
BackgroundA vision transformer for decoding surgeon activity from surgical videos
Kiyasseh, D., Ma, R., Haque, T. F., Miles, B. J., Wagner, C., Donoho, D. A., Anandkumar, A., & Hung, A. J. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01010-8 · Cited by 147
“This is akin to curriculum learning 20 , whereby an AI system is presented with increasingly difficult tasks during the learning process in order to improve its overall performance.”
(2024) · doi:10.1145/3704435 · Cited by 184
“Another approach to improving model generalization is through curriculum learning (Bengio et al., 2009), which starts with simple tools and gradually introduces the model to more complex tools so that it can build upon its prior knowledge and develop a deeper understanding of the tool.”
Latency-Aware Collaborative Perception
Lei, Z., Ren, S., Hu, Y., Zhang, W., & Chen, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19824-3_19 · Cited by 147
A Self-Paced Multiple-Instance Learning Framework for Co-Saliency Detection
Zhang, D., Meng, D., Li, C., Jiang, L., Zhao, Q., & Han, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.75 · Cited by 148
“Self-paced learning: Inspired by the learning process of humans/animals, the theory of self-paced (or curriculum) learning [13, 14] is proposed lately.”
BackgroundMethodGeometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning
Gan, C., Gong, B., Liu, K., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00586 · Cited by 120
“The idea embodies curriculum learning [3], namely, we carefully organize the training data, teach the network with accurate geometry cues in the synthetic imagery first, and then feed the network real-world appearance information conveyed by the 3D movies.”
BackgroundCurriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding
Dai, D., Sakaridis, C., Hecker, S., & Gool, L. V. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01182-4 · Cited by 151
“CMAda adapts segmentation models from clear weather to dense fog and is inspired by curriculum learning [5], in the sense that we first solve easier tasks with our synthetic data, i.”
BackgroundSpecializing Word Embeddings for Similarity or Relatedness
Kiela, D., Hill, F., & Clark, S. R. L. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1242 · Cited by 159
“The fact that joint learning works better when supplementing raw text input with free associates, but skip-gram retrofitting works better with additional thesaurus information, could be due to curriculum learning effects (Bengio et al., 2009).”
BackgroundInvestigating LSTMs for Joint Extraction of Opinion Entities and Relations
Katiyar, A., & Cardie, C. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1087 · Cited by 172
“In future work, we plan to explore the effects of pre-training (Bengio et al., 2009) and scheduled sampling (Bengio et al., 2015) for training our LSTM network.”
BackgroundUnsupervised feature selection by self-paced learning regularization
Zheng, W., Zhu, X., Wen, G., Zhu, Y., Yu, H., & Gan, J. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.06.029 · Cited by 152
“The core concept of curriculum learning [2] is simulating the learning mode of human or animal, i.e., first learning the simple knowledge, and then gradually increasing the learning difficulty, followed by learning more difficult and professional knowledge.”
BackgroundSemantic Image Inpainting with Progressive Generative Networks
Zhang, H., Hu, Z., Luo, C., Zuo, W., & Wang, M. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240625 · Cited by 156
“Curriculum learning was first introduced by Bengio et al. [4].”
BackgroundDifferentiable Hierarchical Graph Grouping for Multi-person Pose Estimation
Jin, S., Liu, W., Xie, E., Wang, W., Qian, C., Ouyang, W., & Luo, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58571-6_42 · Cited by 180
“The model learns to group from easy to hard, in a curriculum fashion [1].”
BackgroundYu, Z., Shen, Y., Shi, J., Zhao, H., Cui, Y., Zhang, J., Torr, P. H. S., & Zhao, G. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01758-1 · Cited by 186
“Curriculum learning (Bengio et al., 2009), as amajormachine learning regime with philosophy of easy-to-hard curriculum, is utilized to train PhysFormer.”
MethodSamanantar : The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
Ramesh, G., Doddapaneni, S., Bheemaraj, A., Jobanputra, M., Raghavan, A. K., Sharma, A., Sahoo, S., Diddee, H., Mahalakshmi, J., Kakwani, D., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00452 · Cited by 409
“As future work, we plan to study different data filtering (Junczys-Dowmunt, 2018) and data sampling techniques (Bengio et al., 2009) and their impact on the performance of the NMTmodel being trained.”
MethodUncertainty-Aware Unsupervised Domain Adaptation in Object Detection
Guan, D., Huang, J., Xiao, A., Lu, S., & Cao, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082687 · Cited by 169
“Curriculum learning has been widely explored in the past decade [55]–[57].”
Data-Driven Dynamical Control for Bottom-up Energy Internet System
Hua, H., Qin, Z., Dong, N., Qin, Y., Ye, M., Wang, Z., Chen, X., & Cao, J. (2021) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2021.3110294 · Cited by 140
“To tackle this issue, we incorporate the concept of curriculum learning (CL) [33] into DRL.”
BackgroundMethodAutomatic Goal Generation for Reinforcement Learning Agents
Florensa, C., Held, D., Geng, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06366 · Cited by 138
“The idea of using a curriculum has been explored in many prior works on supervised learning (Bengio et al., 2009; Zaremba & Sutskever, 2014; Bengio et al., 2015).”
MethodToward Transformer-Based Object Detection
Beal, J., Kim, E. S., Tzeng, E., Park, D. H., Zhai, A., & Kislyuk, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09958 · Cited by 138
RLOC: Terrain-Aware Legged Locomotion Using Reinforcement Learning and Optimal Control
Gangapurwala, S., Geisert, M., Orsolino, R., Fallon, M. F., & Havoutis, I. (2022) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3172469 · Cited by 208
“function tuning, a curriculum learning [36] setup could be leveraged such that the environment complexity is increased as training progresses [14].”
BackgroundHu, X., Li, S., Huang, T., Tang, B., Huai, R., & Chen, L. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3312777 · Cited by 243
“Curriculum learning (CL) [39] is a training strategy in which the model accumulates knowledge by initially learning simple tasks before involving more complex ones.”
BackgroundExploration in Model-based Reinforcement Learning by Empirically Estimating Learning ProgressPreprint
Lopes, M. C., Lang, T., Toussaint, M., & Oudeyer, P. (2012) · Cited by 206
Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer
Yang, S., Jiang, L., Liu, Z., & Loy, C. C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00754 · Cited by 187
“The scheme borrows the idea of curriculum learning [2] to gradually increase the task difficulty in three stages as illustrated in Fig.”
BackgroundTransfer learning in magnetic resonance brain imaging: a systematic review
Valverde, J. M., Imani, V., Abdollahzadeh, A., De Feo, R., Prakash, M., Ciszek, R., & Tohka, J. (2021) · Journal of Imaging · doi:10.3390/jimaging7040066 · Cited by 137
“The incorporation of sparsity-inducing regularizers as in [40], and a closer view to these weights could lead to intelligent sample selection strategies and advances in curriculum learning [170].”
Sampling Methods for Efficient Training of Graph Convolutional Networks: A Survey
Liu, X., Yan, M., Deng, L., Li, G., Ye, X., & Fan, D. (2021) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2021.1004311 · Cited by 175
“The authors also leverage some tricks, such as negative sampling [63], to optimize the PinSage-based training process and further design a curriculum training approach [64] for faster convergence.”
A Deep Hierarchical Approach to Lifelong Learning in MinecraftPreprint
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D. J., & Mannor, S. (2016) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1604.07255 · Cited by 410
“Our work can also be interpreted as a form of curriculum learning (Bengio et al. 2009) for RL.”
BackgroundFederated learning for healthcare applications
Chaddad, A., Wu, Y., & Desrosiers, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3325822 · Cited by 136
“CL is a training approach that gradually introduces more challenging examples throughout the training process, following a proper pedagogical sequence observed in human education [52].”
BackgroundCross-Lingual Word Embeddings for Low-Resource Language Modeling
Adams, O., Makarucha, A. J., Neubig, G., Bird, S., & Cohn, T. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1088 · Cited by 163
“quantities of data (Mnih et al., 2009; Bengio et al., 2009; Collobert and Weston, 2008; Mikolov et al., 2013a), spurring much investigation (Chen et al.”
BackgroundNSCaching: Simple and Efficient Negative Sampling for Knowledge Graph Embedding
Zhang, Y., Yao, Q., Shao, Y., & Chen, L. (2019) · 2019 IEEE 35th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00061 · Cited by 108
“The main idea of self-paced (or curriculum) learning [3], [19] is to pick up easy samples first, and then gradually switch to hard ones.”
BackgroundAnticipating Traffic Accidents with Adaptive Loss and Large-Scale Incident DB
Suzuki, T., Kataoka, H., Aoki, Y., & Satoh, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00371 · Cited by 173
“In our design, the loss adaptively assigns penalty weights depending on how early a model can anticipate a traffic accident at each training epoch, inspired by Curriculum Learning.”
MethodInfluentialSeeker: alignment-free identification of bacteriophage genomes by deep learning
Auslander, N., Gussow, A. B., Benler, S., Wolf, Y. I., & Koonin, E. V. (2020) · Nucleic Acids Research · doi:10.1093/nar/gkaa856 · Cited by 145
ACPL: Anti-curriculum Pseudo-labelling for Semi-supervised Medical Image Classification
Liu, F., Tian, Y., Chen, Y., Liu, Y., Belagiannis, V., & Carneiro, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02004 · Cited by 181
“Curriculum learning focuses on the easy samples in the early training stages and gradually includes the hard samples in the later training stages, where easy samples [4, 15, 20] are usually defined as samples that have small losses during training, and hard samples tend to have large losses.”
BackgroundFrom Baby Steps to Leapfrog: How ``Less is More'' in Unsupervised Dependency Parsing
Spitkovsky, V. I., Alshawi, H., & Jurafsky, D. S. (2010) · Cited by 139
Deep Q-Learning With Q-Matrix Transfer Learning for Novel Fire Evacuation Environment
Sharma, J., Andersen, P., Granmo, O.-C., & Goodwin, M. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.2967936 · Cited by 143
“This approach of training the agent on a simpler version of the problem before training it on the actual complex problem is somewhat similar to curriculum learning [33].”
BackgroundReverse Curriculum Generation for Reinforcement Learning
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05300 · Cited by 127
“Curriculum-based approaches with manually designed schedules have been explored in supervised learning [8, 9, 10, 11] to split particularly complex tasks into smaller, easier-to-solve sub-problems.”
Background
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.