Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 3,201–3,300 of 5,240
Training Super-Resolution Network with Difficulty-Based Adaptive Sampling
Li, J., & Zhao, Z. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859923 · Cited by 3
“Recently, some effective sampling strategies have been proposed in high-level vision tasks, such as Cur riculum Learning (CL) [9] and Self-Paced Learning (SPL) [10].”
BackgroundA Curriculum Learning Based Multi-agent Reinforcement Learning Method for Realtime Strategy Game
Zhang, D., Bao, W., Liang, W., Wu, G., & Cao, J. (2022) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia56350.2022.9874056 · Cited by 3
“The main idea of CL is to optimize the order in which agents gain experience and accumulate experience, so as to make the training process more smooth, i.e., CL first presenting a simple data set, and then gradually increase the complexity and difficulty of it [11].”
BackgroundLeveraging Task Variability in Meta-learning
Aimen, A., Ladrecha, B., Sidheekh, S., & Krishnan, N. C. (2023) · SN Computer Science · doi:10.1007/s42979-023-01951-6 · Cited by 6
“Further, test errors of classes have been widely used to determine their “easy or hardness” [4, 6, 28]. acquaints the attention module with the generalizability aspect of task models and their influence in updating the meta-model.”
MethodLearning and Knowledge Transfer with Memory Networks for Machine Comprehension
Yadav, M., Vig, L., & Shroff, G. M. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1080 · Cited by 3
“It has been shown in the context of language modelling that presenting the training samples in an easy to hard ordering allows for shielding the model from very hard samples during training, yielding faster convergence and better models (Bengio et al., 2009).”
BackgroundInfluentialSelf-Paced Dense Connectivity Learning for Visual Tracking
Ge, D., Song, J., Qi, Y., Wang, C., & Miao, Q. (2019) · IEEE Access · doi:10.1109/access.2019.2904315 · Cited by 3
“[3] proposes the origin version of the SPL, named curriculum learning (CL).”
BackgroundFace Inpainting with Dilated Skip Architecture and Multi-Scale Adversarial Networks
Jiang, B., Liu, H., Yang, C., Huang, S., & Xiao, Y. (2018) · International Symposium on Parallel Architectures, Algorithms and Programming · doi:10.1109/paap.2018.00043 · Cited by 4
“Curriculum strategy has been proposed in [39] and been applied in [11][12], this strategy achieves good performance, we imitate this manner in our approach.”
MethodAutomatic Augmentation by Hill Climbing
Cruz, R., da Costa, J. P., & Cardoso, J. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30484-3_10 · Cited by 3
“Furthermore, proposal * can be contrasted with the proposal column to understand whether the dynamic nature of the method contributed to its performance, as suggested by the curriculum learning literature [3].”
BackgroundOpen-Ended Continuous Learning of Compound Goals
Dhakan, P., Kasmarik, K., Rañó, I., & Siddique, N. H. (2019) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2019.2947896 · Cited by 4
“The basic idea is to learn easier aspects of the goal or easier subgoals, and then gradually increase the difficulty level [8].”
BackgroundSelf-paced hybrid dilated convolutional neural networks
Zhang, W., Lu, G., Zhang, S., & Li, Y. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09868-5 · Cited by 4
“SPL is an improvement based on curriculum learning (CL) [2], in which a curriculum sets a series of training samples arranged in ascending order of learning difficulty for specific problem.”
BackgroundDeep learning in the ultrasound evaluation of neonatal respiratory status
Gravina, M., Gragnaniello, D., Verdoliva, L., Poggi, G., Corsini, I., Dani, C., Meneghin, F., Lista, G., Aversa, S., Raimondi, F., et al. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412732 · Cited by 4
“While the regressionbased training strategy implicitly receive such information directly from the target SF value to predict, as regarding the classification-based training strategy we resort to the use of the curriculum learning [25].”
MethodAutomatic Curriculum Generation by Hierarchical Reinforcement Learning
He, Z., Gu, C., Xu, R. X., & Wu, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63833-7_17 · Cited by 3
A Hierarchical Assessment of Adversarial Severity
Jeanneret, G., Pérez, J. C., & Arbeláez, P. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00013 · Cited by 3
“Thus, our approach takes inspiration from curriculum-based human learning [5, 23].”
BackgroundMethodIncorporating Concreteness in Multi-Modal Language Models with Curriculum Learning
Sezerer, E., & Tekır, S. (2021) · Applied Sciences · doi:10.3390/app11178241 · Cited by 6
“The contribution of this work is two-fold: A new dataset is constructed from Wikimedia Commons based on concrete/abstract terms, and a new multi-modal pre-training approach that is based on curriculum learning [15,16] is proposed.”
MethodFacilitating Human-Wildlife Cohabitation through Conflict Prediction
Ghosh, S., Varakantham, P. R., Bhatkhande, A., Ahmad, T., Andheria, A., Li, W., Taneja, A., Thakkar, D., & Tambe, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21518 · Cited by 4
“As highlighted in the paper (Bengio et al. 2009), a well-chosen curriculum can serve as a continuation method (Allgower and Georg 2012), i.”
BackgroundBalancing Multi-Domain Corpora Learning for Open-Domain Response Generation
Xing, Y., Cai, J., Barlaug, N., Liu, P., & Gulla, J. A. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.162 · Cited by 5
Granskog, J., Schnabel, T. N., Rousselle, F., et al. (2021) · ACM Transactions on Graphics · doi:10.1145/3476576.3476749 · Cited by 3
“Additional work is needed to address this issue, e.g. with a curriculum-based learning approach [Bengio et al. 2009].”
BackgroundCurriculum learning for early Alzheimer's Disease diagnosis
Gracias, C., & Silveira, M. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871601 · Cited by 4
“In this strategy, a curriculum is designed, which defines the order in which the data are presented to the model: the model is first trained with easier data (or tasks) and gradually more complex data (or tasks) are introduced, instead of being randomly presented [23].”
MethodSTARDOM: Semantic Aware Deep Hierarchical Forecasting Model for Search Traffic Prediction
Lu, Y., Ji, Q., Wang, L., et al. (2022) · Proceedings of the ... ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557102 · Cited by 3
“On the one hand, as pointed out by [1, 12], deep neural networks tend to first memorize training data of clean labels and then those of noisy labels.”
BackgroundInvestment and Risk Management with Online News and Heterogeneous Networks
Ang, G., & Lim, E. (2023) · ACM Transactions on the Web · doi:10.1145/3532858 · Cited by 4
“Curriculum learning is a training strategy that imitates the way humans learn by gradually increasing the di culty of the data samples used to train a model [4].”
BackgroundShenouda, M., Flerlage, I., Kaveti, A., Giger, M. L., & Armato, S. G. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.6.064504 · Cited by 2
“…because of its previous success in diagnosing pneumonia and other pathologies on CXRs. 18,19 Further, it followed a curriculum learning methodology as discussed in Bengio et al. 20 The curriculum learning became more specialized at each phase, concentrating the model on COVID-19 by the last step.”
MethodObtaining Better Word Representations via Language Transfer
Li, C., Xu, B., Wu, G., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-54906-9_11 · Cited by 3
Improving the Convergence Speed of Deep Neural Networks with Biased Sampling
Ioannou, G., Tagaris, T., & Stafylopatis, A. (2019) · International Conference on Advances in Artificial Intelligence (ICAAI) · doi:10.1145/3369114.3369116 · Cited by 3
“Inspired by human behaviour, curriculum learning [3] has improved the training of neural networks by choosing easier samples at the first stages of training and gradually introducing harder ones later.”
BackgroundFeature Space Regularization for Person Re-identification with One Sample
Tian, X., Li, J., Wu, H., Yang, H., Gu, X., & Chen, Y. (2019) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2019.00208 · Cited by 5
“The idea comes from the Curriculum Learning [9] (CL) proposed by Bengio et al.. Kumat et al. propose SelfPaced Learning (SPL) [29] which takes curriculum learning as a regularization term to update the model automatically.”
BackgroundMethodLearning domain abstractions for long lived robots
Rosman, B. (2014) · ERA · Cited by 3
Weakly Supervised Learning for Structured Output Prediction
Kumar, M. (2013) · HAL · Cited by 3
On layer-wise representations in deep neural networks
Montavon, G. (2013) · DepositOnce · doi:10.14279/depositonce-3875 · Cited by 5
From One-Trick Ponies to All-Rounders: On-Demand Learning for Image Restoration.Preprint
Gao, R., & Grauman, K. L. (2016) · arXiv (Cornell University) · Cited by 3
Efficient Diverse Ensemble for Discriminative Co-TrackingPreprint
Meshgi, K., Oba, S., & Ishii, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06564 · Cited by 3
“Starting from easiest examples (Curriculum learning) [6], pruning adversarial examples [35], excluding misclassified samples from next rounds of training [51], sorting samples by their training value [30] are some of the proposed approaches in the literature.”
Curriculum Q-Learning for Visual Vocabulary AcquisitionPreprint
Zaidi, A., Moore, R., & Briscoe, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10837 · Cited by 3
“The importance of curriculum learning in training deep learning models and agents has also been discussed by [10] where its use is shown to facilitate the generalisation as well as the rate of convergence and training of deep learning networks.”
BackgroundDeep Cross-media Knowledge TransferPreprint
Huang, X., & Peng, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.03777 · Cited by 3
“The motivation of CL is simple: to first learn from easy samples, and gradually learn from harder samples [2], which aims to reduce the negative e ff ects brought by noisy data in early period of training.”
BackgroundWeakly-Supervised Localization and Classification of Proximal Femur FracturesPreprint
Jiménez-Sánchez, A., Kazi, A., Albarqouni, S., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10692 · Cited by 3
“Other possibilities to deal with the 6 classes are the use of more complex architectures like a ConvNet cascade to fetch deeper features, or different learning strategies such as curriculum learning [23] that gradually makes the task harder.”
BackgroundLearning to Navigate the WebPreprint
Gür, İ., Rueckert, U., Faust, A., & Hakkani-Tür, D. Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.09195 · Cited by 53
“Curriculum learning methods (Bengio et al. (2009); Zaremba & Sutskever (2014); Graves et al. (2017)) are studied to divide a complex task into multiple small sub-tasks that are easier to solve.”
Advancing Acoustic-to-Word CTC Model with Attention and Mixed-UnitsPreprint
Das, A. K., Li, J., Ye, G., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11928 · Cited by 3
“curriculum learning [38], Nesterov momentum-based stochastic gradient descent, dropout, and low rank matrix factorization [39].”
BackgroundCompetitive Experience ReplayPreprint
Liu, H., Trott, A., Socher, R., & Xiong, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00528 · Cited by 42
“Curriculum learning is widely used for training neural networks (see e.g., Bengio et al., 2009; Graves et al., 2017; Elman, 1993; Olsson, 1995).”
MethodMulti-Instance Learning for End-to-End Knowledge Base Question AnsweringPreprint
Wei, M., He, Y., Zhang, Q., & Si, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.02652 · Cited by 3
Learning Monocular Visual Odometry through Geometry-Aware Curriculum LearningPreprint
Saputra, M. R. U., de Gusmão, P. P. B., Wang, S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.10543 · Cited by 3
“[8] proposed CL as a strategy to improve the convergence speed and generalization ability of a machine learning model by learning through highly organized or meaningful order of examples.”
BackgroundInfluentialGraph Distillation for Action Detection with Privileged ModalitiesPreprint
Luo, Z., Hsieh, J.-T., Jiang, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00108 · Cited by 3
“Since it is nontrivial to jointly train on multiple modalities from scratch, we employ curriculum learning [1] to train the distillation graph.”
MethodLearning to Navigate Cloth using HapticsPreprint
Clegg, A., Yu, W., Erickson, Z., Tan, J., Liu, C. K., & Turk, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06905 · Cited by 4
“The neural network is trained using Trust Region Policy Optimization (TRPO) [22] with a curriculum learning strategy [31].”
MethodNSCaching: Simple and Efficient Negative Sampling for Knowledge Graph EmbeddingPreprint
Zhang, Y., Yao, Q., Shao, Y., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06410 · Cited by 3
“The main idea of self-paced (or curriculum) learning [3], [19] is to pick up easy samples first, and then gradually switch to hard ones.”
BackgroundSlim DensePose: Thrifty Learning from Sparse Annotations and Motion CuesPreprint
Neverova, N., Thewlis, J., Güler, R. A., Kokkinos, I., & Vedaldi, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05706 · Cited by 3
“We assume that this is due to both the larger variability of images in the COCO training set, as well as the cleaner nature of COCO images (blur-free, larger resolution), which is known to assist training [1].”
BackgroundSpatial Memory for Context Reasoning in Object DetectionPreprint
Chen, X., & Gupta, A. K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04224 · Cited by 3
“For efficiency, we also follow a curriculum learning [8] strategy: bootstrap a SMN of more iterations (e.”
MethodLong-Term Image Boundary PredictionPreprint
Bhattacharyya, A., Malinowski, M., Schiele, B., & Fritz, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08841 · Cited by 5
“We use a curriculum learning approach [1], where we initialize the models with the weights learned on single and two ball worlds respectively.”
MethodHard Pixels Mining: Learning Using Privileged Information for Semantic Segmentation.Preprint
Gu, Z., Niu, L., & Zhang, L. (2019) · arXiv (Cornell University) · Cited by 3
Automated analysis of spinal MRI using deep learning
Jamaludin, M. H. (2017) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-gppqvrbjm · Cited by 5
Towards Noise-Robust Neural Networks via Progressive Adversarial Training.Preprint
Yu, H., Liu, A., Liu, X., Yang, J., & Zhang, C. (2019) · arXiv (Cornell University) · Cited by 3
Multiple-objective Reinforcement Learning for Inverse Design and IdentificationPreprint
Wei, H., Olarte, M. V., & Goh, G. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03741 · Cited by 7
“A sequence of subtasks is ranked in difficulty ascending order and conducted within different phases(Bengio et al. 2009).”
BackgroundWalking the Tightrope: An Investigation of the Convolutional Autoencoder BottleneckPreprint
Manakov, I., Rohm, M., & Tresp, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07460 · Cited by 15
“Overall, this effect is not clear-cut and warrants a more thorough investigation of the relationship between data complexity and CAE characteristics, especially in the light of compelling results from curriculum learning research (Bengio et al., 2009).”
BackgroundDeepLABNet: End-to-end Learning of Deep Radial Basis Networks with Fully Learnable Basis FunctionsPreprint
Hryniowski, A., & Wong, A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09257 · Cited by 15
“Such improvements are, in part, due to advanced network architecture designs and improved learning methodologies [1, 2, 3, 4, 5].”
BackgroundReview, Analysis and Design of a Comprehensive Deep Reinforcement Learning FrameworkPreprint
Nguyen, N. D., Nguyen, T. T., Nguyen, V., Creighton, D., & Nahavandi, S. (2020) · arXiv (Cornell University) · doi:10.13140/rg.2.2.16789.06883 · Cited by 12
“[116] propose Hindsight Experience Replay that implicitly simulates curriculum learning [117] by creating imagined trajectories in the experience replay with positive rewards.”
BackgroundHard Pixel Mining for Depth Privileged Semantic SegmentationPreprint
Gu, Z., Niu, L., Zhao, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.11437 · Cited by 3
“This training strategy is related to curriculum learning [34] and capable of further improving the segmentation performance.”
BackgroundProgressively Pretrained Dense Corpus Index for Open-Domain Question AnsweringPreprint
Xiong, W., Wang, H., & Wang, W. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00038 · Cited by 3
“Note that our training algorithm shares spirits with Curriculum Learning (Bengio et al., 2009) and SelfPaced Learning (Kumar et al.”
Navigating the Landscape of Games.Preprint
Omidshafiei, S., Tuyls, K., Czarnecki, W. M., Santos, F. C., Rowland, M., Connor, J. T., Hennes, D., Müller, P. F., Pérolat, J., Vylder, B. D., et al. (2020) · arXiv (Cornell University) · Cited by 4
Le, A. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02669 · Cited by 3
“in 2009 [11].”
Simple Sensor Intentions for ExplorationPreprint
Hertweck, T., Riedmiller, M., Bloesch, M., Springenberg, J. T., Siegel, N., Wulfmeier, M., Hafner, R., & Heess, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07541 · Cited by 9
“However, to accelerate learning different perspectives build on curriculum [6, 21, 16], iterative task generation [35, 47].”
BackgroundA Competence-aware Curriculum for Visual Concepts Learning via Question AnsweringPreprint
Li, Q., Huang, S., Hong, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01499 · Cited by 3
“The competence-aware curriculum in our work is related to curriculum learning [7,54,56,20,51,45,21,48] and machine teaching [61,62,39,11,41,15,58].”
BackgroundAdaptive Procedural Task Generation for Hard-Exploration ProblemsPreprint
Fang, K., Zhu, Y., Savarese, S., & Fei-Fei, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00350 · Cited by 18
“Curriculum learning utilizes alternative datasets and tasks to accelerate the learning process of challenging target tasks (Bengio et al., 2009; Graves et al., 2017).”
BackgroundInteraction-limited Inverse Reinforcement LearningPreprint
Troussard, M., Pignat, E., Kamalaruban, P., Calinon, S., & Cevher, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00425 · Cited by 3
“Figure 1: Summary of different learning modalities Curriculum learning and self-paced learning have been applied effectively to the supervised and semisupervised learning problems [16, 18, 19, 20, 21, 22].”
BackgroundMethodInfluentialContinual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay ProcessesPreprint
Lesort, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00487 · Cited by 37
“In deep learning, this function is implemented as a deep neural network, i.e. a neural network with more than one hidden layer Bengio et al. (2009a); Schmidhuber (2015) (Figure 2.11).”
BackgroundMethodInfluentialST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language UnderstandingPreprint
Kim, M., Kim, G., Lee, S., & Ha, J.-W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12283 · Cited by 3
“We adopt a curriculum pre-training [24] for better generalization and fast convergence [25].”
MethodImproved Synthetic Training for Reading ComprehensionPreprint
Chen, Y., Sultan, M. A., & Castelli, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12776 · Cited by 4
“Finally, the proposed idea of using the hardest subset of examples for pre-training has similarities with curriculum learning (CL) (Bengio et al., 2009), which trains models in an easy-to-hard order so that the hardest examples are used at the end.”
BackgroundMBB: Model-Based Baseline for Efficient Reinforcement Learning.Preprint
Lyu, X., Li, S., Siriya, S., Pu, Y., & Chen, M. (2020) · arXiv (Cornell University) · Cited by 3
Temporal LiDAR Frame Prediction for Autonomous DrivingPreprint
Deng, D., & Zakhor, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09409 · Cited by 3
“There exist two popular approaches for training generative, temporal networks: curriculum approaches [3] and teacher-forcing [9].”
BackgroundKNN-enhanced Deep Learning Against Noisy LabelsPreprint
Kong, S., Li, Y., Wang, J., Rezaei, A., & Zhou, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04224 · Cited by 12
“Empirically, the use of CL has been shown to accelerate and improve the learning process [3] and noise robustness [4].”
IOT: Instance-wise Layer Reordering for Transformer StructuresPreprint
Zhu, J., Wu, L., Xia, Y., Xie, S., Qin, T., Zhou, W., Li, H., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03457 · Cited by 6
“Some works attempt to weight each data with different importance (Ren et al., 2018; Hu et al., 2019; Chang et al., 2017) or feed data with curriculum learning according to its difficulty (Bengio et al., 2009; Fan et al., 2018).”
BackgroundEfficient Reinforcement Learning for StarCraft by Abstract Forward Models and Transfer LearningPreprint
Liu, R.-Z., Guo, H., Ji, X., Yu, Y., Pang, Z.-J., Xiao, Z., Wu, Y., & Lü, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.00715 · Cited by 3
“Curriculum learning (CL) [18], [19] is a learning process that allows learning to grow incrementally through a series of tasks, e.g., repeatedly transfer learning from easier task to harder task.”
Learning to Combat Noisy Labels via Classification MarginsPreprint
Lin, J. Z., & Bradić, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.00751 · Cited by 7
“Sample reweighting is related to curriculum learning (Bengio et al., 2009), the idea to learn from easier instances first and gradually including harder ones, thereby forming a curriculum.”
BackgroundA bandit approach to curriculum generation for automatic speech recognitionPreprint
Kuznetsova, A., Kumar, A. K., & Tyers, F. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03662 · Cited by 6
“[3] conducted the experiments on artificially generated tasks and presented proof-of-concept results.”
BackgroundWhere to go next: Learning a Subgoal Recommendation Policy for Navigation Among PedestriansPreprint
Brito, B., Everett, M., How, J. P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.13073 · Cited by 3
“Moreover, to increase the learning speed during training, we gradually increase the number of agents in the training environments (curriculum learning [40]).”
BackgroundSynthesizing Irreproducibility in Deep NetworksPreprint
Snapp, R. R., & Shamir, G. I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10696 · Cited by 3
“For such models, nondeterminism in training may lead optimizers to different optima (Summers & Dinneen, 2021) (see also Nagarajan et al. (2018)), that depend on the training randomness (Achille et al., 2017; Bengio et al., 2009).”
BackgroundVirtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth PredictionPreprint
Yin, W., Liu, Y., & Shen, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04216 · Cited by 3
“Several works have demonstrated that curriculum learning [43], [44], [45] can boost the performance of deep learning methods.”
BackgroundUncertainty-guided Model Generalization to Unseen DomainsPreprint
Qiao, F., & Peng, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07531 · Cited by 3
“Second, we can gradually enlarge the domain transportation by increasing the uncertainty of S in a curriculum learning scheme [2].”
BackgroundSparse Curriculum Reinforcement Learning for End-to-End Driving.Preprint
Agarwal, P., de Beaucorps, P., & de Charette, R. (2021) · arXiv (Cornell University) · Cited by 3
TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLPreprint
Romac, C., Portelas, R., Hofmann, K., & Oudeyer, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.09815 · Cited by 30
“Building upon this, curriculum learning has long been identified as a key component for many machine learning problems (Selfridge et al., 1985; Elman, 1993; Bengio et al., 2009; Cangelosi & Schlesinger, 2015) in order to organize samples showed during learning.”
BackgroundCo-matching: Combating Noisy Labels by Augmentation AnchoringPreprint
Lu, Y., Yang, B., & He, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12814 · Cited by 7
“Inspired from human cognition, Curriculum learning (CL) [4] proposes to start from easy samples and go through harder samples to improve convergence and generalization.”
Meta-Curriculum Learning for Domain Adaptation in Neural Machine TranslationPreprint
Zhan, R., Liu, X., Wong, D. F., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02262 · Cited by 3
“Motivated by curriculum learning for NMT (Bengio et al. 2009; Platanios et al. 2019; Liu et al. 2020), which arranges the learning curricula from easy to difficult during NMT training, we propose a novel learning strategy for lowresource domain translation, called meta-curriculum learning.”
MethodExponentiated Gradient Reweighting for Robust Training Under Label Noise and BeyondPreprint
Majidi, N., Amid, E., Talebi, H., & Warmuth, M. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01493 · Cited by 22
“A similar approach is considered in curriculum learning where the level of difficulty of the training examples is increased gradually over the course of training [6].”
BackgroundCyclic Co-Learning of Sounding Object Visual Grounding and Sound SeparationPreprint
Tian, Y., Hu, D., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02026 · Cited by 3
“To this end, we learn our CCoL model with three steps in a curriculum learning [4] manner.”
MethodModern Subsampling Methods for Large-Scale Least Squares RegressionPreprint
Li, T., & Meng, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01552 · Cited by 3
“Recently, researchers have shifted their focus on using importance sampling to improve and accelerate the training of neural networks (Bengio et al., 2009; Alain et al., 2015; Loshchilov and Hutter, 2015; Schaul et al., 2015; Katharopoulos and Fleuret, 2018).”
BackgroundCurriculum Design for Teaching via Demonstrations: Theory and ApplicationsPreprint
Yengera, G., Devidze, R., Kamalaruban, P., & Singla, A. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04696 · Cited by 15
Progressive Multi-Granularity Training for Non-Autoregressive TranslationPreprint
Ding, L., Wang, L., Liu, X., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05546 · Cited by 3
Coarse-to-Fine Curriculum LearningPreprint
Stretcu, O., Platanios, E. A., Mitchell, T. M., & Póczos, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04072 · Cited by 15
“Using curricula in the context of machine learning was first proposed by [13], before [5] coined the term curriculum learning.”
BackgroundKnowledge Distillation via Instance-level Sequence LearningPreprint
Zhao, H., Sun, X., Dong, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10885 · Cited by 3
“From the perspective of curriculum learning [29], the”
BackgroundIn-N-Out: Towards Good Initialization for Inpainting and OutpaintingPreprint
Jo, C., Im, W., & Yoon, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13953 · Cited by 3
“Regarding this, In-N-Out can be viewed as curriculum learning [1] or reverse curriculum learning [31].”
BackgroundSPACE: A Simulator for Physical Interactions and Causal Learning in 3D EnvironmentsPreprint
Duan, J., Yu, S., & Tan, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06180 · Cited by 3
“The purpose of the SPACE dataset is to serve as the building blocks for curriculum learning [4] in more complex real-world object interaction tasks.”
BackgroundMethodInfluentialLearning Data Teaching Strategies Via Knowledge TracingPreprint
Abdelrahman, G. M., & Wang, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07083 · Cited by 3
“Curriculum learning methods [1], [2], [3]”
Reviewing continual learning from the perspective of human-level intelligencePreprint
Chang, Y., Li, W., Peng, J., Tang, B., Kang, Y., Lei, Y., Gui, Y., Zhu, Q., Liu, Y., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11964 · Cited by 8
“Although difficult examples are more informative than easy examples, they are likely not useful because they confuse the learner rather than helping it establish the correct location of the decision surface [94].”
BackgroundProSelfLC: Progressive Self Label Correction Towards A Low-Temperature Entropy StatePreprint
Wang, X., Yang, H., Kodirov, E., Mukherjee, S. S., Clifton, D. A., & Robertson, N. M. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.07.01.498447 · Cited by 5
“Their objective is to annotate unlabelled samples or correct noisy labels; (5) Sample selection methods aim to drop corrupted examples [14, 81, 84, 93]; (6) Example weighting approaches assign higher weights to more reliable labels [6, 10, 23, 32, 38, 63, 67, 77]; (7) Robust loss functions include [48, 52, 76, 79, 106…”
BackgroundDeep Learning-Based Super-Resolution of Digital Elevation Models in Data Poor Regions.Preprint
Dahal, A., van den Bout, B., van Westen, C. J., et al. (2022) · doi:10.31223/x5dd21 · Cited by 3
Consistent multi-animal pose estimation in cattle using dynamic Kalman filter based trackingPreprint
Perneel, M., Adriaens, I., Aernouts, B., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-3863814/v1 · Cited by 3
Deep LPPLS: Forecasting of temporal critical points in natural, engineering and financial systemsPreprint
Nielsen, J., Sornette, D., & Raissi, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4839066 · Cited by 3
“One promising research path is the exploration of curriculum learning, as proposed by [31].”
BackgroundGao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3
SPLDExtraTrees: robust machine learning approach for predicting kinase inhibitor resistance
Yang, Z., Ye, Z., Xiao, Y., Hsieh, C., & Zhang, S.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac050 · Cited by 18
“Instead of learning from all samples simultaneously, we introduce self-paced learning (SPL) strategy that learns in a meaningful order, from easy samples to hard ones [43, 25] according to the definition of easiness provided in the subsequent discussion.”
Lee, K., Lee, H., Park, J., et al. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3378311 · Cited by 2
“In previous studies [31], [41], [42], the concepts of hard and easy samples within a field of DL have been delineated.”
BackgroundThe Man Behind the Curtain: Appropriating Fairness in AI
Korecki, M., Köstner, G., Martinelli, E., & Carissimo, C. (2024) · Minds and Machines · doi:10.1007/s11023-024-09669-x · Cited by 4
“This means that the data is presented to an AI in a particular order which influenced the training and results which may not have been achieved by a different order, as in curriculum learning (Bengio et al., 2009; Wang et al., 2021).”
CL-NOTEARS: Continuous Optimization Algorithm Based on Curriculum Learning Framework
Liu, K., Liu, L., Xiao, K., Li, X., Zhang, H., Zhou, Y., & Huang, H. (2024) · Mathematics · doi:10.3390/math12172640 · Cited by 4
Hassani, A., & Rekik, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73290-4_4 · Cited by 5
Learning to Compare Hardware Designs for High-Level Synthesis
Bai, Y., Sohrabizadeh, A., Ding, Z., Liang, R., Li, W., Wang, D., Ren, H. M., Sun, Y., & Cong, J. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740257 · Cited by 3
C2F-CHART: A Curriculum Learning Approach to Chart Classification
Shaheen, N., Elsharnouby, T., & Torki, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78107-0_24 · Cited by 3
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.