Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 801–900 of 5,239
OMG: Towards Effective Graph Classification Against Label Noise
Yin, N., Shen, L., Wang, M., Luo, X., Luo, Z., & Tao, D. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3271677 · Cited by 71
“Inspired by curriculum learning [19], an adaptive criterion is adopted for reliable noise removal from easy to hard.”
MethodLin, Q., & Ma, H. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3292004 · Cited by 60
“Inspired by the curriculum learning [30], we design a training pipeline that starts with only 2 agents on 10 × 10 grid maps and gradually increases the number of agents and the size of the map once the success rate reaches a certain threshold.”
MethodDiff4Rec: Sequential Recommendation with Curriculum-scheduled Diffusion Augmentation
Wu, Z., Wang, X. Y., Chen, H., Li, K., Han, Y., Sun, L., & Zhu, W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612709 · Cited by 58
Ge, Y., Song, S., Yu, S., Zhang, X., & Li, X. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.108776 · Cited by 48
Transductive Zero-Shot Learning With Adaptive Structural Embedding
Yu, Y., Ji, Z., Guo, J., & Pang, Y. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2753852 · Cited by 47
“As defined in [20], the process of gradually added training instances is called a curriculum.”
BackgroundActive Learning for Interactive Neural Machine Translation of Data Streams
Peris, Á., & Casacuberta, F. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1015 · Cited by 64
Baar, J. V., Sullivan, A., Cordorel, R., Jha, D., Romeres, D., & Nikovski, D. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793561 · Cited by 45
“Adding additional goals (sub-goals), basically a form of curriculum learning [24], can improve the learning as well [25].”
BackgroundZhong, Z., Sun, L., & Huo, Q. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.106986 · Cited by 46
Joint Learning in the Spatio-Temporal and Frequency Domains for Skeleton-Based Action Recognition
Hu, G., Cui, B., & Yu, S. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2953325 · Cited by 48
“Some previous studies adopt heuristic rules to adjust the sampling probabilities of the train data, such as curriculum learning [46], self-paced learning [47], online batch selection [48], etc. Fan et al. [45] also uses deep reinforcement learning framework to automatically learn what data to learn.”
MethodAsking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus
Liu, B., Wei, H., Niu, D., Chen, H., & He, Y. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380270 · Cited by 92
“First, we can utilize curriculum learning algorithms [2] to select samples during training.”
MethodCurriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning
Cascante-Bonilla, P., Tan, F., Qi, Y., & Ordonez, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16852 · Cited by 281
“Prior work has shown that training with a curriculum improves performance in several machine learning tasks (Bengio et al. 2009; Hacohen and Weinshall 2019).”
BackgroundIncreasing Robustness to Spurious Correlations using Forgettable Examples
Yaghoobzadeh, Y., Mehri, S., Combes, R. T. D., Hazen, T. J., & Sordoni, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.291 · Cited by 94
“While this might seem at odds with the common belief that high-level semantic representations of the input data are formed (Bengio et al., 2009b), the reliance on highly predictive but brittle features is not confined to NLU tasks.”
BackgroundJi, Q., Fu, S., Tan, K., Muralidharan, S. T., Lagrelius, K., Danelia, D., Andrikopoulos, G., Wang, X. V., Wang, L., & Feng, L. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102382 · Cited by 57
Self-Paced Broad Learning System
Liu, L., Cai, L., Xie, T., & Wang, Y. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3181449 · Cited by 48
“Recently, inspired by human cognitive behavior, the selfpaced learning (SPL) has been introduced to deal with noisy data [42], [43].”
BackgroundBiased Importance Sampling for Deep Neural Network TrainingPreprint
Katharopoulos, A., & Fleuret, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00043 · Cited by 46
“Curriculum learning (Bengio et al. 2009) and its evolution self-paced learning (Kumar, Packer, and Koller 2010) present the classifier with easy samples first (samples that are likely to have a small loss) and gradually introduce harder and harder samples.”
BackgroundFeedback Network for Image Super-ResolutionPreprint
Li, Z., Yang, J., Liu, Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09814 · Cited by 46
“Curriculum learning[2], which gradually increases the difficulty of the learned target, is well known as an efficient strategy to improve the training procedure.”
BackgroundMask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary ShapesPreprint
Lyu, P., Liao, M., Yao, C., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02242 · Cited by 46
Natural Walking With Musculoskeletal Models Using Deep Reinforcement Learning
Weng, J., Hashemi, E., & Arami, A. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3067617 · Cited by 57
“CL aims to improve learning progress by decomposing a challenging objective into multiple intermediate steps that are easier to learn [16].”
BackgroundWang, H., Huang, S.-C., Huang, P.-J., Wang, K.-L., Teng, Y.-C., Ko, Y.-T., Jeon, D., & Wu, I. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3251193 · Cited by 48
“The benefits of a “starting small” strategy for machine learning were elucidated in [14].”
BackgroundKuo, N. I.-H., García, F., Sönnerborg, A. B., Böhm, M., Kaiser, R., Zazzi, M., Polizzotto, M. N., Jorm, L. R., & Barbieri, S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104436 · Cited by 65
Rethinking Optical Flow from Geometric Matching Consistent Perspective
Dong, Q., Cao, C., & Fu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00135 · Cited by 74
“FlowNet2 [22] finds that not only the quality of data is important, but also the data presenting order during the optical flow training is critical, which is closely related to the curriculum learning [2].”
BackgroundMethodInfluentialAutoWebGLM: A Large Language Model-based Web Navigating Agent
Lai, H., Liu, X., Iong, I. L., Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671620 · Cited by 274
“If 𝑀 SFT answered all iterations correctly, we consider it devoid of training value and incapable of providing practical negative examples.”
Efficient Optimization of Performance Measures by Classifier Adaptation
Li, N., Tsang, I. W.-H., & Zhou, Z. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.172 · Cited by 51
“Curriculum learning [2] is a learning paradigm which circumvents a challenging learning task by starting with”
BackgroundRobust Speech Recognition Using Generalized Distillation Framework
Markov, K., & Matsui, T. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-852 · Cited by 55
“Generalized Distillation is closely related to applications in methods such as semi-supervised learning, domain adaptation, transfer learning, Universum learning [12] and curriculum learning [13].”
MethodFine-Grained Propaganda Detection with Fine-Tuned BERT
Yoosuf, S., & Yang, Y. (2019) · Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda (NLP4IF) · doi:10.18653/v1/d19-5011 · Cited by 68
“Hence, the sampling method in our final submission strikes a balance through curriculum learning (Bengio et al., 2009), whereby an oversampling strategy is used in the first half of the training and sequential sampling is used in the second half.”
MethodCurriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking
Penha, G., & Hauff, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_46 · Cited by 47
“More recently, Bengio et al. [1] provided additional evidence that curriculum strategies can benefit neural network training with experimental results on different tasks such as shape recognition and language modelling.”
BackgroundInfluentialSampling Rate Decay in Hindsight Experience Replay for Robot Control
Vecchietti, L. F., Seo, M., & Har, D. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2990722 · Cited by 50
“The use of hindsight experience in the HER is a type of curriculum learning [18] where the current policy learns to perform goals that it can achieve implicitly.”
BackgroundSpeech Enhancement Based on Denoising Autoencoder With Multi-Branched Encoders
Yu, C., Zezario, R. E., Wang, S., Sherman, J., Hsieh, Y.-Y., Lu, X., Wang, H., & Tsao, Y. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3025638 · Cited by 64
Training strategies for deep learning gravitational-wave searches
Schäfer, M. B., Zelenka, O., Nitz, A., Ohme, F., & Brügmann, B. (2022) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.105.043002 · Cited by 54
CurGraph: Curriculum Learning for Graph Classification
Wang, Y., Wang, W., Liang, Y., Cai, Y., & Hooi, B. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450025 · Cited by 66
“Prior curriculum learning approaches evaluate difficulty by defining the heuristic metrics by observing the characteristics of the particular target task and data [4], [38].”
BackgroundInfluentialSelf-Paced Enhanced Low-Rank Tensor Kernelized Multi-View Subspace Clustering
Chen, Y., Wang, S., Xiao, X., Liu, Y., Hua, Z., & Zhou, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3112230 · Cited by 45
“Self-paced learning [39] and curriculum learning [40] are inspired by the learning process of humans/animals, i.e. , learning the model iteratively from easy samples to complex ones in a self-pacedfashion.”
BackgroundIdentifying Mislabeled Data using the Area Under the Margin RankingPreprint
Pleiss, G., Zhang, T., Elenberg, E. R., & Weinberger, K. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.10528 · Cited by 425
“As future work, we plan to investigate the use of AUM to estimate sample “hardness” for use in curriculum learning (Bengio et al., 2009) and to detect mis-predictions in the context of semi-supervised learning (Chapelle et al., 2009). learning rate is dropped).”
MethodWeak-to-Strong Consistency Learning for Semisupervised Image Segmentation
Lu, X., Jiao, L., Li, L., Liu, F., Liu, X., Yang, S., Feng, Z., & Chen, P. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3272552 · Cited by 58
Unleashing Unlabeled Data: A Paradigm for Cross-View Geo-Localization
Li, G., Qian, M., & Xia, G.-S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01582 · Cited by 44
“Curriculum-Learning: When humans learn knowledge, they often learn easy courses first and then difficult courses after they have a certain foundation [1].”
BackgroundCurriculum Learning for Motor Skills
Karpathy, A., & van de Panne, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30353-1_31 · Cited by 42
“In the context of deep learning, it has been shown that multi-stage curriculum learning strategies give rise to improved generalization and faster convergence of neural networks on vision and language tasks [2].”
BackgroundRecognition From Web Data: A Progressive Filtering Approach
Yang, J., Sun, X., Lai, Y., Zheng, L., & Cheng, M. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2855449 · Cited by 48
“Meanwhile, it is natural that knowledge is taught from easy to difficult in the form of a curriculum, leading to a learning paradigm called curriculum learning [25].”
Dynamic Difficulty Awareness Training for Continuous Emotion Prediction
Zhang, Z., Han, J., Coutinho, E., & Schuller, B. W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2871949 · Cited by 64
“Additionally, a more recent and promising approach refers to curriculum learning, which was firstly introduced in [21].”
BackgroundA Self-Paced Regularization Framework for Multilabel Learning
Li, C., Wei, F., Yan, J., Zhang, X., Liu, Q., & Zha, H. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2697767 · Cited by 44
“This idea is inspired by the fact that humans often learn from easy concepts to hard ones in the cognitive process [23], [24].”
BackgroundHuang, Y., & Du, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1037 · Cited by 49
“Mean-while, highly organised human education methods using a multilevel curriculum, from easy to hard, is an analogy for curriculum learning training, which advocates the adoption of similar multi-stage strategies (Bengio et al., 2009).”
BackgroundNeural Data Server: A Large-Scale Search Engine for Transfer Learning Data
Yan, X., Acuna, D., & Fidler, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00395 · Cited by 57
“In active learning [42] one searches over unlabeled data to find optimal samples to be labeled by an oracle, while in curriculum learning [6] subsets of data of increasing difficulty are sought for during training.”
BackgroundAutoLoss: Automated Loss Function Search in Recommendations
Zhao, X., Liu, H., Fan, W., Liu, H., Tang, J., & Wang, C. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467208 · Cited by 44
“Motivated by curriculum learning [1, 19], the original class probabilities should be generated according to the ground truth labels 𝒚 and the output of DRS network ˆ 𝒚 .”
BackgroundDeep Anomaly Discovery from Unlabeled Videos via Normality Advantage and Self-Paced Refinement
Yu, G. W., Wang, S., Cai, Z., Liu, X., Xu, C., & Wu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01360 · Cited by 69
“Motivated by the beneficial learning order in human curricula, CL introduces a learning strategy that trains the model with samples in an easy-to-hard manner [2].”
BackgroundWang, X., Yang, H., Kodirov, E., Clifton, D. A., & Robertson, N. M. (2019) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1903.12141 · Cited by 72
“Other related work on weighting samples includes curriculum learning Bengio et al. (2009), self-paced learning Kumar et al.”
BackgroundInfluentialMercury: Efficient On-Device Distributed DNN Training via Stochastic Importance Sampling
Zeng, X., Yan, M., & Zhang, M. (2021) · Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/3485730.3485930 · Cited by 43
“Importance sampling shares similarities with curriculum learning [4] in terms of selecting important data to generate mini-batch for training.”
BackgroundControlling the Solo12 quadruped robot with deep reinforcement learning
Aractingi, M., Léziart, P., Flayols, T., Pérez, J., Silander, T., & Souères, P. (2023) · Scientific Reports · doi:10.1038/s41598-023-38259-7 · Cited by 76
“Curriculum learning is a popular method that introduces easier tasks to learn at the start of training and gradually increases the level of difficulty as training progresses [40].”
BackgroundMapping raised bogs with an iterative one-class classification approach
Mack, B., Roscher, R., Stenzel, S., Feilhauer, H., Schmidtlein, S., & Waske, B. (2016) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2016.07.008 · Cited by 44
Teaching-to-Learn and Learning-to-Teach for Multi-label Propagation
Gong, C., Tao, D., Yang, J., & Liu, W. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10191 · Cited by 45
“Our work also differs from curriculum learning (Bengio et al. 2009; Kumar, Packer, and Koller 2010; Jiang et al. 2015; Khan, Mutlu, and Zhu 2011) in that ML-TLLT requires the interaction between a teaching committee and a learner.”
ResultUnsupervised Learning of Foreground Object Segmentation
Croitoru, I., Bogolin, S.-V., & Leordeanu, M. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01183-3 · Cited by 60
“The idea of increasing the data size and complexity in stages, from fewer easy cases to many and more complex ones, is also related to insights from curriculum learning (Bengio et al. 2009).”
BackgroundMethodInfluentialLaserFlow: Efficient and Probabilistic Object Detection and Motion Forecasting
Meyer, G. P., Charland, J., Pandey, S., Laddha, A., Gautam, S., Vallespi-Gonzalez, C., & Wellington, C. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3047793 · Cited by 58
“In this work, we propose a different approach inspired by curriculum learning [31].”
MethodEnd-to-End Speech Translation With Transcoding by Multi-Task Learning for Distant Language Pairs
Kano, T., Sakti, S., & Nakamura, S. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2986886 · Cited by 47
“Bengio et al. [15] proposed CL for multiple tasks.”
BackgroundLearning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
Reda, D., Tao, T., & van de Panne, M. (2020) · Motion in Games · doi:10.1145/3424636.3426907 · Cited by 69
“These strategies have been applied to supervised learning [Bengio et al. 2009] and are becoming increasingly popular in the RL setting, e.”
MethodElhamod, M., Bu, J., Singh, C. N., Redell, M. D., Ghosh, A., Podolskiy, V. A., Lee, W., & Karpatne, A. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3530911 · Cited by 41
“It is also related to work in the area of curriculum learning [6, 16, 32], where “simpler” examples are presented to the neural network in an incremental manner before showing more “complex” ones.”
BackgroundMethodRemote Sensing Image Super-Resolution via Saliency-Guided Feedback GANs
Wu, H., Zhang, L., & Ma, J. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3042515 · Cited by 45
“The basic idea is to learn a complex task from easy to hard in a self-paced fashion [54], [55].”
BackgroundWildfire Front Monitoring With Multiple UAVs Using Deep Q-Learning
Viseras, A., Meißner, M., & Marchal, J. (2021) · IEEE Access · doi:10.1109/access.2021.3055651 · Cited by 58
Learning when to Communicate at Scale in Multiagent Cooperative and Competitive TasksPreprint
Singh, A., Jain, T., & Sukhbaatar, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.09755 · Cited by 292
“Reward for i th agent which is having C ti collisions at time-step t can be written as: We utilized curriculum learning Bengio et al. (2009) The easy version is a junction of two one-way roads on a 7 × 7 grid.”
MethodCurriculum Learning by Dynamic Instance Hardness
Zhou, T., Wang, S., & Bilmes, J. A. (2020) · neural information processing systems · Cited by 105
Kaur, N., Mittal, A., & Singh, G. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11272-6 · Cited by 52
“Inspired by curriculum learning (CL) [6], Nooralahzadeh et al. [50] divided the ARRG process into two steps.”
MethodJalalifar, S. A., Soliman, H., Sahgal, A., & Naini, A. S. (2022) · Medical Physics · doi:10.1002/mp.15814 · Cited by 47
“49 Since outcome prediction using single 2D images can be extremely difficult in some cases, for a smoother network training, the network was initially trained only on the MRI slices of each lesion with a larger tumor cross-section.”
Multimodal Hate Speech Detection via Cross-Domain Knowledge Transfer
Yang, C., Zhu, F., Liu, G., Han, J., & Hu, S. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548255 · Cited by 60
“Curriculum learning [2] is a training strategy that mimics the human learning process, which helps to transfer knowledge from simple to difficult.”
BackgroundLi, F., Lang, S., Hong, B., & Reggelin, T. (2023) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02094-4 · Cited by 54
“To accelerate the training process while remaining the policy unchanged, we design a two-stage training strategy under the inspiration of curriculum learning (Bengio et al., 2009).”
HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces
Bounareli, S., Tzelepis, C., Argyriou, V., Patras, I., & Tzimiropoulos, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00657 · Cited by 80
“We train the proposed HyperReenact framework following a curriculum learning (CL) scheme [7], where we gradually increase the complexity of the training data.”
MethodCurriculum Learning for Heterogeneous Star Network Embedding via Deep Reinforcement Learning
Qu, M., Tang, J., & Han, J. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159711 · Cited by 49
“[4] rst proposed the concept of curriculum learning, and applied the idea to the tasks of shape recognition and language modeling.”
BackgroundMethodArtificial Intelligence for Prosthetics: Challenge Solutions
Kidziński, Ł., Ong, C. F., Mohanty, S. P., Hicks, J. L., Carroll, S. F., Zhou, B., Zeng, H., Wang, F., Lian, R., Tian, H., et al. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-29135-8_4 · Cited by 64
“Our framework combines Reinforcement Learning with Curriculum Learning [51].”
InfluentialMachines Learning Trends, Perspectives and Prospects in Education Sector
Jalil, N. A., Hwang, H. J., & Dawi, N. M. (2019) · International Conference on Education and Multimedia Technology (ICEMT) · doi:10.1145/3345120.3345147 · Cited by 61
“Human experiments also call for new developments in machine teaching, such as the sequential teaching setting [15], [16], [17], [18], [19], [20], [21], [22], [23].”
BackgroundSelf-paced Multi-view Co-training
Ma, F., Meng, D., Dong, X., & Yang, Y. (2020) · UTS ePRESS (University of Technology Sydney) · Cited by 64
Intra-domain task-adaptive transfer learning to determine acute ischemic stroke onset time
Zhang, H., Polson, J., Nael, K., Salamon, N., Yoo, B. Y., El-Saden, S. M., Scalzo, F., Speier, W. F., & Arnold, C. W. (2021) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2021.101926 · Cited by 53
“A task-adaptive approach, which uses the same data set for pretraining and then refines the model using two different label sets, has been demonstrated to increase feature reuse and enhance performance (Elman, 1993; Bengio et al., 2009).”
MethodUCL: Unsupervised Curriculum Learning for water body classification from remote sensing imagery
Abid, N., Shahzad, M., Malik, M. I., Schwanecke, U., Ulges, A., Kovács, G., & Faisal Shafait, B. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2021.102568 · Cited by 49
Charagram: Embedding Words and Sentences via Character n-grams
Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02789 · Cited by 36
A neural network walks into a lab: towards using deep nets as models for human behavior
Ma, W. J., & Peters, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02181 · Cited by 39
Li, Q., Jia, X., Wang, S., & Yan, J. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72995-9_9 · Cited by 42
On the Limit of English Conversational Speech Recognition
Tüske, Z., Saon, G., & Kingsbury, B. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-211 · Cited by 41
Curriculum Learning for Dense Retrieval Distillation
Zeng, H., Zamani, H., & Vinay, V. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531791 · Cited by 41
“Motivated by curriculum learning’s ability to find better local optima [1], we propose a framework called CL-DRD that introduces an iterative optimization process in which the difficulty level of the training data produced using the teacher model, as made available to the student, increases at…”
BackgroundMethodShan, Y., Yao, Y. K., Zhou, X., Zhao, T., Hu, B., & Wang, L. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3324302 · Cited by 56
“The core concept of DFL derived from curriculum learning [10], which states that the focus on low-confidence samples should be gradually increased during training and fine-tuned in the final stage to enhance the generalization ability of the model.”
BackgroundMethodDecouple and weight semi-supervised semantic segmentation of remote sensing images
Huang, W., Shi, Y., Xiong, Z., & Zhu, X. X. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.04.010 · Cited by 63
Liu, Y., Liu, Z., Wang, G., et al. (2024) · Aerospace Science and Technology · doi:10.1016/j.ast.2024.109812 · Cited by 41
Exploration in relational domains for model-based reinforcement learning
Lang, T., Toussaint, M., & Kersting, K. (2012) · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · doi:10.5555/2503308.2503360 · Cited by 64
Discovering the Spatial Extent of Relative Attributes
Xiao, F., & Lee, Y. J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.171 · Cited by 36
“This idea is related to selfpaced learning in the machine learning literature [22, 1], which has been applied to various computer vision tasks such as object discovery and tracking [25, 27, 38].”
MethodDenoising Gravitational Waves with Enhanced Deep Recurrent Denoising Auto-encoders
Shen, H., George, D., Huerta, E. A., & Zhao, Z. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683061 · Cited by 105
“We found that gradually reducing SNRs during training, an idea taken from curriculum learning literature [8], provides regularization, which allows the network to distill more accurate information of the underlying signals with larger SNRs to signals with lower SNRs.”
BackgroundMethodMulti-modal AD classification via self-paced latent correlation analysis
Zhu, Q., Yuan, N., Huang, J., Hao, X., & Zhang, D. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.04.066 · Cited by 40
Medical-based Deep Curriculum Learning for Improved Fracture Classification
Jiménez-Sánchez, A., Mateus, D., Kirchhoff, S., Kirchhoff, C., Biberthaler, P., Navab, N., Ballester, M. Á. G., & Piella, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32226-7_77 · Cited by 48
“[2] showed that a significant improvement in the generalization of the machine learning algorithm could be achieved when samples were presented from “easy” to “hard”, where the “hardness” of the samples was determined by some heuristic or a human expert.”
BackgroundCho, Y. S., Cho, K., Park, C. J., Chung, M. J., Kim, J. H., Kim, K., Kim, Y. K., Kim, H., Ko, J., Cho, B. H., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-63887-8 · Cited by 51
“Second, we utilized curriculum learning, which applies simple learning concepts first and then gradually introduces more complex concepts 25 .”
MethodAdversarial Weighting for Domain Adaptation in Regression
de Mathelin, A., Richard, G., Deheeger, F., Mougeot, M., & Vayatis, N. (2021) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai52525.2021.00015 · Cited by 62
“A work on Bayesian instance-based transfer learning [24] highlights the links between instance-based domain adaptation and curriculum learning [2].”
BackgroundDistilling Ordinal Relation and Dark Knowledge for Facial Age Estimation
Zhao, Q., Dong, J., Yu, H. L., & Chen, S. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3009523 · Cited by 42
“This training strategy is inspired by the curriculum learning [48]: first, learn ordinal representations via the hint/guided layer transferring, and then, train the whole student network jointly with a small relative weight for hint-based teaching.”
BackgroundMethodInfluentialTransductive semi-supervised metric learning for person re-identification
Chang, X., Ma, Z., Wei, X., Hong, X., & Gong, Y. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107569 · Cited by 41
Flexible Working Memory Through Selective Gating and Attentional Tagging
Kruijne, W., Bohté, S. M., Roelfsema, P. R., & Olivers, C. N. L. (2020) · Neural Computation · doi:10.1162/neco_a_01339 · Cited by 46
“Given the complexity of the task, we trained the agents through curriculum learning (Bengio, Louradour, Collobert, & Weston, 2009; Zaremba & Sutskever, 2014; Graves et al., 2016), a training scheme in which trials were organized into levels, which gradually increased in difficulty.”
MethodAdaptive Task Sampling for Meta-learning
Liu, C., Wang, Z., Sahoo, D., Fang, Y., Zhang, K., & Hoi, S. C. H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_44 · Cited by 71
“Specifically, [5] considers instances that increase difficulty.”
A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
Su, J.-C., Cheng, Z., & Maji, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01277 · Cited by 66
“The pseudo-labels can be added iteratively to induce a “curriculum” [4,8,17].”
BackgroundLearning From Multiple Expert Annotators for Enhancing Anomaly Detection in Medical Image Analysis
Le, K. H., Tran, T. V., Pham, H., Nguyen, H. T., Le, T., & Nguyen, H. Q. (2023) · IEEE Access · doi:10.1109/access.2023.3243845 · Cited by 56
“Works in the group (ii) are instances of broader topics such as curriculum learning [29], which is biologically inspired by human gradual learning, with easier examples are preferred in early training stages; learning with noisy labels [30, 31], which prefers examples with smaller training losses…”
BackgroundEnd-to-End Answer Chunk Extraction and Ranking for Reading ComprehensionPreprint
Yu, Y., Zhang, W., Hasan, K. F., Yu, M., Xiang, B., & Zhou, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09996 · Cited by 54
Hard pixel mining for depth privileged semantic segmentation
Gu, Z., Niu, L., Zhao, H., & Zhang, L. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3035231 · Cited by 41
“This training strategy is related to curriculum learning [34] and capable of further improving the segmentation performance.”
BackgroundMorerio, P., Cavazza, J., Volpi, R., Vidal, R., & Murino, V. (2017) · International Conference on Computer Vision · Cited by 32
A Model-agnostic Data Manipulation Method for Persona-based Dialogue Generation
Cao, Y., Bi, W., Fang, M., Shi, S., & Tao, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.550 · Cited by 64
Curriculum Contrastive Context Denoising for Few-shot Conversational Dense Retrieval
Mao, K., Dou, Z., & Qian, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531961 · Cited by 71
Gao, Y., Chen, J., Chen, X., Wang, C., Hu, J., Deng, F., & Lam, T. L. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3257541 · Cited by 45
“first proposed by Elman [68] as a psychological concept, and it was then argued by Bengio that it could be used in the DRL systems [69].”
MethodVideo-Based Visible-Infrared Person Re-Identification With Auxiliary Samples
Du, Y., Lei, C., Zhao, Z., Dong, Y., & Su, F. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3337972 · Cited by 53
“• 9 state-of-the-art VI-ReID methods are reproduced on BUPTCampus, and our method outperforms them by a large margin.”
MethodKiani, R., & Sheng, V. S. (2024) · Electronics · doi:10.3390/electronics13122295 · Cited by 47
Khomenko, V., Shyshkov, O., Radyvonenko, O., & Bokhan, K. (2016) · IEEE International Conference on Data Stream Mining & Processing (DSMP) · doi:10.1109/dsmp.2016.7583516 · Cited by 63
“Many researchers have focused their efforts on experiments with different network architectures and parallelization techniques [6, 9 and 10].”
BackgroundImage Difficulty Curriculum for Generative Adversarial Networks (CuGAN)
Soviany, P., Ardei, C., Ionescu, R. T., & Leordeanu, M. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093408 · Cited by 60
“In this paper, we study the hypothesis of improving the training process of GANs in terms of both accuracy and time, by employing curriculum learning [3].”
BackgroundMethodInfluentialAdaptive Importance Learning for Improving Lightweight Image Super-Resolution Network
Zhang, L., Wang, P., Shen, C., Liu, L., Wei, W., Zhang, Y., & van den Hengel, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01253-6 · Cited by 45
“For example, in (Bengio et al, 2009), the curriculum is derived based on the variability in shape to enable shapes with less variability being learned earlier.”
BackgroundLearning Representations for Facial Actions From Unlabeled Videos
Li, Y., Zeng, J., & Shan, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3011063 · Cited by 51
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.