Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 101–200 of 5,240
Medical image classification using synergic deep learning
Zhang, J., Xie, Y., Wu, Q., & Xia, Y. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.02.010 · Cited by 397
Generation and evaluation of synthetic patient data
Goncalves, A., Ray, P., Soper, B., Stevens, J., Coyle, L., & Sales, A. P. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-00977-1 · Cited by 388
“This is similar to the idea of curriculum learning [53].”
ResultDynamic Task Prioritization for Multitask Learning
Guo, M., Haque, A., Huang, D.-A., Yeung, S., & Fei-Fei, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_17 · Cited by 356
“[24], curriculum learning divides a single task into simpler subtasks which are presented to a model in increasing difficulty.”
BackgroundInfluentialNTIRE 2018 Challenge on Single Image Super-Resolution: Methods and Results
Timofte, R., Gu, S., Wu, J., Van Gool, L., Zhang, L., Yang, M.-H., Haris, M., Shakhnarovich, G., Ukita, N., Hu, S., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00130 · Cited by 341
“In the training phase, curriculum learning [5] strategy has been adopted to achieve better SR performance and shorter training time.”
MethodMultiPoseNet: Fast Multi-Person Pose Estimation Using Pose Residual Network
Kocabas, M., Karagoz, S., & Akbas, E. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_26 · Cited by 360
“We utilized a sort of curriculum learning [51] by sorting annotations based on number of keypoints and bounding box areas.”
MethodDisentangling User Interest and Conformity for Recommendation with Causal Embedding
Zheng, Y., Gao, C., Li, X., He, X., Li, Y., & Jin, D. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449788 · Cited by 356
“Inspired by curriculum learning [7], we adopt an easy-to-hard strategy on training DICE by adding decay on margin values and loss weights.”
MethodRosetta: Large Scale System for Text Detection and Recognition in Images
Borisyuk, F., Gordo, A., & Sivakumar, V. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219861 · Cited by 353
“The second approach was based on curriculum learning [6], i.e ., starting with a simpler problem and increasing the difficulty as the model improves.”
BackgroundDeep Learning in Medical Imaging
Kim, M., Yun, J., Cho, Y., Shin, K., Jang, R., Bae, H.-J., & Kim, N. (2019) · Neurospine · doi:10.14245/ns.1938396.198 · Cited by 351
“21 This curriculum learning with weak labeling of high-scale chest X-ray scans performed well for classification of 5 disease patterns and required less preparation to train the model.”
BackgroundKoller, O., Camgoz, N. C., Ney, H., & Bowden, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2911077 · Cited by 342
“Comparable to [4] andwith a similar effect as ‘curriculum learning’ [69], we observe fastest convergence if we first train a CNN-HMM model with randomly shuffled input images for 4 EM re-alignment iterations starting from a flat start in each stream.”
BackgroundEntity-Relation Extraction as Multi-Turn Question Answering
Li, X., Yin, F., Sun, Z., Li, X., Yuan, A., Chai, D., Zhou, M., & Li, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1129 · Cited by 336
“For the RESUME dataset, we use the strategy of curriculum learning (Bengio et al., 2009), i.e., we gradually increase the number of turns from 2 to 4 at training.”
MethodMultimodal MR Synthesis via Modality-Invariant Latent Representation
Chartsias, A., Joyce, T., Giuffrida, M. V., & Tsaftaris, S. A. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2764326 · Cited by 341
“We also show in our experiments that our architecture and cost function can be used in conjunction with various fusion methods, including the one proposed in HeMIS [26], and the model can be trained end-to-end without the need for the added complexity of curriculum learning [14].”
BackgroundMethodImage classification with deep learning in the presence of noisy labels: A survey
Algan, G., & Ulusoy, I. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106771 · Cited by 337
“Curriculum learning (CL) [161], inspired from human cognition, proposes to start from easy samples and go”
BackgroundMotionBERT: A Unified Perspective on Learning Human Motion Representations
Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., & Wang, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01385 · Cited by 348
“We first train on 3D data only for 30 epochs, then train on both 3D data and 2D data for 60 epochs, following the curriculum learning practices [8,111].”
MethodFan, T., Long, P., Liu, W., & Pan, J. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364920916531 · Cited by 332
“Run policy p u for T i time-steps, collecting f o ti , r ti , a ti g , where t 2 ½ 0 , T i 6: Estimate advantages using GAE (Schulman et al., 2015b) (Bengio et al., 2009), we propose a two-stage training process, which accelerates the policy to converge to a satisfactory solution, and gets higher…”
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
“Our approach may be seen as a form of implicit curriculum learning (Elman, 1993; Bengio et al., 2009).”
Kellenberger, B., Marcos, D., & Tuia, D. (2018) · Remote Sensing of Environment · doi:10.1016/j.rse.2018.06.028 · Cited by 335
“We propose to countersteer this e ff ect based on the concept of curriculum learning (Bengio et al., 2009), where a model is adaptively trained with di ff erent portions of the dataset.”
MethodCurricular Contrastive Regularization for Physics-Aware Single Image Dehazing
Zheng, Y., Zhan, J., He, S., Dong, J., & Du, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00560 · Cited by 339
“[3] formally propose the curriculum learning strategy to arrange the training samples according to their difficulty.”
BackgroundWebly Supervised Learning of Convolutional Networks
Chen, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.168 · Cited by 351
“After seventh layer, another fully connected layer is used to predict class labels.”
MethodCurriculumNet: Weakly Supervised Learning from Large-Scale Web Images
Guo, S., Huang, W., Zhang, H., Zhuang, C., Dong, D., Scott, M. R., & Huang, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_9 · Cited by 319
“Curriculum learning was originally proposed in [1].”
Teacher–Student Curriculum Learning
Matiisen, T., Oliver, A., Cohen, T., & Schulman, J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2934906 · Cited by 320
“One approach to overcome this problem is to use curriculum learning [6]–[9], where tasks are ordered by increasing difficulty and training only proceeds to harder tasks once easier ones are mastered.”
BackgroundInfluentialCompetence-based Curriculum Learning for Neural Machine Translation
Platanios, E. A., Stretcu, O., Neubig, G., Poczos, B., & Mitchell, T. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1119 · Cited by 304
“Bengio et al. (2009) propose a curriculum learning approach that aims to address these issues by changing the mini-batch sampling strategy.”
BackgroundMethodInfluentialDeep Reinforcement Learning-Based Image Captioning with Embedding Reward
Ren, Z., Wang, X., Zhang, N., Lv, X., & Li, L.-J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.128 · Cited by 331
“To handle this problem, we follow [35] to apply curriculum learning [1] to train our actor-critic model.”
MethodThe Developing Infant Creates a Curriculum for Statistical Learning
Smith, L. B., Jayaraman, S., Clerkin, E., & Yu, C. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.02.004 · Cited by 299
“Machine learning approaches such as curriculum learning and iterative teaching explicitly seek to optimize learning by ordering the training material in time [59–61].”
BackgroundA survey on incorporating domain knowledge into deep learning for medical image analysis
Xie, X., Niu, J., Liu, X., Chen, Z., Tang, S., & Yu, S. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101985 · Cited by 311
“This training pattern can be introduced in the training process of deep neural networks via curriculum learning [104].”
BackgroundSelf-Paced Learning with Diversity
Jiang, L., Meng, D., Yu, S.-I., Lan, Z., Shan, S., & Hauptmann, A. G. (2014) · neural information processing systems · Cited by 410
Zaremba, W., & Sutskever, I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.4615 · Cited by 283
“This strategy is has been examined in previous work on curriculum learning (Bengio et al., 2009).”
BackgroundMethodInfluentialSelf-Paced Learning for Long-Term Tracking
Supancic III, J. S., & Ramanan, D. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.308 · Cited by 290
“Our main contribution is an algorithm that minimizes drift by carefully choosing which frames from which to learn, using the framework of self-paced learning [2, 3].”
BackgroundMethodPolyglot: Distributed Word Representations for Multilingual NLP
Al‐Rfou, R., Perozzi, B., & Skiena, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.1662 · Cited by 289
“In our work, we start from the example construction method outlined in (Bengio et al., 2009).”
MethodPhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
Yu, Z., Shen, Y., Shi, J., Zhao, H., Torr, P., & Zhao, G. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00415 · Cited by 300
“Curriculum learning [2], as a major machine learning regime with philosophy of easy-to-hard curriculum, is utilized to train PhysFormer.”
MethodDenoising Implicit Feedback for Recommendation
Wang, W., Feng, F., He, X., Nie, L., & Chua, T.-S. (2021) · Proceedings of the 14th ACM International Conference on Web Search and Data Mining · doi:10.1145/3437963.3441800 · Cited by 290
“The findings also support the prior theory in robust learning and curriculum learning [2, 14].”
BackgroundResultInfluentialMulti-Modal Curriculum Learning for Semi-Supervised Image Classification
Gong, C., Tao, D., Maybank, S. J., Liu, W., Kang, G., & Yang, J. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2563981 · Cited by 285
“This learning approach was proposed by [10], which hypothesizes that curriculum learning is able to boost the convergence speed of the training process as well as find a better local minima than the existing solvers for non-convex problems.”
BackgroundEasy Samples First: Self-paced Reranking for Zero-Example Multimedia Search
Jiang, L., Meng, D., Mitamura, T., & Hauptmann, A. G. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654918 · Cited by 271
“f(v; k) is the self-paced function, and in [19], it is defined based on the l1 norm of v ∈ [0, 1] :”
BackgroundMethodInfluentialSakaridis, C., Dai, D., & Van Gool, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00747 · Cited by 281
“This is in line with the concept of curriculum learning [2].”
ResultDataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., & Choi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.746 · Cited by 283
“Similar intuitions have guided other work such as curriculum learning (Bengio et al., 2009) and self-paced learning (Kumar et al., 2010; Lee and Grauman, 2011) where all examples are prioritized based on their “difficulty”.”
BackgroundAdversarial Geometry-Aware Human Motion Prediction
Gui, L.-Y., Wang, Y.-X., Liang, X., & Moura, J. M. F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_48 · Cited by 276
“Some work [12,24] alleviates this problem via a noise scheduling scheme [6] by adding noise to the input during training; nevertheless, this scheme makes the prediction discontinuous and makes the hyper-parameters difficult to tune.”
BackgroundSpeaker-Independent Speech Separation With Deep Attractor Network
Luo, Y., Chen, Z., & Mesgarani, N. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2795749 · Cited by 260
“During training, we split the input into 100-frame long segments, and then continue training the network with 400-frame long segments, which is known as the curriculum training strategy [25].”
MethodMassively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1907.05019 · Cited by 279
“We highlight a few open problems along these lines: Task Scheduling: Scheduling tasks has been widely studied in the context of multitask learning and meta-learning, but remains relatively under-explored for multilingual NMT (Bengio et al., 2009; Pentina et al., 2015).”
BackgroundLearning From Multiple Experts: Self-paced Knowledge Distillation for Long-Tailed Classification
Xiang, L., Ding, G., & Han, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_15 · Cited by 268
“Similar to human learning, designing an organized curriculum has proved to be effective in the context of machine learning as well [1].”
BackgroundWeakly Supervised Object Localization and Detection: A Survey
Zhang, D., Han, J., Cheng, G., & Yang, M.-H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3074313 · Cited by 277
“By integrating the curriculum learning [5] with the self-paced learning [84], the established learning framework provides a more theo-retical-sounded way to improve the learning robustness.”
Pulmonary nodule classification with deep residual networks
Nibali, A., He, Z., & Wollersheim, D. (2017) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-017-1605-6 · Cited by 276
“[2] suggests that gradually increasing the difficulty of examples as training progresses can be beneficial.”
BackgroundWeakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
Yang, J., Reed, S., Yang, M., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.00706 · Cited by 246
“To effectively train our recurrent network, we found it beneficial to use curriculum learning [4], in which we gradually increase the difficulty of training by increasing the trajectory length.”
Method3DRegNet: A Deep Neural Network for 3D Point Registration
Pais, G. D., Ramalingam, S., Govindu, V. M., Nascimento, J. C., Chellappa, R., & Miraldo, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00722 · Cited by 268
“Taking inspiration from [4, 37, 42], we propose the use of Curriculum Learning (CL) data augmentation.”
MethodLearning Multiagent Communication with Backpropagation
Sukhbaatar, S., Szlam, A. D., & Fergus, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07736 · Cited by 261
A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems
Silva, F. L. D., & Costa, A. H. R. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11396 · Cited by 273
An Embedding Cost Learning Framework Using GAN
Yang, J., Ruan, D., Huang, J., Kang, X., & Shi, Y.-Q. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2019.2922229 · Cited by 268
“4 bpp for 120,000 iterations, then fine-tuned the parameters for other payloads by using curriculum learning [21], [25], [34].”
BackgroundLanguage2Pose: Natural Language Grounded Pose Forecasting
Ahuja, C., & Morency, L.-P. (2019) · 2019 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2019.00084 · Cited by 271
“Starting with simpler examples before moving on to tougher ones can be beneficial to the training process [4, 37, 36].”
BackgroundUnifying distillation and privileged information
Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015) · arXiv · doi:10.48550/arxiv.1511.03643 · Cited by 270
“distillation sheds light to applications in semi-supervised learning, domain adaptation, transfer learning, Universum learning (Weston et al., 2006), reinforcement learning, and curriculum learning (Bengio et al., 2009); some of them discussed in our numerical simulations. 2 DISTILLATION We focus on c-class classificat…”
BackgroundMethodInfluentialModel Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding
Sakaridis, C., Dai, D., Hecker, S., & Van Gool, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_42 · Cited by 262
“the dense synthetic foggy dataset from step6and the light real foggy one from step5. CMAda adapts segmentation models from light synthetic fog to dense real fog and is inspired by curriculum learning [4], in the sense that we rst solve easier tasks with our synthetic data, i.e. fog density estimation and semantic scen…”
MethodA Fully Progressive Approach to Single-Image Super-Resolution
Wang, Y., Perazzi, F., McWilliams, B., Sorkine-Hornung, A., Sorkine-Hornung, O., & Schroers, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00131 · Cited by 262
“In this framework, we can naturally utilize a form of curriculum learning, which is known to improve training [4] by organizing the learning process from easy (small upsampling factors) to hard (large upsampling factors).”
BackgroundMethodREMIND Your Neural Network to Prevent Catastrophic Forgetting
Hayes, T. L., Kafle, K., Shrestha, R., Acharya, M., & Kanan, C. (2019) · arXiv · doi:10.48550/arxiv.1910.02509 · Cited by 268
“We hypothesize that the q-type ordering may be acting as a natural curriculum [11], allowing our streaming model to train more efficiently.”
BackgroundO2U-Net: A Simple Noisy Label Detection Approach for Deep Neural Networks
Huang, J., Qu, L., Jia, R., & Zhao, B. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00342 · Cited by 261
“Recent studies [4, 7] leverage curriculum learning [2] to build noise-robust models.”
BackgroundPractical recommendations for gradient-based training of deep architecturesPreprint
Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5533 · Cited by 266
“The hypothesis proposed in (Bengio et al., 2009) is that curriculum learning can act similarly to a continuation method, i.”
BackgroundInfluentialSelf-citationDeep Learning for Video Game Playing
Justesen, N., Bontrager, P., Togelius, J., & Risi, S. (2020) · IEEE Transactions on Games · doi:10.1109/tg.2019.2896986 · Cited by 257
“Curriculum learning attempts to speed up learning by training on a set of progressively harder environments [7].”
BackgroundC-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection
Wan, F., Liu, C., Ke, W., Ji, X., Jiao, J., & Ye, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00230 · Cited by 248
“In machine learning, curriculum learning [5] was inspired by this principle to define a sequence of gradually increasing difficulty training tasks (or training distributions) which converge to the task of interest.”
BackgroundOng, B. T., Sugiura, K., & Zettsu, K. (2016) · Neural Computing and Applications · doi:10.1007/s00521-015-1955-3 · Cited by 261
“A previous approach that supports the findings of this paper has been introduced in [34].”
MethodAdversarial Representation Learning for Text-to-Image Matching
Sarafianos, N., Xu, X., & Kakadiaris, I. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00591 · Cited by 250
“[32] demonstrated that a joint representation can be learned from facial and voice information and introduced a curriculum learning strategy [3, 40, 41] to perform hard negative mining during training.”
BackgroundDynamic Curriculum Learning for Imbalanced Data Classification
Wang, Y., Gan, W., Yang, J., Wu, W., & Yan, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00512 · Cited by 261
“The idea of curriculum learning was originally proposed in [1], it demonstrates that the strategy of learning from easy to hard significantly improves the generalization of the deep model.”
BackgroundMissing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network
Sharma, A., & Hamarneh, G. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2945521 · Cited by 254
“4) Curriculum learning: In order to train our proposed method we use a curriculum learning (CL) [45]”
A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation
Han, D., Mulyana, B., Stankovic, V., & Cheng, S. (2023) · Sensors · doi:10.3390/s23073762 · Cited by 253
“Curriculum learning [41] is a training method in which the complexity of the data samples used increases gradually over time.”
Clothes-Changing Person Re-identification with RGB Modality Only
Gu, X., Chang, H., Ma, B., Bai, S., Shan, S., & Chen, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00113 · Cited by 256
“This is consistent with curriculum learning [2].”
Learning a Deep ConvNet for Multi-Label Classification With Partial Labels
Durand, T., Mehrasa, N., & Mori, G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00074 · Cited by 245
“The idea of Curriculum Learning is inspired by the way humans learn: start to learn with easy samples/subtasks, and then gradually increase the difficulty level of the samples/subtasks.”
BackgroundMethodInfluentialLearning by Playing - Solving Sparse Reward Tasks from Scratch
Riedmiller, M., Hafner, R., Lampe, T., Neunert, M., Degrave, J., Van de Wiele, T., Mnih, V., Heess, N., & Springenberg, J. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10567 · Cited by 242
“…and execution of different auxiliary tasks can be understood from the perspective of “teaching” a set of increasingly more complicated problems – see e.g. the literature on curriculum learning (Bengio et al., 2009) – where we consider a fixed number of problems and learn a teaching policy online.”
BackgroundMask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes
Liao, M., Lyu, P., He, M., Yao, C., Wu, W., & Bai, X. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2937086 · Cited by 244
“[27] applied a curriculum learning paradigm [1] in the training period, where the sub-network for text recognition is locked at the early iterations and the training data for each period is carefully selected.”
MethodPoint in, Box Out: Beyond Counting Persons in Crowds
Liu, Y., Shi, M., Zhao, Q., & Wang, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00663 · Cited by 239
“Instead of training the model on the entire set once, we adopt a curriculum learning strategy [3, 38, 48] to train the model from images of relatively accurate and easy pseudo ground truth first.”
BackgroundMethodREMIND Your Neural Network to Prevent Catastrophic Forgetting
Hayes, T. L., Kafle, K., Shrestha, R., Acharya, M., & Kanan, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58598-3_28 · Cited by 248
“We hypothesize that the q-type ordering may be acting as a natural curriculum [11], allowing our streaming model to train more efficiently.”
BackgroundPark, D., Kang, D. U., Kim, J., & Chun, S. Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_20 · Cited by 234
“Our MT approach, that can be seen as another form of curriculum learning [5] in a wide sense, also breaks down a challenging deblurring problem into a series of easy deblurring problems with small blurs.”
BackgroundTowards End-to-End Text Spotting with Convolutional Recurrent Neural Networks
Li, H., Wang, P., & Shen, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.560 · Cited by 230
“Curriculum Learning In order to improve generalization and accelerate the convergence speed, we design a curriculum learning [1] paradigm to train the model from gradually more complex data.”
MethodXiao, T., Zhang, J., Yang, K., Peng, Y., & Zhang, Z. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654926 · Cited by 234
“While many incremental learning works paid more attention on efficiency than accuracy, Bengio [2] offered a deeper insight that speed and quality can be obtained simultaneously if increments are made properly, noted as curriculum learning.”
BackgroundMethodContinuous Learning AI in Radiology: Implementation Principles and Early Applications
Pianykh, O. S., Langs, G., Dewey, M., Enzmann, D. R., Herold, C. J., Schoenberg, S. O., & Brink, J. A. (2020) · Radiology · doi:10.1148/radiol.2020200038 · Cited by 237
Adaptive Graph Encoder for Attributed Graph Embedding
Cui, G., Zhou, J., Yang, C., & Liu, Z. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403140 · Cited by 235
“aining set is denoted by O. Accordingly, our cross entropy loss is given by L= Õ (v i,v j)∈O −lij log(sij)−(1−lij)log(1−sij). (15) 3.4.2 Thresholds Update. Inspired by the idea of curriculum learning [1], we design a specific update strategy forrpos and rneд to control the size of training set. At the beginning of tra…”
MethodInfluentialCurriculum learning of multiple tasks
Pentina, A., Sharmanska, V., & Lampert, C. H. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299188 · Cited by 213
“In [4] Bengio et al. showed experimentally that choosing training examples in an order of gradually increasing difficulty can lead to faster training and higher prediction quality.”
BackgroundText2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction
Lu, Y., Lin, H., Xu, J., Han, X., Tang, J., Li, A., Sun, L., Liao, M., & Chen, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.217 · Cited by 229
“To address the above challenges, we employ a curriculum learning (Bengio et al., 2009; Xu et al., 2020) strategy.”
MethodMulti-scale Deep Learning for Gesture Detection and Localization
Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_33 · Cited by 229
“allowing the network to learn more and more sophisticated concepts (as it is done, for example, in curriculum learning [39]).”
BackgroundGe, W., Yang, S., & Yu, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00139 · Cited by 210
“Following the divide-and-conquer idea in curriculum learning [3], we decompose the pixel labeling task into three stages: the image level stage, the instance level stage and the pixel level stage.”
BackgroundMethodInfluentialA Comparison of Transformer and LSTM Encoder Decoder Models for ASR
Zeyer, A., Bahar, P., Irie, K., Schluter, R., & Ney, H. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9004025 · Cited by 218
“Alongside this scheduling, some curriculum learning [54] can also be important.”
BackgroundCurriculum Learning for Reinforcement Learning Domains: A Framework and Survey
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., & Stone, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04960 · Cited by 224
“Bengio et al. (2009) and Taylor (2009) both emphasize the importance of better understanding how humans approach designing curricula.”
BackgroundInfluentialMachine Teaching: An Inverse Problem to Machine Learning and an Approach Toward Optimal Education
Zhu, X. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9761 · Cited by 216
“Human experiments also call for new developments in machine teaching, such as the sequential teaching setting (Bengio et al. 2009; Khan, Zhu, and Mutlu 2011; McCandliss et al. 2002; Kumar, Packer, and Koller 2010; Lee and Grauman 2011; Cakmak and Lopes 2012; Pashler and Mozer 2013; Kobayashi and…”
BackgroundProgressive Learning for Person Re-Identification With One Example
Wu, Y., Lin, Y., Dong, X., Yan, Y., Bian, W., & Yang, Y. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2891895 · Cited by 211
Weisfeiler-Lehman Neural Machine for Link Prediction
Zhang, M., & Chen, Y. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3097996 · Cited by 216
“We adopt the Adam update rule [35] for optimization with Table 3: Comparison of different graph labeling methods (K=10).”
MethodOne-Pass Multi-Task Networks With Cross-Task Guided Attention for Brain Tumor Segmentation
Zhou, C., Ding, C., Wang, X., Lu, Z., & Tao, D. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2973510 · Cited by 219
“As the three tasks are of increasing difficulty levels, we propose to train OM-Net more effectively based on curriculum learning [40], which is useful for improving the convergence quality of machine learning models.”
MethodSoviany, P., & Ionescu, R. T. (2018) · 2018 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc.2018.00041 · Cited by 217
“Based on the principles of curriculum learning [5], we hypothesize that using more complex (two-stage) object detectors for difficult images and less complex (single-stage) detectors for easy images will provide an optimal trade-off between accuracy and speed, without ever having to change anything about the object de…”
BackgroundMethodFog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather
Hahner, M., Sakaridis, C., Dai, D., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01500 · Cited by 221
“Trying out more sophisticated techniques like curriculum learning [1] is kept for future work.”
MethodFace recognition: Past, present and future (a review)
Taskiran, M., Kahraman, N., & Erdem, C. E. (2020) · Digital Signal Processing · doi:10.1016/j.dsp.2020.102809 · Cited by 217
Approximating Explicit Model Predictive Control Using Constrained Neural Networks
Chen, S., Saulnier, K., Atanasov, N., Lee, D. D., Kumar, V., Pappas, G. J., & Morari, M. (2018) · 2018 Annual American Control Conference (ACC) · doi:10.23919/acc.2018.8431275 · Cited by 209
“Applying a continuation method involves tracking the minimizing solution of increasingly non-smooth cost functions as η goes from 0 to N , and has been empirically shown to yield better solutions, especially in optimizing DNNs [48].”
End-to-End Speech Recognition: A Survey
Prabhavalkar, R., Hori, T., Sainath, T. N., Schlüter, R., & Watanabe, S. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3328283 · Cited by 223
“Sequencelearningapproachesalsoconsidercurriculumlearn-ing [171], [172], e.g., by considering short sequences first [173], [174]; interim increase of sub-sampling [147] initially more sub-sampling; or, for multi-speaker ASR training sort mixed speech by SNR and start with speakers of balanced energy…”
BackgroundAdversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 208
“Instead of randomly sampling a mismatching negative image sample, in Text-SeGAN [65] several strategies based on curriculum learning [66] are introduced to select negative samples with gradually increasing semantic difficulty.”
MethodPath Planning Method With Improved Artificial Potential Field—A Reinforcement Learning Perspective
Yao, Q., Zheng, Z., Qi, L., Yuan, H., Guo, X., Zhao, M., Liu, Z., & Yang, T. (2020) · IEEE Access · doi:10.1109/access.2020.3011211 · Cited by 207
“To enhance the adaptability of the agent in BHDQN, we use the curriculum learning (CL) [42].”
MethodSelf-supervised 6D Object Pose Estimation for Robot Manipulation
Deng, X., Xiang, Y., Mousavian, A., Eppner, C., Bretl, T., & Fox, D. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9196714 · Cited by 207
“idea [39] to bootstrap the system sequentially.”
Ophthalmic diagnosis using deep learning with fundus images – A critical review
Sengupta, S., Singh, A., Leopold, H. A., Gulati, T., & Lakshminarayanan, V. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101758 · Cited by 201
“[94] used curriculum learning [95] in DCNN’s to achieve better results using a reduced set of training examples.”
MethodDiffsound: Discrete Diffusion Model for Text-to-Sound Generation
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., & Yu, D. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3268730 · Cited by 211
“Thus, a curriculum learning [48] strategy is proposed.”
BackgroundScan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention
Bluche, T., Louradour, J., & Messina, R. (2017) · 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2017.174 · Cited by 198
“Finally, we applied several levels of curriculum learning [29].”
MethodSelf-citationMulti-Agent Deep Reinforcement Learning for Multi-Robot Applications: A Survey
Orr, J., & Dutta, A. (2023) · Sensors · doi:10.3390/s23073625 · Cited by 206
“Curriculum learning [197] has also been used for multi-robot path planning in [198], where the path planning is modeled as a lesson, going from easy to hard difficulty levels.”
BackgroundPhaseNet for Video Frame Interpolation
Meyer, S., Djelouah, A., McWilliams, B., Sorkine-Hornung, A., Gross, M., & Schroers, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00059 · Cited by 184
“This training procedure can be seen as a form of curriculum learning [7] that aims at improving training by gradually increasing the difficulty of the learning task.”
BackgroundCombinatorial Optimization and Reasoning with Graph Neural Networks
Cappart, Q., Chételat, D., Khalil, E. B., Lodi, A., Morris, C., & Veličković, P. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/595 · Cited by 202
“Executingmultiple related algorithms simultaneously or in a curriculum [8] In this case, the processor network is shared across algorithms and becomes a multi-task learner.”
BackgroundAntonio, G.-P., & Maria-Dolores, C. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3169907 · Cited by 198
EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching
Song, X., Zhao, X., Hu, H., & Fang, L. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20873-8_2 · Cited by 198
“The decoder is formulated as a residual pyramid inspired by curriculum and residual learning [3,14].”
MethodSalvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200
“The agent uses a curriculum learning approach [91], in which ξ , υ ′(k), and ζ ′(k) alternately assume the form of isometrically scaled versions of Fast Gradient Sign Method (FGSM) [92].”
MethodBetween-Class Learning for Image Classification
Tokozume, Y., Ushiku, Y., & Harada, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00575 · Cited by 187
“Furthermore, we assume that the usage of curriculum learning [4] would be helpful to speed up the training; namely, at the early stage, we generate a mixing ratio close to 0 or 1 and input relatively pure examples, and we gradually change the distribution of r to flat.”
BackgroundLi, A., Liu, W., Zheng, C., Fan, C., & Li, X. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3079813 · Cited by 192
“Inspiredbycurriculumlearningconcept[28],multi-stage mappingalgorithmsbegintothriveinthespeechfront-endtasks recently[29]–[34],wheretheoriginalmappingproblemcanbe decomposedintomultiplesimplersub-problemsandabetter solutioncanbeobtainedineachstageprogressivelyasaresult.”
Paraphrase Generation with Deep Reinforcement Learning
Li, Z., Jiang, X., Shang, L., & Li, H. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1421 · Cited by 201
“We employ a curriculum learning strategy (Bengio et al., 2009) for it.”
Method
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.