Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 401–500 of 5,240
An Empirical Exploration of Curriculum Learning for Neural Machine TranslationPreprint
Zhang, X., Kumar, G., Khayrallah, H., Murray, K., Gwinnup, J., Martindale, M. J., McNamee, P., Duh, K., & Carpuat, M. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00739 · Cited by 138
“Bengio et al. (2009) improve neural language model training using a curriculum based on increasing vocabulary size.”
BackgroundMethodInfluentialWebly-Supervised Fine-Grained Visual Categorization via Deep Domain Adaptation
Xu, Z., Huang, S., Zhang, Y., & Tao, D. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2637331 · Cited by 98
“Self training approaches such as bootstrapping [31] and curriculum learning [30] are particularly useful when there is considerable uncertainty in the training data, such as in webly-supervised learning problems.”
BackgroundLearning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix
Luo, B., Feng, Y., Wang, Z., Zhu, Z., Huang, S., Yan, R., & Zhao, D. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1040 · Cited by 97
“The idea of curriculum learning (Bengio et al., 2009) is simple: starting with the easiest aspect of a task, and leveling up the dif-ficulty gradually, which fits well to our problem.”
BackgroundMIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition
Chang, X., Zhang, W., Qian, Y., Roux, J. L., & Watanabe, S. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003986 · Cited by 97
“According to previous researches, starting from easier subtasks can lead the model to learn better, an approach called curriculum learning [30, 31].”
BackgroundBinocular Mutual Learning for Improving Few-shot Classification
Zhou, Z., Qiu, X., Xie, J., Wu, J., & Zhang, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00829 · Cited by 133
“Treating all tasks equally during cross-epoch training like [33] will lead to an “unhealthy” phenomenon: further performance improvement is hard to obtain in the later training, because the model is dynamically growing while the difficulty is static [2].”
BackgroundNot All Samples Are Created Equal: Deep Learning with Importance SamplingPreprint
Katharopoulos, A., & Fleuret, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00942 · Cited by 283
“Bengio et al. (2009) manually design a sampling scheme inspired by the perceived way that human children learn; in practice they provide the network with examples of increasing difficulty in an arbitrary manner.”
BackgroundCurriculum Pre-training for End-to-End Speech Translation
Wang, C., Wu, Y., Liu, S., Zhou, M., & Yang, Z. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.344 · Cited by 135
“This idea is inspired by the human learning process and is first applied in the context of machine learning by Bengio et al. (2009).”
BackgroundMethodEfficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 240
ESAM: Discriminative Domain Adaptation with Non-Displayed Items to Improve Long-Tail Performance
Chen, Z., Xiao, R., Li, C., Ye, G., Sun, H., & Deng, H. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401043 · Cited by 140
“More-over, based on curriculum learning [1], the model may learn target-discriminative information from reliable samples, thereby transforming more samples into reliable samples.”
BackgroundSharma, D., Purushotham, S., & Reddy, C. K. A. (2021) · Scientific Reports · doi:10.1038/s41598-021-98390-1 · Cited by 133
“The novel characteristics of our answer generation decoder module are as follows: • Teacher Forcing: Due to the inherent complexity of the task of sequence generation, the decoder is susceptible to a slower convergence rate.”
MethodDist-YOLO: Fast Object Detection with Distance Estimation
Vajgl, M., Hurtík, P., & Nejezchleba, T. (2022) · Applied Sciences · doi:10.3390/app12031354 · Cited by 122
“Therefore, the suitable way to train Dist-YOLO is to use curriculum learning [53].”
Adversarial Generation of Natural Language
Rajeswar, S., Subramanian, S., Dutil, F., Pal, C., & Courville, A. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10929 · Cited by 88
“As a way to provide training signals of incremental difficulty, we use curriculum learning (Bengio et al., 2009) and train our generator to produce sequences of gradually increasing lengths as training progresses.”
MethodMorales, E. F., Murrieta-Cid, R., Becerra, I., & Esquivel-Basaldua, M. A. (2021) · Intelligent Service Robotics · doi:10.1007/s11370-021-00398-z · Cited by 101
“An alternative approach is to use a self-play scheme for learning a complex task [157]; it is in a way, a curriculum learning [10] approach where the tasks to learn are dynamically defined.”
MethodMitash, C., Bekris, K. E., & Boularias, A. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202206 · Cited by 120
“Curriculum learning [28] and self-paced learning [29] have been adapted to improve the performance of object detectors [30], [31].”
Norm-Based Curriculum Learning for Neural Machine Translation
Liu, X., Lai, H., Wong, D. F., & Chao, L. S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.41 · Cited by 106
“Curriculum learning (CL), which aims to train machine learning models better and faster (Bengio et al., 2009), is gaining an intuitive appeal to both academic and industrial NMT systems.”
BackgroundThe AI economist: taxation policy design via two-level deep multiagent reinforcement learning
Zheng, S., Trott, A., Srinivasa, S., Parkes, D. C., & Socher, R. (2022) · Science Advances · doi:10.1126/sciadv.abk2607 · Cited by 105
“To stabilize learning, our approach combines two key ideas: curriculum learning ( 21 ) and entropy regularization ( 22 ).”
Using human brain activity to guide machine learning
Fong, R., Scheirer, W. J., & Cox, D. (2018) · Scientific Reports · doi:10.1038/s41598-018-23618-6 · Cited by 124
“This work builds on previous machine learning approaches that weight training 8,37 , but here we propose to do such weighting using a separate stream of data, derived from human brain activity.”
MethodProduction-level facial performance capture using deep convolutional neural networks
Laine, S., Karras, T., Aila, T., Herva, A., Saito, S., Yu, R., Li, H., & Lehtinen, J. (2017) · Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation · doi:10.1145/3099564.3099581 · Cited by 92
“The augmentation ramp-up process can be seen as a form of curriculum learning [Bengio et al. 2009].”
BackgroundLearning Contextual and Attentive Information for Brain Tumor Segmentation
Zhou, C., Chen, S., Ding, C., & Tao, D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11726-9_44 · Cited by 136
Learning Sequential Distribution System Restoration via Graph-Reinforcement Learning
Zhao, T., & Wang, J. (2021) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2021.3102870 · Cited by 119
Medical Image Segmentation With Limited Supervision: A Review of Deep Network Models
Peng, J., & Wang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3062380 · Cited by 132
“For example, it is still unclear how to automatically design an adaptive curriculum for the given segmentation task instead of using a predefined curriculum [131] and when will curriculum-like strategies, especially data curriculum, benefit the deep model”
BackgroundA Model of Two Tales: Dual Transfer Learning Framework for Improved Long-tail Item Recommendation
Zhang⋆, Y., Cheng, D. Z., Yao, T., Yi, X., Hong, L., & Ed, H. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450086 · Cited by 148
“large amount of tail items and we further enhanced the metamapping to tail items by learning connections among items via curriculum learning. Curriculum Learning. The curriculum learning, proposed by [4], has shown good performance in different areas [15, 19]. The basic idea is to simulates the humans learning process…”
BackgroundA General Approach for the Automation of Hydraulic Excavator Arms Using Reinforcement Learning
Egli, P., & Hutter, M. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3152865 · Cited by 107
Curriculum Learning for Speech Emotion Recognition From Crowdsourced Labels
Lotfian, R., & Busso, C. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2898816 · Cited by 127
“[24] considered discrete numbers of λ and also discrete weights (0 or 1) for the training examples (z) at each step.”
BackgroundMethodInfluentialDeep Learning Logo Detection with Data Expansion by Synthesising Context
Su, H., Zhu, X., & Gong, S. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.65 · Cited by 107
“Curriculum Learning (CL), designed for reducing the difficulties in learning tasks by easy-to-hard staged learning [4].”
BackgroundMethodA Survey of Label-noise Representation Learning: Past, Present and FuturePreprint
Han, B., Yao, Q., Liu, T., Niu, G., Tsang, I. W.-H., Kwok, J. T.-Y., & Sugiyama, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.04406 · Cited by 245
“Their designs are based on different principles, such as gradient clipping [35] and curriculum learning [89].”
BackgroundMulti-task Curriculum Transfer Deep Learning of Clothing Attributes
Dong, Q., Gong, S., & Zhu, X. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.64 · Cited by 105
Deep Self-Evolution Clustering
Chang, J., Meng, G., Wang, L., Xiang, S., & Pan, C. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889949 · Cited by 104
“For this purpose, DSEC gradually selects labeled pair-wise patterns with high likelihood with the progress of the learning, which is inspired from the curriculum learning [54] in the supervised learning.”
BackgroundMin-Entropy Latent Model for Weakly Supervised Object Detection
Wan, F., Wei, P., Han, Z., Jiao, J., & Ye, Q. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2898858 · Cited by 87
“The non-convex optimization problem is turned into a proximate problem, which is easier to be optimized [49], [50].”
BackgroundAl-Sharman, M., Dempster, R., Daoud, M. A., Nasr, M., Rayside, D., & Melek, W. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3285440 · Cited by 127
“CL was proposed in [29] as a way to accelerate learning by first training the system on simple tasks and thereafter progressively increasing the difficulty of the tasks given to the learning agent.”
BackgroundReinforcement Learning Neural Turing MachinesPreprint
Zaremba, W., & Sutskever, I. (2015) · arXiv (Cornell University) · Cited by 113
(2021) · Journal of Machine Learning Research · Cited by 224
“Such methods often utilize curricula provided by human designers [28, 322, 300], though several recent methods seek to automatically generate a curriculum [273, 374, 115].”
BackgroundViZDoom Competitions: Playing Doom From Pixels
Wydmuch, M., Kempka, M., & Jaśkowski, W. (2018) · IEEE Transactions on Games · doi:10.1109/tg.2018.2877047 · Cited by 169
“Tian)—the winning bot of Track 1 was trained with a variant of the A3C algorithm [19] with curriculum learning [2].”
MethodMeta-Transfer Learning through Hard Tasks
SUN, Q., LIU, Y., CHEN, Z., CHUA Tat-Seng, & Schiele, B. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 95
Curriculum Manager for Source Selection in Multi-source Domain Adaptation
Yang, L., Balaji, Y., Lim, S.-N., & Shrivastava, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_36 · Cited by 146
“To address the MS-UDA problem, we propose an adversarial agent that learns a dynamic curriculum [4] for multiple source domains, named Curriculum Manager for Source Selection (CMSS).”
MethodA Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images
Messina, P., Pino, P., Parra, D., Soto, A., Besa, C., Uribe, S., Andía, M., Tejos, C., Prieto, C., & Capurro, D. (2022) · ACM Computing Surveys · doi:10.1145/3522747 · Cited by 142
“For example, curriculum learning [15] and self-paced learning [80] could be used to imitate the learning curve from easier to harder instances that radiologists go through when they learn to interpret and diagnose images.”
BackgroundLi, J., Meng, Y., Li, Y., Cui, Q., Yang, X., Tao, C., Wang, Z., Li, L., & Zhang, W. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.128202 · Cited by 111
Toward Development of a Face Recognition System for Watchlist Surveillance
Kamgar-Parsi, B., Lawson, W., & Kamgar-Parsi, B. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.68 · Cited by 100
Unsupervised Topic Modeling for Short Texts Using Distributed Representations of Words
Sridhar, V. K. R. (2015) · Workshop on Vector Space Modeling for Natural Language Processing (VSM-NLP) · doi:10.3115/v1/w15-1526 · Cited by 97
“While they are typically induced in the context of a deep neural network framework for a given task (Bengio et al., 2003; Collobert and Weston, 2008; Bengio et al., 2009; Turian et al., 2010; Mikolov et al., 2010), recent work in (Mikolov et al., 2013) has also shown that they can also be induced…”
BackgroundToward Training Recurrent Neural Networks for Lifelong Learning
Sodhani, S., Chandar, S., & Bengio, Y. (2020) · Neural Computation · doi:10.1162/neco_a_01246 · Cited by 87
“Specifically, we train the models in the curriculum style setup (Bengio, Louradour, Collobert, & Weston, 2009) where the tasks are ordered by difficulty.”
MethodSelf-citationDomain Adaptation with BERT-based Domain Classification and Data Selection
Ma, X., Xu, P., Wang, Z., Nallapati, R. M., & Xiang, B. (2019) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/d19-6109 · Cited by 131
“Curriculum Learning: Curriculum Learning (Bengio et al., 2009) deals with the question of how to use prior knowledge about the difficulty of the training examples, to boost the rate of learning and the performance of the final model.”
BackgroundThe Best Defense Is a Good Offense: Adversarial Attacks to Avoid Modulation Detection
Hameed, M. Z., György, A., & Gündüz, D. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3025441 · Cited by 101
“We consider two different training strategies: (i) randomly shuffle the training data of all channel SNR values to train the intruder’s DNN; and (ii) curriculum learning [36], where the training data is arranged in descending order of their SNR”
MethodA Deep Learning Approach for Blind Drift Calibration of Sensor Networks
Wang, Y., Yang, A., Chen, X., Wang, P., Wang, Y., & Yang, H. (2017) · IEEE Sensors Journal · doi:10.1109/jsen.2017.2703885 · Cited by 110
“We apply a simple curriculum learning [33] strategy on selecting the drift emulation parameters σ 0 , σ b , σ d and σ n .”
MethodTextWorld: A Learning Environment for Text-based GamesPreprint
Côté, M.-A., Kádár, Á., Yuan, X., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.11532 · Cited by 93
“The CommAI framework [Baroni et al., 2017] emphasizes the ability to generate curricula [Bengio et al., 2009], so that agents may learn incrementally from environments of increasing complexity.”
MethodBui, T., Ribeiro, L., Ponti, M. A., & Collomosse, J. P. (2018) · Computers & Graphics · doi:10.1016/j.cag.2017.12.006 · Cited by 108
Tay, Y., Wang, S., Luu, A. T., Fu, J., Phan, M. C., Yuan, X., Rao, J., Hui, S. C., & Zhang, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1486 · Cited by 88
“Our work draws inspiration from curriculum learning (CL) (Bengio et al., 2009).”
BackgroundInfluentialLearning Fast Sample Re-weighting Without Reward Data
Zhang, Z., & Pfister, T. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00076 · Cited by 153
“This phenomenon is prominent at early training phases [51, 34] because models tend to learn simple samples before hard (mislabeled) samples [45, 3].”
BackgroundIterative Machine TeachingPreprint
Liu, W., Dai, B., Humayun, A., Tay, C., Yu, C., Smith, L. B., Rehg, J. M., & Song, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10470 · Cited by 148
“Curriculum learning (Bengio et al., 2009)”
BackgroundInfluentialOptimal Client Sampling for Federated LearningPreprint
Chen, W., Horváth, S., & Richtárik, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13723 · Cited by 92
“For instance, a manually designed sampling scheme was proposed in Bengio et al. (2009).”
MethodJia, X., Xing, X., Yuan, Y., Xing, L., & Meng, M. Q. ‐. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2950506 · Cited by 100
“It is noteworthy that the curriculum learning strategy [111] was utilized in the training procedure.”
MethodUD-MIL: Uncertainty-Driven Deep Multiple Instance Learning for OCT Image Classification
Wang, X., Tang, F., Chen, H., Luo, L., Tang, Z., Ran, A. R., Cheung, C. Y.-L., & Heng, P. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2983730 · Cited by 98
“Our approach differs from self-paced learning or curriculum learning [48], [49] in that: in our setting, initially, all instances are equally involved in training the instance-level classifier, and the variables R are subsequently updated based on the classifier predictions and further applied to…”
MethodKarim, N., Mithun, N. C., Rajvanshi, A., Chiu, H.-P., Samarasekera, S., & Rahnavard, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02310 · Cited by 163
Re-ID done right: towards good practices for person re-identificationPreprint
Almazán, J., Gajic, B., Murray, N., & Larlus, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05339 · Cited by 102
“Similarly to humans, who learn a set of concepts more easily when the concepts to be learned are presented by increasing degree of complexity, it has been shown that curriculum learning has a positive impact on the speed and quality of the convergence of deep neural networks [3].”
BackgroundGenerative Face CompletionPreprint
Li, Y., Liu, S., Yang, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.05838 · Cited by 91
“To effectively train our network, we use the curriculum strategy [3] by gradually increasing the difficulty level and network scale.”
MethodLIDAR and Position-Aided mmWave Beam Selection With Non-Local CNNs and Curriculum Training
Zecchin, M., Mashhadi, M. B., Jankowski, M., Gündüz, D., Kountouris, M., & Gesbert, D. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3142513 · Cited by 103
“This procedure is motivated by curriculum learning [26], [27], which suggests to expose the training process to easier instances at the initial”
MethodA Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond
Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., & Liu, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3277122 · Cited by 179
“Curriculum learning [100] is a machine learning strategy inspired by human learning, which trains the model by feeding training instances in an order (e.”
BackgroundCardinality estimation with local deep learning models
Woltmann, L., Hartmann, C., Thiele, M., Habich, D., & Lehner, W. (2019) · International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (aiDM) · doi:10.1145/3329859.3329875 · Cited by 107
“The approach of curriculum learning [1] transfers this principle to supervised learning.”
BackgroundFedBalancer: data and pace control for efficient federated learning on heterogeneous clients
Shin, J., Li, Y., Liu, Y., & Lee, S.-J. (2022) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3498361.3538917 · Cited by 143
“sample selection approaches in the field of machine learning research that could be arranged in threefold: (1) Curriculum Learning (CL) [9, 26, 27, 29, 73]: CL is a sample ordering technique which trains a network with easier samples in early training stage and”
BackgroundA survey on intrinsic motivation in reinforcement learning
Aubret, A., Matignon, L., & Hassas, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06976 · Cited by 87
“This issue is currently studied by curriculum learning [ Bengio et al. , 2009 ] .”
BackgroundMultimodal feature fusion for CNN-based gait recognition: an empirical comparison
Castro, F. M., Marín-Jiménez, M. J., Guil, N., & de la Blanca, N. P. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04811-z · Cited by 97
“For 2D and 3D models, we perform a curriculum learning strategy [52] to speed up and to facilitate the convergence.”
MethodUnsupervised Domain Adaptation for Depth Prediction from Images
Tonioni, A., Poggi, M., Mattoccia, S., & Stefano, L. D. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2940948 · Cited by 90
“Overall the behaviour of t resembles a curriculum learning [76] schedule.”
BackgroundIs neuro-symbolic AI meeting its promises in natural language processing? A structured review
Hamilton, K., Nayak, A., Božić, B., & Longo, L. (2024) · Semantic Web · doi:10.3233/sw-223228 · Cited by 88
“Type of learning Machine learning algorithms are classified as supervised, unsupervised, semi-supervised, curriculum or reinforcement learning, depending on the amount and type of supervision required during training [10,16,79].”
BackgroundLanguage Generation with Recurrent Generative Adversarial Networks without Pre-training
Press, O., Bar, A., Bogin, B., Berant, J., & Wolf, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01399 · Cited by 82
“To succeed in training the model we utilize curriculum learning (Elman, 1993; Bengio et al., 2009), and train over sequences of increasing length.”
MethodExtractive Summarization Using Multi-Task Learning with Document Classification
Isonuma, M., Fujino, T., Mori, J., Matsuo, Y., & Sakata, I. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1223 · Cited by 86
“We introduce curriculum learning (Bengio et al., 2009) to overcome this difficulty.”
Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning
Perez-D'Arpino, C., Liu, C., Goebel, P., Martin-Martin, R., & Savarese, S. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560893 · Cited by 86
“Similar idea is behind methods on curriculum learning [28] that create a series of increasingly”
MethodConvolutional Autoencoders for Human Motion Infilling
Kaufmann, M., Aksan, E., Song, J., Pece, F., Ziegler, R., & Hilliges, O. (2020) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv50981.2020.00102 · Cited by 95
“To induce robustness against the amount of perturbation we employ a curriculum learning training scheme [5].”
MethodMeta-Based Self-Training and Re-Weighting for Aspect-Based Sentiment Analysis
He, K., Mao, R., Gong, T.-L., Li, C., & Cambria, E. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3202831 · Cited by 85
Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias
Panda, R., Zhang, J., Li, H., Lee, J., Lu, X., & Roy–Chowdhury, A. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_36 · Cited by 108
“Our work is related to curriculum learning [33,11,64,19,44,2] that learns a model by gradually including easy to complex samples in training so as to increase the entropy of training samples.”
BackgroundMethodYuan, Z., Mou, L., Wang, Q., & Zhu, X. X. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3173811 · Cited by 97
“Curriculum learning (CL) [48], self-paced learning (SPL) [49], and SPCL [26] are proposed as more reasonable training algorithms for machine learning models.”
BackgroundVariance Reduction in SGD by Distributed Importance Sampling
Alain, G., Lamb, A., Sankar, C., Courville, A. C., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06481 · Cited by 78
“tion of examples, and a good tutor is able to pick examples that are useful for a student to learn the current lesson. This work can therefore be seen as a follow-up on the curriculum learning ideas (Bengio et al., 2009), where the model itself is used to figure out which examples are currently informative for the lear…”
BackgroundMethodSelf-citationNoise-Tolerant Paradigm for Training Face Recognition CNNs
Hu, W., Huang, Y., Zhang, F., & Li, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01216 · Cited by 78
“Based on Curriculum learning [2], MentorNet [18] and Coteach-ing [13] try to select clean samples using the small-loss strategy, but the noise level should be provided in advanced in [13], and a small clean set and a pre-training model are suggested in [18].”
Understanding and Improving Knowledge DistillationPreprint
Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Ed, H., & Jain, S. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03532 · Cited by 202
“Bengio et al. (2009) suggested that this may speedup training convergence and helps optimization to reach a better local minima.”
BackgroundInfluentialTang, G., Kumar, N., & Michmizos, K. P. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340948 · Cited by 112
“This form of curriculum training has been shown to result in better generalization and faster convergence [19], [20].”
BackgroundDomain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey
Saunders, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13566 · Cited by 164
“…forms of curriculum learning are implemented in a similar way to instance weighting, with a higher weight applied to examples that fall into the current section of the curriculum, or a zero weight applied to examples that should not yet be shown to the model (Bengio et al., 2009; Dou et al., 2020).”
BackgroundRelax and focus on brain tumor segmentation
Wang, P., & Chung, A. C. S. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102259 · Cited by 99
Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations
Hu, D., Bao, Y., Wei, L., Zhou, W. G., & Hu, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.606 · Cited by 140
Attacks Which Do Not Kill Training Make Adversarial Learning StrongerPreprint
Zhang, J., Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., & Kankanhalli, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11242 · Cited by 445
“Second, the curriculum learning strategy has been shown to be effective (Bengio et al., 2009).”
BackgroundInfluentialAn Easy-to-hard Learning Paradigm for Multiple Classes and Multiple Labels
Liu, W., Tsang, I. W.-H., & Müller, K. (2017) · MPG.PuRe (Max Planck Society) · Cited by 100
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval
Mithun, N. C., Panda, R., Papalexakis, E. E., & Roy-Chowdhury, A. K. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240712 · Cited by 77
“more complex ones, which leads to a better performance in the final task. It has been shown in many previous works that appropriate curriculum strategies guide the learner towards better local minima [1]. Our idea is to gradually inject difficult information to the learner such that in the early stages of training, th…”
BackgroundRe-Caption: Saliency-Enhanced Image Captioning Through Two-Phase Learning
Zhou, L., Zhang, Y., Jiang, Y., Zhang, T., & Fan, W. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2928144 · Cited by 88
“to realize the curriculum learning [61] in image captioning where the order of training samples needs to be decided.”
BackgroundUncertainty-Aware Curriculum Learning for Neural Machine Translation
Zhou, Y., Yang, B., Wong, D. F., Wan, Y., & Chao, L. S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.620 · Cited by 133
“tive way to speed up the training process and boost the performance of a neural network is to exploit CL (Elman, 1993; Krueger and Dayan, 2009; Bengio et al., 2009).”
BackgroundMethodMulti-USV Cooperative Chasing Strategy Based on Obstacles Assistance and Deep Reinforcement Learning
Gan, W., Qu, X., Song, D., & Yao, P. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3319510 · Cited by 96
The Expressive Power of Word EmbeddingsPreprint
Chen, Y., Perozzi, B., Al‐Rfou, R., & Skiena, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3226 · Cited by 108
“Since (Bengio et al., 2003a), there has been a significant interest in speeding up the generation process (Bengio et al., 2003b; 2009).”
BackgroundMethodSelf-Paced Multi-Task Learning
Li, C., Yan, J., Wei, F., Dong, W., Liu, Q., & Hongyuan Zha, B. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10847 · Cited by 81
“This idea is inspired by the fact that humans often learn from easy concepts to hard ones in the cognitive process (Elman 1993; Bengio et al. 2009).”
BackgroundMethodOvertaking Maneuvers in Simulated Highway Driving using Deep Reinforcement Learning
Kaushik, M., Prasad, V., Krishna, K. M., & Ravindran, B. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500718 · Cited by 94
“I. INTRODUCTION Over the last few decades, the area of autonomous driving has made significant progress owing to the rise of low cost sensors, availability of vast amounts of driving data, and the boom of Learning based methods.”
MethodInfluentialCooke, C., & Scott, K. A. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2892723 · Cited by 88
“be considered an example of curriculum learning [18] in the sense that the CNN is given a slightly easier problem”
BackgroundTeacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection
Dong, X., & Yang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00087 · Cited by 93
“Others exploit pseudo-labels of unlabeled data in the semi-supervised scenario [1, 2, 23, 28].”
BackgroundProgressive learning: A deep learning framework for continual learning
Fayek, H. M., Cavedon, L., & Wu, H. R. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.05.011 · Cited by 103
CDL: Curriculum Dual Learning for Emotion-Controllable Response Generation
Shen, L., & Feng, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.52 · Cited by 85
“559 (Bengio et al., 2009) is to design an evaluation for”
BackgroundLearning Selective Communication for Multi-Agent Path Finding
Ma, Z., Luo, Y., & Pan, J. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3139145 · Cited by 122
“As pointed in [39], presenting training examples not randomly but in a meaningful order will benefit machine learning algorithms.”
A Roadmap Towards Machine Intelligence
Mikolov, T., Joulin, A., & Baroni, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-75477-2_2 · Cited by 80
“This roadmap constitutes only the beginning of a long journey towards AI, and we hope other researchers will be joining it in pursuing the goals it outlined.”
BackgroundNoisy Activation FunctionsPreprint
Gülçehre, Ç., Moczulski, M., Denil, M., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00391 · Cited by 274
“The authors had to use curriculum learning (Bengio et al., 2009) to let the model capture knowledge about the easier examples first and increase the level of difficulty of the examples further down the training.”
BackgroundSelf-citationAsymmetric multi-task learning based on task relatedness and loss
Lee, G., Yang, E., & Hwang, S. J. (2016) · Scholarworks@UNIST (Ulsan National Institute of Science and Technology) · Cited by 130
On-demand Learning for Deep Image Restoration
Gao, R., & Grauman, K. L. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.124 · Cited by 94
“[2].”
One-Pass Multi-task Convolutional Neural Networks for Efficient Brain Tumor Segmentation
Zhou, C., Ding, C., Lu, Z., Wang, X., & Tao, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00931-1_73 · Cited by 104
Progressive Ensemble Networks for Zero-Shot Recognition
Ye, M., & Guo, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01200 · Cited by 86
“In [4], progressive curriculum learning is used to train a model on "easy-to-hard" samples with a pre-defined scheme.”
MethodProCAN: Progressive growing channel attentive non-local network for lung nodule classification
Al-Shabi, M., Shak, K., & Tan, M. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108309 · Cited by 90
Enhanced physics‐informed neural networks for hyperelasticity
Abueidda, D., Korić, S., Guleryuz, E., & Sobh, N. (2022) · International Journal for Numerical Methods in Engineering · doi:10.1002/nme.7176 · Cited by 88
“For instance, they have proposed using curriculum PINN regularization as a potential technique to enhance PINNs’ performance [28].”
BackgroundDetector discovery in the wild: Joint multiple instance and representation learning
Hoffman, J., Pathak, D., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298906 · Cited by 75
“Approaches to better optimize the non-convex problem involve using multifold learning as a measure of regularizing overfitting [8], optimize Latent SVM for the area under the ROC curve (AUC) [6] and training with easy examples in beginning to avoid bad local optimization [5, 24].”
MethodLearning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning
Tsvetkov, Y., Faruqui, M., Wang, L., MacWhinney, B., & Dyer, C. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1013 · Cited by 86
“…research focusing on curriculum strategies in NLP is scarce, and has conventionally been following a paradigm of “starting small” (Elman, 1993), i.e., initializing the learner with “simple” examples first, and then gradually increasing data complexity (Bengio et al., 2009; Spitkovsky et al., 2010).”
BackgroundMethodInfluential
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.