Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 4,001–4,100 of 5,240
StencilTorch: An Iterative and User-Guided Framework for Anime Lineart Colorization
Hati, Y., Thevenin, V., Nolot, F., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25825-1_1 · Cited by 1
“Curriculum leaning [3,6] is a type of pipeline in which a model is trained on tasks that gradually increase in difficulty.”
BackgroundImproving Robustness: When and How to Minimize or Maximize the Loss Variance
Balaban, V., Bidkhori, H., & Bogdan, P. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00229 · Cited by 2
“This fact simplifies the implementation, but hinders the robustness since not all samples are equal in practice [28].”
BackgroundLiu, H., Zhao, P., Xu, T., Bian, Y., Huang, J., Zhu, Y., & Mu, Y. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583211 · Cited by 3
“Our essential insight is that the poisoning method can learn trivial yet easier graph poisoning tasks at the beginning and then fully exploit the learned adversarial knowledge to tackle harder ones, in a fashion of curriculum learning [3].”
BackgroundLanYUAN, a GPT large model using Curriculum Learning and Sparse Attention
Zhou, G., Zhang, Y., Hu, R., & Zhang, Y. (2023) · International Conference on Computing, Networks and Internet of Things (CNIOT) · doi:10.1145/3603781.3603827 · Cited by 2
Curriculum-guided Abstractive Summarization for Mental Health Online Posts
Sotudeh, S., Goharian, N., Deilamsalehy, H., & Dernoncourt, F. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.17 · Cited by 10
Multi-Agent Language Learning: Symbolic Mapping
Feng, Y., & Lu, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.491 · Cited by 1
“Instead, a natural process is probably that a language is first formed in simple tasks, and then applied in more complex tasks, meanwhile it evolves to become more complicated and complete, similar to the concept of curriculum learning (Bengio et al., 2009).”
BackgroundHuCurl: Human-induced Curriculum Discovery
Elgaar, M., & Amiri, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.104 · Cited by 10
“In terms of data ordering, sub-sampling approaches sample the easiest or hardest instances at every training iteration (Bengio et al., 2009; Kumar et al., 2010; Guo et al., 2018; Platan-ios et al., 2019; Xu et al., 2020), sample weighting techniques weight instances according to their estimated…”
BackgroundInfluentialOn the effect of curriculum learning with developmental data for grammar acquisition
Opper, M., Morrison, J., & Siddharth, N. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.31 · Cited by 5
Mi, M. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.23 · Cited by 1
“In the language modelling experiment carried out by Bengio et al. (2009), a corpus replacement method was used to make the data increasingly difficult.”
BackgroundMethodA surprisal oracle for active curriculum language modeling
Hong, X., Loáiciga, S., & Sayeed, A. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.22 · Cited by 1
“To counteract this, Jafarpour et al. (2021) use CL, a technique that mimics how humans learn by regulating the training according to some schedule criterion, e.g., easy to difficult or short to long examples (Bengio et al., 2009).”
MethodCrowd Counting on Heavily Compressed Images with Curriculum Pre-Training
Bakhtiarnia, A., Zhang, Q., & Iosifidis, A. (2023) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci52147.2023.10371805 · Cited by 1
“Training can start with the simplest examples and harder examples can be gradually introduced during the training process, which can ultimately lead to higher accuracy [17].”
BackgroundNeural Network Architectures for Prepositional Phrase Attachment Disambiguation
Belinkov, Y. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1
Minimal supervision for language learning: bootstrapping global patterns from local knowledge
Roth, D., & Connor, M. (2011) · Cited by 1
Toward Implicit Sample Noise Modeling: Deviation-driven Matrix FactorizationPreprint
Lee, G.-H., Yang, S., & Lin, S.-D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09274 · Cited by 1
“For instance, (Bengio et al. 2009; Kumar, Packer, and Koller 2010) propose to organize, rather than randomly present as SG, data points in a meaningful manner for training a better non-convex model.”
BackgroundMethodCorrecting Writing Errors with Convolutional Neural Networks
Dronen, N. (2016) · CU Scholar (University of Colorado Boulder) · Cited by 1
Target Curricula via Selection of Minimum Feature Sets: a Case Study in Boolean NetworksPreprint
Fenn, S., & Moscato, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04721 · Cited by 1
“The typical explanation for the success of example curricula in non-convex scenarios is that it imposes a form of transfer learning where the simpler concept is discovered first and subsequently informs the more complex concept (Bengio et al., 2009).”
BackgroundDomain adaptation for statistical machine translation and neural machine translation
Zhang, J. (2017) · Dublin City University Open Access Institutional Repository (Dublin City University) · Cited by 2
Deictic Image Maps: An Abstraction For Learning Pose Invariant Manipulation PoliciesPreprint
Platt, R., Kohler, C., & Gualtieri, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10045 · Cited by 1
“Curriculum learning: Another thing that can speed up training is curriculum learning (Bengio et al. 2009).”
BackgroundAmanuensis: The Programmer's ApprenticePreprint
Dean, T., Chiang, M., Gomez, M., Gruver, N., Hindy, Y., Lam, M. S., Lu, P. J., Sanchez, S., Saxena, R., Smith, M. R., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00082 · Cited by 1
“regarded as the easiest to come up with, but even then it is easier if you start with simple I/O pairs like [A; B]![A; B];[B; A]![A; B]and work up to longer lists — referred to as curriculum learning [6]. As Dan Abolafia pointed out in his class presentation, it is relative easy to learn to sort lists of length no more…”
BackgroundDeep Learned Full-3D Object Completion from Single ViewPreprint
Rethage, D., Tombari, F., Achilles, F., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.06843 · Cited by 1
“ations from 10 classes of ModelNet. Finally,thebestperforminghigh-resolutionnetworkwasfinetuned with a data set of 30 classes, in a sense applying curFigure 2. Compressing Auto-encoder riculum learning[1]. 2.1. Architecture The CNN architecture consists of three convolutional layers and two fully-connected layers, with…”
MethodInfluentialA Fully Sequential Methodology for Convolutional Neural Networks.Preprint
Huang, Y., Wu, R., & Ou, P. (2018) · arXiv (Cornell University) · Cited by 1
Liu, K.-C., Shen, Y., Klopp, J. P., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09781 · Cited by 1
“[44] applies domain adaptation in a curriculum learning [4] fashion, learning scene parsing from tasks that are less sensitive to the aforementioned domain discrepancy.”
BackgroundTTR-Based Reward for Reinforcement Learning with Implicit Model PriorsPreprint
Lyu, X., & Chen, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09762 · Cited by 1
Deep Likelihood Network for Image Restoration with Multiple Degradation LevelsPreprint
Guo, Y., Lu, M., Zuo, W., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09105 · Cited by 1
“Curriculum [3] and self-paced learning [23] approaches are hence proposed to control the sampling strategies and allocate more training samples to more challenging degradation levels with relatively lower PSNRs [16].”
BackgroundAll Neural Networks are Created EqualPreprint
Hacohen, G., Choshen, L., & Weinshall, D. (2019) · arXiv (Cornell University) · Cited by 1
Generalized Coarse-to-Fine Visual Recognition with Progressive TrainingPreprint
Ren, X., Xie, L., Wei, C., Qiao, S., Su, C., Liu, J., Tian, Q. C., Fishman, E. K., & Yuille, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12047 · Cited by 1
“It is named curriculum learning [2], which measures the diffi-culty of each training sample and adjusts data distribution during the training process, so that the model is optimized with gradually increasing complexity, and thus becomes more stable and less likely to over-fit data.”
BackgroundInfluentialWang, W., Caswell, I. R., & Chelba, C. I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01130 · Cited by 1
“Curriculum learning (CL) (Bengio et al., 2009) has been used to further improve traditional static selection.”
MethodLow-Shot Learning from Imaginary 3D ModelPreprint
Pahde, F., Puscas, M., Wolff, J., Klein, T., Sebe, N., & Nabi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01868 · Cited by 1
“[1] first proposed a general learning strategy: curriculum learning.”
BackgroundOutcome Prediction and Hierarchical Models in Real-Time Strategy Games
Stanescu, A. M. (2019) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3-rb0t-4v23 · Cited by 1
Robust Optimization for Deep RegressionPreprint
Belagiannis, V., Rupprecht, C., Carneiro, G., & Navab, N. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.06606 · Cited by 12
“Furthermore, weighting training samples also conforms with the idea of curriculum [5] and self-paced learning [22], where each training sample has different contribution to the minimization depending on its error.”
BackgroundReinforcement Learning based Curriculum Optimization for Neural Machine TranslationPreprint
Kumar, G., Foster, G. F., Cherry, C., & Krikun, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.00041 · Cited by 16
“The idea of a curriculum was popularized by Bengio et al. (2009), who viewed it as a way to improve convergence by presenting heuristicallyidentified easy examples first.”
BackgroundCurriculum Learning for Handwritten Text Line RecognitionPreprint
Louradour, J., & Kermorvant, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.1737 · Cited by 1
“This idea has been exploited in classification [9], grammar induction [10], [11], robotics [12], cognitive science [13] and human teaching [14].”
BackgroundMethodInfluentialSelf-citationMin-Entropy Latent Model for Weakly Supervised Object DetectionPreprint
Wan, F., Wei, P., Han, Z., Jiao, J., & Ye, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06057 · Cited by 5
Dynamic Feature Scaling for Online Learning of Binary ClassifiersPreprint
Bollegala, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.7584 · Cited by 1
“One promising criterion for selecting examples for training is curriculum learning [1].”
BackgroundLearning Aligned Cross-Modal Representations from Weakly Aligned DataPreprint
Castrejón, L., Aytar, Y., Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.13016/m2dnjv-coqd · Cited by 12
On-Demand Learning for Deep Image RestorationPreprint
Gao, R., & Grauman, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01380 · Cited by 1
“[2].”
Comparing Task Simplifications to Learn Closed-Loop Object Picking Using Deep Reinforcement Learning
Breyer, M., Furrer, F., Novkovic, T., Siegwart, R., & Nieto, J. (2018) · arXiv · doi:10.48550/arxiv.1803.04996 · Cited by 1
“methodology of curriculum learning [1], we guide the training of our models by progressively increasing the workspace with the agent’s performance and compare this method against reward function shaping and using a heuristic to bootstrap the full problem.”
BackgroundMethodLalor, J. P., Wu, H., & Yu, H. Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11421 · Cited by 1
“Curriculum learning (CL) is a training procedure where models are trained to learn simple concepts before more complex concepts are introduced (Bengio et al., 2009).”
BackgroundFast Task-Adaptation for Tasks Labeled Using Natural Language in Reinforcement LearningPreprint
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04040 · Cited by 5
“A I] 9 O ct 2 Current algorithms capable of quickly adapting their control policies to solve related tasks, mostly rely on intensive training using a diverse set of tasks, often guided by a curriculum of increasingly more difficult and diverse tasks (Bengio et al., 2009).”
BackgroundMapping high-performance RNNs to in-memory neuromorphic chipsPreprint
Nair, M. V., & Indiveri, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10692 · Cited by 4
“We adopt a curriculum learning (Bengio et al., 2009) procedure for this task, by first training the RNN cell on a short sequence and progressively increasing the number of time steps.”
MethodInfluentialLearning to Auto Weight: Entirely Data-driven and Highly Efficient Weighting FrameworkPreprint
Li, Z., Wu, Y., Chen, K., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11058 · Cited by 1
“Although common techniques such as cost-sensitive learning (Lin et al. 2017) and curriculum learning (Bengio et al. 2009; Kumar, Packer, and Koller 2010) demonstrate the effectiveness of example reweighting, they are usually predefined for specific tasks with prior knowledge and require laborious…”
MethodAutomated curriculum generation for Policy Gradients from DemonstrationsPreprint
Srinivasan, A., Bahdanau, D., Chevalier-Boisvert, M., & Bengio, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.00444 · Cited by 1
“Bengio et al. [2009] proposed curriculum learning and showed that pre-training an LSTM (for language modelling) with easier tasks meant training on more complex tasks could happen much faster.”
BackgroundMethodInfluentialSelf-citationNeural Subgraph Isomorphism CountingPreprint
Liu, X., Pan, H., He, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11589 · Cited by 1
“We also find that using curriculum learning (Bengio et al., 2009) can help models to converge better.”
MethodEfficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural NetworksPreprint
Siqueira, H., Magg, S., & Wermter, S. (2020) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2001.06338 · Cited by 141
“Thus, we trained the ESR in a curriculum learning fashion (Bengio et al. 2009), where ESR-9, trained for discrete facial expression recognition, was fine-tuned for arousal and valence prediction.”
MethodHide-and-Tell: Learning to Bridge Photo Streams for Visual StorytellingPreprint
Jung, Y., Kim, D., Woo, S., Kim, K., Kim, S.-J., & Kweon, I. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00774 · Cited by 1
“To ease the training difficulty in early steps, we adopt a curriculum learning scheme (Bengio et al. 2009).”
BackgroundMethodInfluentialDynamic Graph Correlation Learning for Disease Diagnosis with Incomplete LabelsPreprint
Liu, D., Xu, S., Zhou, P., He, K., Wei, W., & Xu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11629 · Cited by 2
CityLearn: Diverse Real-World Environments for Sample-Efficient Navigation Policy LearningPreprint
Chancán, M. (2020) · doi:10.36227/techrxiv.12063582 · Cited by 1
Kicki, P., Gawron, T., & Skrzypczyński, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.00946 · Cited by 5
“We used the curriculum learning technique [18] and train the network first to perform the overtaking maneuver, then perpendicular parking task, and finally angle parking.”
MethodProgressive Multi-Stage Learning for Discriminative TrackingPreprint
Li, W., Li, X., Bourahla, O. E. F., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00255 · Cited by 1
“Curriculum learning is a learning paradigm inspired by the intrinsic learning principle of humans/animals [47].”
BackgroundAttention-based Assisted Excitation for Salient Object Segmentation.Preprint
Masoudnia, S., Kheirieh, M., Vahabie, A., & Araabi, B. N. (2020) · arXiv (Cornell University) · Cited by 1
Data Annealing for Informal Language Understanding TasksPreprint
Gu, J., & Yu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13833 · Cited by 1
“Curriculum learning suggests a proper order of training data improves the performance and speed up training process in single dataset setting (Bengio et al., 2009).”
BackgroundWord Interdependence Exposes How LSTMs Compose RepresentationsPreprint
Saphra, N., & Lopez, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13195 · Cited by 1
“For feedforward language models, a curriculum-based approach has been shown to improve performance from small training data (Bengio et al., 2009).”
BackgroundChang, H., Xu, Z., & Tomizuka, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04213 · Cited by 1
“There are other methods seeking to learn a globally general policy: Curriculum learning (CL)(Bengio et al. (2009)) trains a model on a sequence of cognate tasks that get more and more challenging gradually, so as to solve hard tasks that could not be learned from scratch.”
BackgroundPyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASRPreprint
Shao, Y., Wang, Y., Povey, D., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09824 · Cited by 1
“Similar to the findings in [14], we also find out that curriculum training [31] (i.”
ResultBridging the Gap Between Training and Inference for Spatio-Temporal ForecastingPreprint
Liu, H., & Lee, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09343 · Cited by 1
“[1] further improve the idea by a Curriculum Learning [2] based approach to close the gap between training and inference by sampling previous ground-truth and previously predicted context by a changing probability.”
BackgroundDynamics-Aware Latent Space Reachability for Exploration in Temporally-Extended Tasks.Preprint
Bharadhwaj, H., Garg, A., & Shkurti, F. (2020) · arXiv (Cornell University) · Cited by 3
Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer ArchitecturePreprint
Tanneberg, D., Rueckert, E., & Peters, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00926 · Cited by 1
“The algorithmic solutions are learned in a curriculum learning setup (Bengio et al., 2009) with sampling from old lessons (Zaremba & Sutskever, 2014) to prevent unlearning and to foster generalization.”
BackgroundIterative Machine Teaching without TeachersPreprint
Yang, M., & Baba, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15339 · Cited by 3
“Curriculum learning is a general training strategy that presents teaching examples ranging from easy to difficult [11].”
BackgroundLearning to Read through Machine TeachingPreprint
Sen, A., Cox, C. R., Borkenhagen, M. J. C., Seidenberg, M. S., & Zhu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16470 · Cited by 1
“Gains in the efficiency of learning could mitigate limitations on human learning arising from perception, attention, memory, and other aspects of human cognition [1, 25].”
BackgroundTENet: Triple Excitation Network for Video Salient Object DetectionPreprint
Ren, S., Han, C., Yang, X., Han, G., & He, S. (2020) · European Conference on Computer Vision · doi:10.48550/arxiv.2007.09943 · Cited by 72
“This strategy shares a similar spirit to the curriculum learning framework [2], in which they argue that training with an easy task at first then continues with more difficult tasks gradually may leads to better optimization.”
MethodCurriculum Learning for Recurrent Video Object SegmentationPreprint
Gonzalez-i-Calabuig, M., Ventura, C., & Giró-i-Nieto, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.06698 · Cited by 1
“Curriculum learning [3] is a training strategy for machine learning that consists of presenting simple concepts to the model first to, gradually, increasing their complexity.”
BackgroundP-DIFF: Learning Classifier with Noisy Labels based on Probability Difference DistributionsPreprint
Hu, W., Zhao, Q., Huang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06382 · Cited by 1
“Based on Curriculum learning [35], some recent proposed approaches select clean training samples 2020 25th International Conference on Pattern Recognition (ICPR) Milan, Italy, Jan 10-15, 2021”
MethodOLALA: Object-Level Active Learning Based Layout Annotation.Preprint
Shen, Z., Zhao, J., Dell, M., Yu, Y., & Li, W. (2020) · arXiv (Cornell University) · Cited by 2
Self-play deep learning for games: Maximising experiences
West, J. (2020) · Queensland University of Technology · doi:10.5204/thesis.eprints.204194 · Cited by 1
Neural Correction Model for Open-Domain Named Entity RecognitionPreprint
Zhu, M., Deng, Z., Xiong, W., Yu, M., Zhang, M., & Wang, W. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06058 · Cited by 6
“Curriculum learning (Bengio et al. 2009) is a training strategy proposed by Bengio et al.”
BackgroundSelf-paced Data Augmentation for Training Neural NetworksPreprint
Takase, T., Karakida, R., & Asoh, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15434 · Cited by 1
“Here, we discuss why the proposed SPA method performed well in consideration of two well-established training strategies, i.e., curriculum learning [7] and desirable changes to loss function instability [25].”
BackgroundMethodHu, B.-G., & Qu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06156 · Cited by 4
“1) Objective functions and constraints [65], 2) Activation functions or kernels [78], [79], 3) Weight initialization [80], [81], 4) Searching steps [82], 5) Meta learning [83], 6) Transfer learning [84], 7) Curriculum learning and self-paced learning [85], [86], 8) Dropout [87].”
BackgroundGonnochenko, A., Semochkin, A., Egorov, D. E., Statovoy, D., Zabihifar, S., Postnikov, A., Seliverstova, E., Zaidi, A. M., Stemmler, J., & Limkrailassiri, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01356 · Cited by 2
“Second, following the methodology of curriculum learning [16], [17], we guide the training of our models by progressively increasing the complexity of the training environment with the agent’s performance.”
MethodBoosting the Performance of Semi-Supervised Learning with Unsupervised ClusteringPreprint
Lerner, B., Shiran, G., & Weinshall, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00504 · Cited by 6
“From a broader perspective, our approach can take advantage of curriculum learning [1], as one might look for”
BackgroundLearning by Passing Tests, with Application to Neural Architecture SearchPreprint
Du, X., Xie, P., & Zhang, H. (2020) · doi:10.36227/techrxiv.13300961.v1 · Cited by 1
“Our work is also related to curriculum learning (CL) (Bengio et al., 2009; Kumar et al., 2010; Jiang et al., 2014; Matiisen et al., 2019).”
BackgroundSelf-Supervised Person Detection in 2D Range Data using a Calibrated CameraPreprint
Jia, D. P., Steinweg, M., Hermans, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08890 · Cited by 1
“Curriculum learning methods [38], [26] may help mitigating this problem, and thus relaxing the requirement of 0 0.”
MethodDeep Policy Networks for NPC Behaviors that Adapt to Changing Design Parameters in Roguelike GamesPreprint
Sestini, A., Kuhnle, A., & Bagdanov, A. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03532 · Cited by 12
“We also use curriculum learning (Bengio et al. 2009) with the same phases as the original paper and during training the agents fight against an opponent that always makes random moves .”
BackgroundMethodSimulation-Driven Machine Learning Control of a Forestry Crane Manipulator
Andersson, J. (2020) · Uppsala University Publications (Uppsala University) · Cited by 1
Whispered and Lombard Neural Speech SynthesisPreprint
Hu, Q., Bleisch, T., Petkov, P. N., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05313 · Cited by 1
“Similarly, in low-resource environments, the pre-training/finetuning model [25, 26] is utilized to adapt to each style.”
BackgroundPhases of learning dynamics in artificial neural networks: with or without mislabeled dataPreprint
Feng, Y., & Tu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06509 · Cited by 1
“It would be interesting to use this general framework to address other fundamental questions in machine learning such as generalization [24–26] in particular the mechanism for the double descent behavior in learning as described above; the relation between task complexity and network architecture; information flow in…”
MethodThe Intersection of Planning and Learning
Moerland, T. M. (2021) · Research Repository (Delft University of Technology) · doi:10.4233/uuid:5437884e-0078-4b36-b2c7-c6edfea3b418 · Cited by 3
TraND: Transferable Neighborhood Discovery for Unsupervised Cross-domain Gait RecognitionPreprint
Zheng, J., Liu, X., Yan, C. C., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04621 · Cited by 1
“Inspired by curriculum learning [1], we refine the backbone CNN in a from-easy-to-hard manner, by which we train the backbone in R rounds.”
Large-Scale Meta-Learning with Continual Trajectory ShiftingPreprint
Shin, J., Lee, H. B., Gong, B., & Hwang, S. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07215 · Cited by 12
“Curriculum learning (Bengio et al., 2009) is a learning strategy where we present training examples from easy to more difficult ones, thereby sequentially controlling the complexity of the loss landscape.”
Luo, H., Zhou, F., Liao, G., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01698 · Cited by 1
“According to curriculum learning strategy [49], we regard the training in the first few iterations as easy tasks by setting smaller loss weight ρ j for smaller j .”
Statistical Measures For Defining Curriculum Scoring FunctionPreprint
Sadasivan, V. S., & Dasgupta, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00147 · Cited by 2
“The notation of difficulty scoring of examples has not really been formalized and various heuristics have been tried out: Bengio et al. (2009) uses manually crafted scores, self-paced learning (SPL) (Kumar et al., 2010) uses the loss values with respect to the learner’s current parameters, and CL by…”
BackgroundInfluentialDistilling Knowledge via Intermediate Classifier Heads.Preprint
Asadian, A., & Salehi‐Abari, A. (2021) · arXiv (Cornell University) · Cited by 1
Medical Image Segmentation with Limited Supervision: A Review of Deep Network ModelsPreprint
Peng, J., & Wang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00429 · Cited by 1
“For example, it is still unclear how to automatically design an adaptive curriculum for the given segmentation task instead of using a predefined curriculum [131] and when will curriculum-like strategies, especially data curriculum, benefit the deep model”
BackgroundA Deep Reinforcement Learning based approach for movement training of neuro-musculoskeletal systemsPreprint
Iyengar, R. S., Mallampalli, K., & Raghavan, M. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.03.28.437396 · Cited by 1
“This technique, where a machine learning algorithm is trained with tasks whose complexity is gradually increasing is known as Curriculum Learning ( [61]).”
Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised LearningPreprint
Liu, L., & Tan, R. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.05657 · Cited by 1
“Our method adapts a progressive learning strategy [14] to gradually learn from certain targets to less certain targets in the consistency regularization.”
BackgroundMethodIncompatibility Clustering as a Defense Against Backdoor Poisoning AttacksPreprint
Jin, C., Sun, M., & Rinard, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03692 · Cited by 6
“, 2010) is a type of curriculum learning (CL) (Bengio et al., 2009) that dynamically creates a curriculum by including samples of increasing difficulty based on the losses of the partially trained model.”
BackgroundData Augmentation for Text Generation Without Any Augmented DataPreprint
Bi, W., Li, H., & Huang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13650 · Cited by 1
“The sample weights are often carefully defined (Freund and Schapire, 1997; Bengio et al., 2009) or learnt by another network (Jiang et al.”
BackgroundTraining With Data Dependent Dynamic Learning RatesPreprint
Saxena, S., Vyas, N., & DeCoste, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13464 · Cited by 4
“In the same spirit, our work also has connections to the field of curriculum learning [4, 9, 36] and self-paced learning [7, 22, 29, 37].”
BackgroundProgressive Stage-wise Learning for Unsupervised Feature Representation EnhancementPreprint
Li, Z., Liu, C., Yuille, A., Ni, B., Zhang, W., & Gao, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05554 · Cited by 1
“As curriculum learning [5] suggests, when dealing with a complex learning target, learning things progressively can be very useful.”
Backgroundpix2rule: End-to-end Neuro-symbolic Rule LearningPreprint
Cingillioglu, N., & Russo, A. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07487 · Cited by 13
“This might suggest that a form a curriculum learning [11] is bene cial to learn and reason with 2 objects, then 3 and nally 4 objects.”
BackgroundTowards Scheduling Federated Deep Learning using Meta-Gradients for Inter-Hospital LearningPreprint
El-Bouri, R., Zhu, T., & Clifton, D. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01707 · Cited by 2
Pattern and Function Transfer for Learning Classifier Systems
Nguyễn, T. T. (2021) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.14411120.v1 · Cited by 1
Improving Reverberant Speech Separation with Multi-stage Training and Curriculum LearningPreprint
Aralikatti, R. C., Ratnarajah, A. J., Tang, Z., & Manocha, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09177 · Cited by 2
“In our approach, we make use of curriculum learning [11].”
MethodGoal-constrained Sparse Reinforcement Learning for End-to-End DrivingPreprint
Agarwal, P., de Beaucorps, P., & de Charette, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.09189 · Cited by 5
“To fight this, we use curriculum learning [2] where complexity is the distance the agent needs to drive.”
BackgroundMethodTolerance-Guided Policy Learning for Adaptable and Transferrable Delicate Industrial InsertionPreprint
Niu, B., Wang, C., & Liu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02303 · Cited by 5
“It can be viewed as a form of curriculum learning [30].”
Iwamoto, S., Raytchev, B., Tamaki, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.11693 · Cited by 1
“In stage 2 of the proposed method, we utilize Curriculum Learning [2] to leverage the information in the Uncertainty Map about the uncertainty of the model to improve its reliability (methods for evaluating the reliability of a model are explained in Section 3).”
BackgroundMethodParamCrop: Parametric Cubic Cropping for Video Contrastive LearningPreprint
Qing, Z., Huang, Z., Zhang, S., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10501 · Cited by 1
“Inspired by curriculum learning [22], we slightly increase the difficulty in the later stage of contrastive training by increasing the central distance between two cropped views (Fig.”
BackgroundImproved Techniques for Quantizing Deep Networks with Adaptive Bit-WidthsPreprint
Sun, X., Panda, R., Chen, C.-F., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01435 · Cited by 1
“Therefore, we linearly increase the probability to execute the students with respect to the layer depth: Following curriculum learning [4], we set the initial value for p 1 and gradually increase it to 1 during training, i.e. it auto-regresses towards training the network with all student blocks at…”
BackgroundPrefix-to-SQL: Text-to-SQL Generation from Incomplete User QuestionsPreprint
Deng, N., Chang, S., Shi, P., Yu, T., & Zhang, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13066 · Cited by 7
“Based on our discovery of the relationship between RECALL and the length to complete, we propose the use of curriculum learning (Bengio et al., 2009) to improve the T5 performance.”
MethodMemBERT: Injecting Unstructured Knowledge into BERTPreprint
Ruggeri, F., Lippi, M., & Torroni, P. (2021) · arXiv (Cornell University) · Cited by 1
Plane-Wave Ultrasound Beamforming: A Deep Learning ApproachPreprint
Goudarzi, S., & Rivaz, H. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13119 · Cited by 1
“Otherwise, the training procedure does not converge, as theoretically outlined in curriculum learning [32].”
Background
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.