Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 4,201–4,300 of 5,240
Scaffolding Haptic Attention with Controller Gating
Moringen, A., Fleer, S., & Ritter, H. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30487-4_51
On educating machinesPreprint
Leu, G., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06017
“Optimisation of the learning cost has been addressed under the concept of curriculum learning , starting with the pioneering work of Bengio et al. [2].”
BackgroundMethodInfluentialData Ordering Patterns for Neural Machine Translation: An Empirical StudyPreprint
Garg, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10642
“Curriculum learning (Bengio et al., 2009) proposes that choosing an ordering of the training samples from easier to learn examples to harder to learn examples can help train better models and achieve faster convergence.”
BackgroundInteraction-Aware Multi-Agent Reinforcement Learning for Mobile Agents with Individual GoalsPreprint
Mohseni-Kabir, A., Isele, D., & Fujimura, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12925
“As an alternative, we consider learning via a curriculum [9].”
MethodLearning from Mistakes: Weakly Supervised Learning of Rocks
Szoke-Sieswerda, J., Cross, M., Kampen, L. V., & McIsaac, K. (2019) · Conference proceedings - Canadian Conference on Electrical and Computer Engineering · doi:10.1109/ccece.2019.8861784
“Wang et al [15] used curriculum learning [24] to increase the precision of object detectors trained under the WSL paradigm.”
MethodSelf-Supervised Learning of Class Embeddings from Video
Wiles, O., Koepke, A. S., & Zisserman, A. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00364 · Cited by 25
“This can be regarded as a form of curriculum learning [3] where the decoder is progressively expanded in levels which increase the resolution of the generated image.”
BackgroundImber, J., Tings, B., & Velotto, D. (2019) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss.2019.8898436
“Identification of the relevant image features is shared, and in this way the network can benefit from a simple form of curriculum learning [10].”
BackgroundMask-dependent Phase Estimation for Monaural Speaker SeparationPreprint
Ni, Z., & Mandel, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02746
“Inspired by [19], we also apply a curriculum training strategy [20] that trains the network using α = 0.”
MethodCarpe Diem, Seize the Samples Uncertain "At the Moment" for Adaptive Batch SelectionPreprint
Song, H., Kim, M., Kim, S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08050
“[1] first took easy samples and then gradually increased the difficulty of samples using heuristic rules.”
BackgroundBiological Blueprints for Next Generation AI SystemsPreprint
Dean, T., Fan, C., Lewis, F. E., & Sano, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.00421 · Cited by 1
“igure 12, is that end-to-end training with stochastic gradient descent isn’t going to work, and that training will likely require some form of layer-bylayer developmentally-staged curriculum learning [147, 83, 22, 25] and a strategy for holding some weights xed while adjusting other weights to account for new informat…”
BackgroundInfluentialIterative Policy-Space Expansion in Reinforcement LearningPreprint
Lichtenberg, J. M., & Şimşek, Ö. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02532
“In curriculum learning [6, 8], neural networks are trained with progressively more noisy and less relevant training data.”
3D Objectness Estimation via Bottom-up Regret GroupingPreprint
Ye, Z., Yan, H., Xu, L., Zhu, R., & Lu, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02332
“Inspired by [2], we train our grouping predictor in phases and update the corresponding training set dynamically.”
MethodImplementing Inductive bias for different navigation tasks through diverse RNN attrractors
Xu, T. J., & Barak, O. (2020) · International Conference on Learning Representations
Diversity Transfer Network for Few-Shot LearningPreprint
Chen, M., Fang, Y., Wang, X., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.13182
“In addition to the new sample generation scheme, we utilize an effective meta-training curriculum called OAT (Organized Auxiliary task co-Training), inspired by the auxiliary task co-training in TADAM (Oreshkin, Rodríguez López, and Lacoste 2018) and curriculum learning (Bengio et al. 2009).”
BackgroundMethodInfluentialDeep Reinforcement Learning for Complex Manipulation Tasks with Sparse FeedbackPreprint
Manela, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03877 · Cited by 3
“2 Curriculum learning Curriculum learning [31], also known as sequential transfer learning, is an approach where instead of training the model on the desired task from the beginning, we train the agent first an a source task, which is a simplified version of the task, and its complexity is gradually increasing.”
BackgroundImplementing Inductive bias for different navigation tasks through diverse RNN attractorsPreprint
Xu, T. J., & Barak, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02496 · Cited by 2
“Thus it is interesting to explore whether this approach could be used to accelerate learning in other domains, similar to curriculum learning (Bengio et al., 2009).”
BackgroundAn End-to-End Framework for Unsupervised Pose Estimation of Occluded PedestriansPreprint
Das, S. K., Kishore, P. S. R., & Bhattacharya, U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06429
“We employ Curriculum Learning [18] in the Mask and Predict strategy by gradually increasing the masking percentage with training.”
MethodSafety-Oriented Stability Biases for Continual Learning
Gaurav, A. (2020) · UWSpace (University of Waterloo) · Cited by 1
Learning to Inpaint by Progressively Growing the Mask RegionsPreprint
Hedjazi, M. A., & Genç, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09280
“Another attempt to stabilize the training of GANs is to employ a curriculum learning (CL) approach [21].”
BackgroundBarbu, A., & Zhu, S. (2020) · Springer eBooks · doi:10.1007/978-981-13-2971-5_11
Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue GenerationPreprint
Cai, H., Chen, H., Zhang, C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.00639
“Similar to other curriculum learning framework (Bengio et al. 2009; Tsvetkov et al. 2016), the learning status consists of several features, including the passed mini-batch number, the average historical training loss, the loss value on the training data, the margin value of predicted probabilities and the last valida…”
Coaching: accelerating reinforcement learning through human-assisted approach
Suppakun, N., & Maneewarn, T. (2020) · Progress in Artificial Intelligence · doi:10.1007/s13748-020-00204-4
“Bengio et al. [21] proposed curriculum learning in which the learning process has started from an easy task and then move onto more difficult tasks.”
BackgroundProgressive Growing of Neural ODEsPreprint
Ayyubi, H. A., Yao, Y., & Divakaran, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03695
“Bengio et al. (2009) extended this idea and showed it’s effectiveness in language modeling and geometric shape recognition task. More recently, this idea of progressive learning has been explored in many contexts. Zaremba & Sutskever (2014) use this approach for evaluating short computer programs. Matiisen et al. (201…”
BackgroundMethodInfluentialA Machine Learning Framework for Data Ingestion in Document ImagesPreprint
Fu, H., Bai, Y., Li, Z., Shen, J., & Sun, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.00838 · Cited by 1
“To address this challenge, we propose a warm-up training strategy to pre-train the model on synthesized data, inspired by [2].”
MethodPractical Annotation Strategies for Question Answering DatasetsPreprint
Kratzwald, B., Yue, X., Sun, H., & Feuerriegel, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03235 · Cited by 1
“It is different from curriculum learning (Bengio et al., 2009) in which data from a labeled dataset is subject to non-uniform sampling, whereas our task is to infer a subset from an unlabeled pool of samples.”
BackgroundLearning from Web Data with Self-Organizing Memory ModulePreprint
Tu, Y., Niu, L., Chen, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.12028
Alleviate Cross-chunk Permutation through Chunk-level Speaker Embedding for Blind Speech Separation
Gu, R., Peng, J., Zou, Y., et al. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023256
“In this study, the TFE is trained from scratch using 100-frame chunks, and then fine-tuned with 400-frame chunks for curriculum learning following the method suggested in [15].”
MethodDynamic Composition of Functions for Modular Learning
Rosenbaum, C. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/15649011
“Another strategy that can be applied is curriculum learning [Bengio et al., 2009], as is done by [Chang et al.”
BackgroundTrying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum LearningPreprint
Portelas, R., Hofmann, K., & Oudeyer, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03168 · Cited by 3
“, 2007) and supervised learning (Elman, 1993; Bengio et al., 2009), from which the Deep RL community borrowed the term Curriculum Learning (CL).”
BackgroundYouMakeup VQA Challenge: Towards Fine-grained Action Understanding in Domain-Specific VideosPreprint
Chen, S., Wang, W., Ruan, L., Yao, L., & Jin, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05573 · Cited by 2
“We utilize a curriculum learning strategy [2] to train the model from easy samples to hard samples.”
MethodDivergent Search for Few-Shot Image ClassificationPreprint
Tan, J., & Kainz, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07903 · Cited by 1
“This may unintentionally regularize the difficulty of divergent tasks and form a sort of curriculum [1].”
BackgroundMali, A., Ororbia, A. G., Kifer, D., & Giles, C. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07623 · Cited by 16
“o training processes, one was sequential [18, 48], which is widely used in natural language processing related tasks, while the other was based on incremental learning [20, 45] or curriculum learning [3], which is a popular training scheme for tensor RNNs. We compare these two variants in Table 3 and show that sequent…”
BackgroundInfluentialSelf-Paced Deep Reinforcement LearningPreprint
Klink, P., D’Eramo, C., Peters, J., & Pajarinen, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11812 · Cited by 82
“We are specifically interested in the field of Curriculum Learning (Bengio et al., 2009) for RL, which investigates how sequences of tasks can be designed that benefit the learning progress of an agent by allowing it to reuse previously learned behavior.”
BackgroundInfluentialEvolutionary Stochastic Policy DistillationPreprint
Sun, H., Pan, X.-Y., Dai, B., Lin, D., & Zhou, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12909 · Cited by 1
“…works, reward shaping [Ng et al., 1999], hierarchical reinforcement learning [Dietterich, 2000; Barto & Mahadevan, 2003], curriculum learning [Bengio et al., 2009] and learning from demonstrations [Schaal, 1997; Atkeson & Schaal, 1997; Argall et al., 2009; Hester et al., 2018; Nair et al.,…”
BackgroundRelabel the Noise: Joint Extraction of Entities and Relations via Cooperative MultiagentsPreprint
Chen, D., Li, Y., Lei, K., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09930
“In this study, we extend the curriculum learning strategy (Bengio et al., 2009) to our cooperative multia-gents.”
MethodSIBRE: Self Improvement Based REwards for Reinforcement Learning.Preprint
Nath, S., Verma, R., Ray, A., & Khadilkar, H. (2020) · arXiv (Cornell University)
Towards Multimodal Response Generation with Exemplar Augmentation and Curriculum OptimizationPreprint
Lei, Z., Li, Z., Zhang, J., Meng, F., Feng, Y., Yang, Y., Niu, C., & Zhou, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12429
“Curriculum learning is a machine learning strategy, which starts from simple subtasks and then gradually handles harder ones (Bengio et al., 2009).”
BackgroundDeep Reinforcement Learning for High Level Character ControlPreprint
Souza, C., & Velho, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.10391
“Yet, this 12Curriculum learning is an approach of increasing the environment difficulty according with the agent learning which was proposed in (Bengio et al., 2009) exposition need to be done without a steep difficulty variation, or the agent may not be able to learn anything.”
BackgroundGraph Learning with Loss-Guided Training
Buchnik, E., & Cohen, E. (2020) · ACM SIGMOD Joint International Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA) (GRADES-NDA) · doi:10.1145/3398682.3400060
“A prolific research thread proposed methods that accelerate the training or improve accuracy by dynamically modifying the distribution of examples in the course of training [1, 3, 21, 32, 35, 36, 41]: These approaches include Curriculum/selfpaced learning [3], where the selection is altered to mimic human learning: Fi…”
MethodSomethingFinder: Localizing undefined regions using referring expressions
Eum, S., Han, D. K., & Briggs, G. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00198
“We demonstrate how our model benefits from curriculum learning [3].”
BackgroundNNC: Neural-Network Control of Dynamical Systems on GraphsPreprint
Asikis, T., Böttcher, L., & Antulov-Fantulin, N. (2020) · arXiv (Cornell University) · doi:10.3929/ethz-b-000450528
Curriculum DropoutPreprint
Morerio, P., Cavazza, J., Volpi, R., Vidal, R., & Murino, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06229 · Cited by 7
Deep learning analysis of eye fundus images to support medical diagnosis
Charry, P., & Julián, Ó. (2020)
A Survey on Algorithms in Deep Learning
Menon, S. P. (2020) · Advances in computer and electrical engineering book series · doi:10.4018/978-1-7998-2803-7.ch017 · Cited by 1
Variance Based Sample Weighting for Supervised Deep LearningPreprint
Novello, P., Poëtte, G., Lugato, D., & Congedo, P. M. (2021) · arXiv (Cornell University)
Learning Error-Driven Curriculum for Crowd CountingPreprint
Li, W., Cao, Z., Wang, Q., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.09676
“Inspired by the recent success of curriculum learning [21], [22], we tackle data imbalance by learning a curriculum.”
BackgroundMethodChaichanawirote, C., Tokumaru, M., & Charoenseang, S. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-50729-9_22
Reinforcement Learning for Elimination of Reentrant Spiral Waves in Excitable Media
Senter, J. K., Wilson, D. D., & Sadovnik, A. (2020) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc45564.2020.9147623
“These goals can be designed manually given an expertise in the subject area [25], [26] or found automatically [27], [28].”
BackgroundAARSynth: App-Aware Response Synthesis for User Reviews.Preprint
Farooq, U., Siddique, A. B., Jamour, F., et al. (2020) · arXiv (Cornell University)
Deep Reinforcement Learning with Label Embedding Reward for Supervised Image HashingPreprint
Wang, Z., Hong, W., & Yuan, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.03973 · Cited by 1
Mastering Rate based Curriculum LearningPreprint
Willems, L., Lahlou, S., & Bengio, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.06456 · Cited by 10
“An alternative way to improve the sample efficiency is curriculum learning ([8]).”
BackgroundSelf-citationNavigating the Landscape of Multiplayer Games to Probe the Drosophila of AIPreprint
Omidshafiei, S., Tuyls, K., Czarnecki, W. M., Santos, F. C., Rowland, M., Connor, J. T., Hennes, D., Müller, P. F., Pérolat, J., Vylder, B. D., et al. (2020) · arXiv (Cornell University)
Whole page recognition of historical handwritingPreprint
Dolfing, H. J. G. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10634 · Cited by 3
“To speed up convergence, we use a form of curriculum training [53] to bootstrap the page models.”
MethodWeakly Supervised Object Localization Using Self-Paced Pyramid Adversarial Learning
Pan, F.-C., Bian, B., Wang, B., Yang, Y., & Ju, X. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207099
“[26] first initialized the concept of curriculum learning (CL), in which a model is learned by gradually including samples into training from easy to complex.”
BackgroundPopulation Based Training for Data Augmentation and Regularization in Speech RecognitionPreprint
Haziza, D., Rapin, J., & Synnaeve, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03899 · Cited by 1
“Having a schedule – or a curriculum [1] – is common for other hyperparameters like the learning rate [2, 16] or task difficulty [22] but rarely used for data augmentation as the search space for a schedule is even bigger than for a fixed policy.”
BackgroundRevisiting Optical Flow Estimation in 360 VideosPreprint
Bhandari, K., Zong, Z., & Yan, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08045
“Training architectures like this using better strategy to cope with gradually increasing task is an adoption of a popular philosophy, curriculum learning[15].”
Less is More: Data-Efficient Complex Question Answering over Knowledge BasesPreprint
Hua, Y., Li, Y.-F., Qi, G., Wu, W., Zhang, J., & Qi, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15881
“To adaptively control the exploration-exploitation trade-o ff , we employ a curriculum learning [15] scheme to dynamically change the influence of the two reward bonuses, namely the proximity and the novelty.”
MethodDynamic Data Selection for Curriculum Learning via Ability EstimationPreprint
Lalor, J. P., & Yu, H. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00080
“For example, sentence length is often used as a proxy for difficulty in NLP tasks (Bengio et al., 2009; Platanios et al., 2019).”
BackgroundMethodResultInfluentialSuppressing Mislabeled Data via Grouping and Self-AttentionPreprint
Peng, X., Wang, K., Zeng, Z., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15603
“Some others resort to curriculum learning [4] by either designing a step-wise easy-to-hard strategy for training [10] or introducing an extra MentorNet [12] for sample weighting.”
BackgroundCuriosity Based Reinforcement Learning on Robot Manufacturing CellPreprint
Hameed, M. S. A., Khan, M. M., & Schwung, A. (2020) · arXiv (Cornell University) · doi:10.13140/rg.2.2.11814.75847
Understanding Word Embedding Stability Across Languages and Applications
Burdick, L. (2020) · Deep Blue (University of Michigan)
Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervisionPreprint
Zhong, Y., Xie, L., Wang, S., Specia, L., & Miao, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09634
“This training scheme shares a similar spirit with the curriculum learning [2] which presents training examples in a meaningful order, i.e. gradually illustrating more complex ones.”
BackgroundDynamic Curriculum Learning for Low-Resource Neural Machine TranslationPreprint
Chen, X., Hu, B., Jiang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14608
“Curriculum learning (Bengio et al., 2009) is motivated by the learning strategy of biological organisms which orders the training samples in an easy-to-hard manner (Elman, 1993).”
InfluentialConvergence Analysis of Homotopy-SGD for non-convex optimizationPreprint
Gargiani, M., Zanelli, A., Tran-Dinh, Q., Diehl, M., & Hutter, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10298 · Cited by 5
“In this regard, the proposed method, despite being new in its general formulation and analysis, finds many interesting similarities and connections with existing heuristics in the machine learning literature, e.g. (Bengio et al., 2009; Hinton et al., 2012; Sutskever et al., 2013a).”
BackgroundMethodAda-Segment: Automated Multi-loss Adaptation for Panoptic SegmentationPreprint
Zhang, G., Gao, Y., Xu, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03603
“Multi-objective Learning.”
BackgroundSimuGAN: Unsupervised forward modeling and optimal design of a LIDAR CameraPreprint
Diamant, N., Mund, T., Menashe, O., Zabatani, A., & Bronstein, A. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08951
“Another enhancement to the LIDAR performance was done by Bergman et al. [2020], involving deep learning methods to optimize the depth estimation of dense 3D point clouds, which on its own is limited when sampling with low rates. Different uses of signal generation using GANs were previously done: Nik Aznan et al. [201…”
MethodInfluentialRobust Collaborative Learning with Noisy LabelsPreprint
Sun, M., Xing, J., Chen, B., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13670
“Inspired by the fact that humans learn better when trained with a curriculum-like strategy, [21] first proposed curriculum learning.”
BackgroundA Comprehensive Study on Optimization Strategies for Gradient Descent In Deep LearningPreprint
Yadav, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02397 · Cited by 7
“[7] as a concept for attaining convergence faster and finding a local minima for non- convex functions.”
BackgroundCan Selfless Learning improve accuracy of a single classification task?
Roy, S. D., & Sau, B. B. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00409
“In curriculum learning, we rank samples based on their difficulty and set up a pacing strategy to serve the classifier with progressively harder samples [2].”
Learning to Reweight with Deep InteractionsPreprint
Yang, F., Xia, Y., Wu, L., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04649
“Assigning weights to different data points have been widely investigated in literature, where the weights can be either continuous [8, 14] or binary [6, 4].”
BackgroundA Taylor Based Sampling Scheme for Machine Learning in Computational PhysicsPreprint
Novello, P., Poëtte, G., Lugato, D., & Congedo, P. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11105
“Though non specific to Deep Learning, we applied this method to the approximation of the solution of a physical ODE system by a Deep Neural Network and increased its accuracy for a same model architecture.”
BackgroundRethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
Ganesh, M. R., & Corso, J. J. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.142 · Cited by 10
“There are two alternative lines of work which offer strategies to guide deep networks to better solutions than batch learning: Curriculum Learning [3, 12, 15] and Label Smoothing [9, 38].”
BackgroundMethodInfluentialChang, E., Yeh, H.-S., & Demberg, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03554
Sound Event Detection Based on Curriculum Learning Considering Learning Difficulty of EventsPreprint
Tonami, N., Imoto, K., Okamoto, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05288
“To utilize the difference in the difficulty of training between the sound events, we employ curriculum learning [17].”
MethodAdaptation Across Extreme Variations using Unlabeled Bridges
Dai, S., Sohn, K., Tsai, Y., Duke, L. C., & Chandraker, M. K. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.51
“In contrast, our method allows stable performance earlier in the training, which implies that knowing the curriculum [4] ( i . e .”
ResultTwo-perspective Biomedical Named Entity Recognition with Weakly Labeled Data Correction
Zhou, H., Liu, Z., Lang, C., Xu, Y., & Du, L. (2020) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313320
“Bengio et al. [19] demonstrate that curriculum learning train models better than random.”
BackgroundSearching for Needles in the Cosmic Haystack
Devine, T. R. (2020) · The Research Repository @ WVU (West Virginia University) · doi:10.33915/etd.7727
“They described self-paced training in general as an evolution of curriculum learning, where a model is learned by gradually including more complex samples during training to increase the entropy of training samples [8].”
BackgroundVijjini, A. R., Anuranjana, K., & Mamidi, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.09990
“(Bengio et al., 2009) show that Curriculum Learning (CL)”
BackgroundInfluentialLearning to Represent and Predict Sets with Deep Neural NetworksPreprint
Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04957
Walk2Map: Extracting Floor Plans from Indoor Walk Trajectories
Mura, C., Pajarola, R., Schindler, K., et al. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.2103.00262
A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement LearningPreprint
Klink, P., Abdulsamad, H., Belousov, B., D’Eramo, C., Peters, J., & Pajarinen, J. (2021) · Aaltodoc (Aalto University) · doi:10.48550/arxiv.2102.13176 · Cited by 40
“This idea is at the heart of curriculum learning (CL), initially investigated by Bengio et al. (2009) for supervised learning problems.”
BackgroundVISUAL AND SEMANTIC KNOWLEDGE TRANSFER FOR NOVEL TASKS
Ye, M. (2019) · TUScholarShare (Temple University)
Joint Weakly Supervised AT and AED Using Deep Feature Distillation and Adaptive Focal Loss.Preprint
Liang, Y., Long, Y., Li, Y., et al. (2021) · arXiv (Cornell University)
Learning Policies for Multilingual Training of Neural Machine Translation SystemsPreprint
Kumar, G., Koehn, P., & Khudanpur, S. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06964 · Cited by 3
“Curriculum learning (Bengio et al., 2009; Elman, 1993; Rohde and Plaut, 1994) hypothesizes that presenting training samples in a meaningful order to machine learners during training may help improve model quality and convergence speed.”
BackgroundSupport-Target Protocol for Meta-Learning.Preprint
Su, L., Ye, H.-J., & Zhan, D. (2021) · arXiv (Cornell University)
New Deep Neural Networks for Unsupervised Feature Learning on Graph Data
Gao, H. (2020) · D-Scholarship@Pitt (University of Pittsburgh)
Self-Paced Uncertainty Estimation for One-shot Person Re-IdentificationPreprint
Zhang, Y., Ma, B., Liu, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09152
Weakly Supervised Object Localization and Detection: A SurveyPreprint
Zhang, D., Han, J., Cheng, G., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07918
“By integrating the curriculum learning [5] with the self-paced learning [84], the established learning framework provides a more theo-retical-sounded way to improve the learning robustness.”
Perceptually-Informed No-Reference Image Harmonisation
Dolhasz, A., Harvey, C., & Williams, I. (2022) · Communications in computer and information science · doi:10.1007/978-3-030-94893-1_18
Ferrari, C., Berretti, S., & Bimbo, A. D. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412994
“Moriero et al. [20] drawn connections between the original dropout framework and the regularization theory [3], and curriculum learning [2].”
BackgroundEfficient Hyperparameter Optimization for Physics-based Character AnimationPreprint
Yang, Z., & Yin, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.12365
“Curriculum learning (CL) is a learning method where task difficulty gradually increases during training[Bengio et al. 2009].”
BackgroundMethodStreaming end-to-end speech recognition with jointly trained neural feature enhancementPreprint
Kim, C., Garg, A., Gowda, D., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01254
“Inspired by Curriculum Learning [1], we propose two training strategies in this work: Gradual Application of Enhanced Feature (GAEF) and Gradual Reduction of Enhancement Loss (GREL).”
BackgroundMethodInfluentialLearning multiple concepts in description logic through three perspectives
Melo, R., Revoredo, K. C., & Paes, A. (2021) · The Knowledge Engineering Review · doi:10.1017/s0269888921000059
Fixed $β$-VAE Encoding for Curious Exploration in Complex 3D EnvironmentsPreprint
Lehuger, A., & Crosby, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08568 · Cited by 3
“Curriculum Learning (CL) is a general approach to optimising the learning process of a machine learning algorithm (Bengio et al., 2009).”
BackgroundSelf-Guided Curriculum Learning for Neural Machine TranslationPreprint
Zhou, L., Ding, L., Duh, K., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04475
CARLS: Cross-platform Asynchronous Representation Learning SystemPreprint
Lu, C.-T., Zeng, Y., Juan, D.-C., Fan, Y., Li, Z., Dlabal, J., Chen, Y., Gopalan, A., Heydon, A., Ferng, C.-S., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12849
“Curriculum learning [4] refers to a learning paradigm where the challenge of a learning task gradually increases over time for improving the model performance.”
BackgroundDistribution Matching for Machine TeachingPreprint
Cao, X., & Tsang, I. W.-H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13809
“For instance, in curriculum learning [15], [30], a human can intelligently annotate or recognize examples when the teaching examples are not randomly presented but organized in a meaningful order.”
BackgroundInterpretable UAV Collision Avoidance using Deep Reinforcement LearningPreprint
Thomas, D.-G., Olshanskyi, D., Krueger, K., Wongpiromsarn, T., & Jannesari, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12254 · Cited by 13
“In addition, they implemented curriculum learning to augment their algorithm, wherein training starts on a single task whose complexity is increased as learning progresses [26].”
BackgroundTowards Enabling Meta-Learning from Target ModelsPreprint
Lu, S., Ye, H.-J., Gan, L., & Zhan, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.03736 · Cited by 3
“Thirdly, we can treat the target model as a teacher and incorporate a teacher-student framework like knowledge distillation [7] and curriculum learning [1] in metalearning.”
Method
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.