Curriculum learning
The DOI matches its Crossref record; 35 of the 40 references it cites were found in a registry or an index.
Abstract
Humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones. Here, we formalize such training strategies in the context of machine learning, and call them "curriculum learning". In the context of recent research studying the difficulty of training in the presence of non-convex training criteria (for deep deterministic and stochastic neural networks), we explore curriculum learning in various set-ups. The experiments show that significant improvements in generalization can be achieved. We hypothesize that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and, in the case of non-convex criteria, on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, the lack of acoustical signals could even account for spatial disorientation 91,92.”
From The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence (2021) · cites this paper as [92]
In the citing paper
“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”
From Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory (2025) · cites this paper as Bengio et al., 2009
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 6×
“In Collobert and Weston (2008), the main objective is to learn an embedding for the words as a side-e.ect of learning to compute this score.”
BackgroundMethodInfluentialSelf-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 8×
“The idea of training a learning machine with a curriculum can be traced back at least to Elman (1993).”
BackgroundMethodInfluentialRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Research works printed but not linked: 3
- 10.5555/89603 · doi:10.5555/89603
- 10.5555/866740 · doi:10.5555/866740
- 10.5555/525571 · doi:10.5555/525571
Cited by
Items 601–700 of 5,239
Real-Time Human Motion Capture with Multiple Depth Cameras
Shafaei, A., & Little, J. J. (2016) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2016.25 · Cited by 65
“This strategy is hypothesized to improve the convergence speed and the quality of the final local minima [6].”
BackgroundMethodInfluentialGao, H., & Huang, H. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220041 · Cited by 62
“According to the learning process of human beings, we should start with easy samples when learning a newmodel and then learn difficult samples gradually [2, 13].”
BackgroundTowards Federated Learning against Noisy Labels via Local Self-Regularization
Jiang, X., Sun, S., Wang, Y., & Liu, M. (2022) · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557475 · Cited by 65
“MentorNet [23] introduces curriculum learning [3] into sample reweighting by pre-training an extra network, which is utilized to set higher weights for correct-labeled inputs during the training process.”
BackgroundExploring the Limitations of Behavior Cloning for Autonomous Driving
Codevilla, F., Santana, E., Lopez, A., & Gaidon, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00942 · Cited by 509
“data assumption towards curriculum learning [7] for sensorimotor control [46, 4].”
BackgroundHistory for Visual Dialog: Do we really need it?
Agarwal, S., Bui, T., Lee, J.-Y., Konstas, I., & Rieser, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.728 · Cited by 58
“• We show that curriculum fine-tuning (Bengio et al., 2009) on annotations of semantically equivalent answers further improves results.”
BackgroundInfluentialBabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
Zhu, W., Hu, H., Chen, J., Deng, Z., Jain, V., Ie, E., & Sha, F. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.229 · Cited by 59
“Curriculum Learning Since proposed in (Bengio et al., 2009), curriculum learning was successfully used in a range of tasks: training robots for goal reaching (Florensa et al., 2017), visual question answering (Mao et al., 2019), image generation (Karras et al., 2018).”
BackgroundMethodAdaptive reverse graph learning for robust subspace learning
Yuan, C., Zhong, Z., Lei, C., et al. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102733 · Cited by 64
Efficient Federated Learning for Modern NLP
Cai, D., Wu, Y., Wang, S., Lin, F. X., & Xu, M. (2023) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3570361.3592505 · Cited by 123
“Such upgrading mechanism is inspired by curriculum learning [11], a learning strategy that trains a model beginning from easier data samples to harder ones.”
BackgroundSPARC: Self-Paced Network Representation for Few-Shot Rare Category Characterization
Zhou, D., He, J., Yang, H., & Fan, W. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219968 · Cited by 73
Deep Learning of Representations
Bengio, Y., & Courville, A. C. (2013) · Intelligent systems reference library · doi:10.1007/978-3-642-36657-4_1 · Cited by 96
Self-citationWhat Objective Does Self-paced Learning Indeed Optimize?Preprint
Meng, D., Zhao, Q., & Jiang, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06049 · Cited by 64
“Since being raised, curriculum learning (CL) [1] and self-paced learning (SPL) [2] have been attracting increasing attention in machine learning, computer vision and multimedia analysis circles.”
BackgroundSelf-paced and auto-weighted multi-view clustering
Ren, Y., Huang, S., Zhao, P., Han, M., & Xu, Z. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.11.104 · Cited by 65
Csurka, G. (2020) · 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc51798.2020.00013 · Cited by 52
“To minimise the impact of noisy pseudo-labels during alignment, curriculum learning-based [108] approaches have been explored.”
BackgroundA literature survey of MR-based brain tumor segmentation with missing modalities
Zhou, T., Ruan, S., & Hu, H. (2022) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102167 · Cited by 77
Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances
Moser, B. B., Raue, F., Frolov, S., Palacio, S., Hees, J., & Dengel, A. R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243794 · Cited by 86
“Curriculum learning follows the idea of training a model under easy conditions and gradually involving more complexity [84], i.e., additional scaling sizes.”
BackgroundActive Task Selection for Lifelong Machine Learning
Ruvolo, P., & Eaton, E. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8684 · Cited by 83
Sun, Y., Cheng, J., Zhang, G., & Xu, H. (2019) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-019-01004-2 · Cited by 68
“by Bengio [1]; it was originally a technique applied to deep learning training.”
MethodInteraction-aware Decision Making with Adaptive Strategies under Merging Scenarios
Hu, Y., Nakhaei, A., Tomizuka, M., & Fujimura, K. (2019) · 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968478 · Cited by 65
“Therefore, we introduced a two-stage training process which utilized the curriculum learning [17] strategy.”
MethodTowards Practical Multi-Object Manipulation using Relational Reinforcement Learning
Li, R., Jabri, A., Darrell, T., & Agrawal, P. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197468 · Cited by 78
“Prior work has shown that ordering tasks by heuristic measures of difficulty can be effective [45], [46].”
BackgroundSimplify and Robustify Negative Sampling for Implicit Collaborative FilteringPreprint
Ding, J., Quan, Y., Yao, Q., Li, Y., & Jin, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.03376 · Cited by 146
“Typical learning approaches include curriculum learning [2], self-pace learning [24] and instance re-weighting [21, 31, 35].”
Learning to collaborate from simulation for robot-assisted dressing
Clegg, A., Erickson, Z., Grady, P., Turk, G., Kemp, C. C., & Liu, C. K. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2972852 · Cited by 62
“to this problem and have been successfully applied to deep reinforcement learning of motor skills tasks in the past [26], [27].”
BackgroundRecurrent Neural Networks for Semantic Instance Segmentation
Salvador, A., Bellver, M., Campos, V., Baradad, M., Marques, F., Torres, J., & Giro-i-Nieto, X. (2017) · arXiv · doi:10.48550/arxiv.1712.00617 · Cited by 62
“learning [7] to guide the optimization process, where we begin optimizing the model to predict only two objects and increase this value by one once the validation loss plateaus.”
MethodStagger: an Open-Source Part of Speech Tagger for Swedish
Östling, R. (2013) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.1331 · Cited by 76
“The test set (10 les, 2% of the total corpus) was chosen to be identical to 3Turian et al. (2010) and Bengio et al. (2009) use f = n, that is, they replace the last word in each n-gram.”
BackgroundMethodAggregating Deep Pyramidal Representations for Person Re-Identification
Martinel, N., Foresti, G. L., & Micheloni, C. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00196 · Cited by 64
“Thus, we adopt a curriculum learning strategy [4].”
MethodWei, J., Suriawinata, A. A., Ren, B., Liu, X., Lisovsky, M., Vaickus, L. J., Brown, C. L., Baker, M. L., Nasir-Moin, M., Tomita, N., et al. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00252 · Cited by 80
“One of the earliest works demonstrating the benefit of curriculum learning [1] posits that learning occurs better when examples are not randomly presented but instead organized in a meaningful order that gradually shows more concepts and complexity.”
BackgroundInfluentialLearning-Based End-to-End Path Planning for Lunar Rovers with Safety Constraints
Yu, X., Wang, P., & Zhang, Z. (2021) · Sensors · doi:10.3390/s21030796 · Cited by 83
“In addition, in order to improve the adaptability of the planner to different terrains and different scales of movement planning on the lunar surface, this article adopts the idea of curriculum learning and uses different terrain features and different sizes of lunar environment training planning networks [30].”
MethodTowards Multi-Modal Perception-Based Navigation: A Deep Reinforcement Learning Method
Huang, X., Deng, H., Zhang, W., Song, R., & Li, Y. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3064461 · Cited by 80
“Second, we adopt a curriculum learning paradigm [13] to accelerate the training.”
MethodTrigger is Not Sufficient: Exploiting Frame-aware Knowledge for Implicit Event Argument Extraction
Wei, K., Sun, X., Zhang, Z., Zhang, J., Guo, Z. M., & Jin, L. (2021) · doi:10.18653/v1/2021.acl-long.360 · Cited by 121
“Curriculum Learning is a learning strategy firstly proposed by Bengio et al. (2009) that trains a neural network better through increasing data complexity of training data.”
BackgroundMethodCDAda: A Curriculum Domain Adaptation for Nighttime Semantic Segmentation
Xu, Q., Ma, Y., Wu, J. B., Long, C., & Huang, X. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00331 · Cited by 79
“from easy to complex, which has been proved to promote the optimization of non-convex problems [2].”
BackgroundDeep Reinforcement Learning-based Image Captioning with Embedding RewardPreprint
Ren, Z., Wang, X., Zhang, N., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03899 · Cited by 62
“To handle this problem, we follow [35] to apply curriculum learning [1] to train our actor-critic model.”
MethodVariance Reduction for Reinforcement Learning in Input-Driven Environments
Mao, H., Venkatakrishnan, S. B., Schwarzkopf, M., & Alizadeh, M. (2018) · arXiv · doi:10.48550/arxiv.1807.02264 · Cited by 62
“With structural knowledge of complex environments, reward shaping [38] and curriculum learning [2, 23, 61] help learn a high quality control policy.”
BackgroundPOET: open-ended coevolution of environments and their optimized solutions
Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019) · Proceedings of the Genetic and Evolutionary Computation Conference · doi:10.1145/3321707.3321799 · Cited by 61
Unsupervised Text Normalization Using Distributed Representations of Words and Phrases
Sridhar, V. K. R. (2015) · Workshop on Vector Space Modeling for Natural Language Processing (VSM-NLP) · doi:10.3115/v1/w15-1502 · Cited by 60
“While they are typically induced in the context of a deep neural network framework for a given task (Bengio et al., 2003; Collobert and Weston, 2008; Bengio et al., 2009; Turian et al., 2010; Mikolov et al., 2010), recent work in (Mikolov et al., 2013) has also shown that they can also be induced…”
BackgroundArtificial Intelligence and the Common Sense of Animals
Shanahan, M., Crosby, M., Beyret, B., & Cheke, L. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.09.002 · Cited by 58
“confronted with more complex ones [74,75].”
21 000 birds in 4.5 h: efficient large‐scale seabird detection with machine learning
Kellenberger, B., Veen, T., Folmer, E. O., & Tuia, D. (2021) · Remote Sensing in Ecology and Conservation · doi:10.1002/rse2.200 · Cited by 73
“To mitigate this effect, we resorted to curriculum learning (Bengio et al., 2009), which has been shown to be bene-ficial for aerial wildlife detection tasks with deep learning (Kellenberger et al., 2018).”
MethodRobust Physical-World Attacks on Face Recognition
Zheng, X., Fan, Y., Wu, B., Zhang, Y., Wang, J., & Pan, S. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109009 · Cited by 78
“CAA adheres to the principles of curriculum learning (Bengio et al. 2009), which has shown the benefits in obtaining better local minima and superior generalization for the non-convex optimization.”
Drivable Volumetric Avatars using Texel-Aligned Features
Remelli, E., Bagautdinov, T., Saito, S., et al. (2022) · doi:10.1145/3528233.3530740 · Cited by 61
“Intuitively, this helps our model pick up useful signals by gradually shifting from easy samples to harder ones in the spirit of curriculum learning [Bengio et al. 2009].”
Training Deep Models Faster with Robust, Approximate Importance Sampling
Johnson, T. B., & Guestrin, C. (2018) · neural information processing systems · Cited by 171
Distributed Learning for Vehicle Routing Decision in Software Defined Internet of Vehicles
Lin, K., Li, C., Li, Y., Savaglio, C., & Fortino, G. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3023958 · Cited by 65
“In order to facilitate vehicle collaboration, course-based learning method is combined to achieve collaborative distributed reinforcement learning [37].”
MethodBridging the Ultimate Semantic Gap: A Semantic Search Engine for Internet Videos
Jiang, L., Yu, S.-I., Meng, D., Mitamura, T., & Hauptmann, A. G. (2015) · International Conference on Multimedia Retrieval · doi:10.1145/2671188.2749399 · Cited by 61
“Self-Paced Reranking (SPaR): This method is inspired by the cognitive and learning process of humans and animals that gradually learning from easy to more complex samples [2, 22, 17, 46].”
MethodNo Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures
Ahuja, C., Lee, D. W., Ishii, R., & Morency, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.170 · Cited by 94
“, 2015; Bengio et al., 2009; Matiisen et al., 2019). Byrd and Lipton (2018) observe that sub-sampling from unbalanced categorical classes demonstrates a significant effect on the network’s predictions.”
BackgroundHavenstrøm, S. T., Rasheed, A., & San, O. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.566037 · Cited by 70
“• Exploring the potential of curriculum learning (Bengio et al., 2009) in the context of training AUVs.”
BackgroundDeepWalk: Omnidirectional Bipedal Gait by Deep Reinforcement Learning
Rodríguez, D., & Behnke, S. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561717 · Cited by 83
“We formulate the problem of learning a control policy for omnidirectional walking as a curriculum learning problem [20].”
MethodSemantic Image Segmentation: Two Decades of Research
Csurka, G., Volpi, R., & Chidlovskii, B. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000095 · Cited by 109
Self-paced dictionary learning for image classification
Tang, Y., Yang, Y.-B., & Gao, Y. (2012) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2393347.2396324 · Cited by 56
“Recent work in [2] presented curriculum learning for training with non-convex objectives and advocated a well-defined training order which used easy samples first and gradually expanded to more complex ones.”
BackgroundDeep Local Trajectory Replanning and Control for Robot Navigation
Pokle, A., Martín-Martín, R., Goebel, P., Chow, V., Ewald, H. M., Yang, J., Wang, Z., Sadeghian, A., Sadigh, D., Savarese, S., et al. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794062 · Cited by 82
“We learn the parameters of the Neural Networks of our system through back-propagation with three Adam optimization procedures [59] aimed at providing an easier path to learning, similar to curriculum learning [60] or shaping [61].”
MethodImage similarity using Deep CNN and Curriculum Learning
Appalaraju, S., & Chaoji, V. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.08761 · Cited by 56
“This is inspired from Curriculum Learning [4].”
BackgroundLeibo, J. Z., Hughes, E., Lanctot, M., & Graepel, T. (2019) · arXiv · doi:10.48550/arxiv.1903.00742 · Cited by 60
“Researcher-controlled non-stationary dynamics in machine learning, known as curricula, can facilitate the acquisition of complex behaviors that would not be learnable otherwise, e.g. Asmuth et al. (2008); Bengio et al. (2009); Czarnecki et al. (2018); Narvekar (2017); Zaremba and Sutskever (2015).”
BackgroundOn the Intrinsic Dimensionality of Image Representations
Gong, S., Boddeti, V. N., & Jain, A. K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00411 · Cited by 59
“We adopt a curriculum learning [3] approach to overcome this challenge and progressively reduce the dimensionality of the mapping in multiple stages.”
BackgroundMethodDetecting Cross-Lingual Semantic Divergence for Neural Machine Translation
Carpuat, M., Vyas, Y., & Niu, X. (2017) · NMT@ACL · doi:10.18653/v1/w17-3209 · Cited by 65
“Online training also motivates curriculum learning approaches: ordering examples from easier short sentences to harder long sentences has also been found advantageous for neural language modeling (Bengio et al., 2009).”
BackgroundEstimating Example Difficulty using Variance of Gradients
Agarwal, C., D'souza, D., & Hooker, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01012 · Cited by 65
“Given the size of current datasets, this can be a powerful interpretability tool to isolate a tractable subset of examples for human-in-the-loop auditing and aid in curriculum learning [8] or distinguishing between sources of uncertainty [16, 33].”
BackgroundAn End-to-End Curriculum Learning Approach for Autonomous Driving Scenarios
Anzalone, L., Barra, P., Barra, S., Castiglione, A., & Nappi, M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3160673 · Cited by 85
“Since the problem of autonomous driving is extremely complex we adopt a stage-based learning procedure for our PPO agent, inspired by Curriculum Learning [12].”
MethodBecoming the expert - interactive multi-class machine teaching
Johns, E., Aodha, O. M., & Brostow, G. J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298877 · Cited by 57
“Similar to curriculum learning [5], our strategy initially focuses on representative images and then introduces more difficult ones over time, as the student’s performance improves.”
BackgroundBridging logic and kernel machines
Diligenti, M., Gori, M., Maggini, M., & Rigutini, L. (2011) · Machine Learning · doi:10.1007/s10994-011-5243-x · Cited by 59
“…subject of an in-depth analysis in children by Jean Piaget, we acquire experimental evidence on the crucial role of stage-based learning sketched in (Gori 2009), and reinforce the importance of insights on teaching issues like those pointed out by the notion of curriculum learning (Bengio 2009).”
BackgroundInfluentialAggregating Continuous Word Embeddings for Information Retrieval
Clinchant, S., & Perronnin, F. (2013) · Cited by 92
Ge, W., Huang, W., Guo, S., & Scott, M. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00344 · Cited by 68
“In this work, we propose a weakly-supervised learning approach for instance segmentation (Label-PEnet), which is inspired by the divide-and-conquer idea in curriculum learning [3].”
BackgroundMethodProgressive Adversarial Networks for Fine-Grained Domain Adaptation
Wang, S., Chen, X., Wang, Y., Long, M., & Wang, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00923 · Cited by 85
“Inspired by curriculum learning [3], we begin with the easy granularity, and then progressively move to the difficult granularity.”
BackgroundMethodTeacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
Portelas, R., Colas, C., Hofmann, K., & Oudeyer, P.-Y. (2019) · arXiv · doi:10.48550/arxiv.1910.07224 · Cited by 59
“Curriculum learning , as formulated in the supervised machine learning community, initially refers to techniques aimed at organizing labeled data to optimize the training of neural networks [11, 12, 13].”
BackgroundFew-shot Generative Modelling with Generative Matching Networks
Bartunov, S., & Vetrov, D. (2018) · International Conference on Artificial Intelligence and Statistics · Cited by 75
Improving de novo molecular design with curriculum learning
Guo, J., Fialková, V., Arango, J. D., Margreitter, C., Janet, J. P., Papadopoulos, K., Engkvist, O., & Patronov, A. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00494-4 · Cited by 75
“Curriculum learning (CL) has been proposed as a training strategy to overcome difficulties in learning complex tasks 21 .”
BackgroundCurriculum Learning for Multi-task Classification of Visual Attributes
Sarafianos, N., Giannakopoulos, T., Nikou, C., & Kakadiaris, I. A. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.306 · Cited by 66
“[2].”
BackgroundExperience Selection in Deep Reinforcement Learning for Control
de Bruin, T., Kober, J., Tuyls, K., & Babuška, R. (2018) · Research Repository (Delft University of Technology) · doi:10.5555/3291125.3291134 · Cited by 68
Narayana, P. A., Coronado, I., Sujit, S. J., Wolinsky, J. S., LUBLIN, F. D., & Gabr, R. E. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26959 · Cited by 60
Single Headed Attention Based Sequence-to-Sequence Model for State-of-the-Art Results on Switchboard
Tüske, Z., Saon, G., Audhkhasi, K., & Kingsbury, B. E. D. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1488 · Cited by 74
“Curriculum learning simplifies deep neural network training by presenting training examples in a meaningful order, usually by increasing order of difficulty [21].”
BackgroundMeta Self-training for Few-shot Neural Sequence Labeling
Wang, Y., Mukherjee, S., Chu, H., Tu, Y., Wu, M., Gao, J., & Awadallah, A. H. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467235 · Cited by 80
“Curriculum learning [2] techniques are based on the idea of learning easier aspects of the task first followed by the more complex ones.”
BackgroundGenet: automatic curriculum generation for learning adaptation in networking
Xia, Z., Zhou, Y., Yan, F. Y., & Jiang, J. (2022) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3544216.3544243 · Cited by 86
“The theoretical intuition behind curriculum learning is that a curriculum allows the model to optimize a family of gradually less smooth loss functions and prevents it from being trapped in local minima [7].”
BackgroundRDP-Net: Region Detail Preserving Network for Change Detection
Chen, H., Pu, F., Yang, R. Q., Tang, R., & Xu, X. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3227098 · Cited by 60
“Curriculum learning [37] suggests that simple, basic tasks are more useful for network training.”
BackgroundTeachAugment: Data Augmentation Optimization Using Teacher Knowledge
Suzuki, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01063 · Cited by 91
“This would be because the online optimization frameworks have aspects similar to curriculum learning [2].”
A low-cost, high-coverage legal named entity recognizer, classifier and linker
Cardellino, C., Teruel, M., Alemany, L. A., et al. (2017) · doi:10.1145/3086512.3086514 · Cited by 57
“We have explored more complex configurations of the neural network, including Curriculum Learning [5], a learning strategy that is specially adequate for hierarchically structured problems like ours, with subsequent levels of granularity.”
Web application firewall using character-level convolutional neural network
Ito, M., & Iyatomi, H. (2018) · IEEE International Colloquium on Signal Processing & Its Applications (CSPA) · doi:10.1109/cspa.2018.8368694 · Cited by 65
“In natural language processing(NLP)task, recurrent neural network (RNN) [10] and long short term memory (LSTM) [11] are often used and demonstrated their effects [12].”
BackgroundVisual sense of number vs. sense of magnitude in humans and machines
Testolin, A., Dolfi, S., Rochus, M., & Zorzi, M. (2020) · Scientific Reports · doi:10.1038/s41598-020-66838-5 · Cited by 56
“…improves when starting with easier trials and gradually progressing to harder ones, compared with the reverse (an effect known as perceptual hysteresis 44 ; in the machine learning literature, progression from easy to hard training samples is instead known as curriculum learning 45,46 ).”
BackgroundBeyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
Wu, L., Tan, X., He, D., Tian, F., Qin, T., Lai, J., & Liu, T.-Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1396 · Cited by 54
“In this section, we study different branching languages from two aspects, including the n-gram statistics of a target language, which has been used as a kind of characterization of hardness of learning (Bengio et al., 2009), and the dependency statistics in parse trees.”
BackgroundSalient Object Detection Based on Progressively Supervised Learning for Remote Sensing Images
Zhang, L., & Ma, J. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3045708 · Cited by 59
“2) Curriculum Learning in Training Phase: Curriculum learning [74], which gradually increases the difficulty of the task, has shown great power to enhance the representation capacity in the training procedure.”
BackgroundCo-Correcting: Noise-Tolerant Medical Image Classification via Mutual Label Correction
Liu, J., Li, R., & Sun, C. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3091178 · Cited by 62
“Curriculum learning [30] is a learning strategy in machine learning, which starts from easy instances and then gradually handles harder ones.”
ST3D++: Denoised Self-Training for Unsupervised Domain Adaptation on 3D Object Detection
Yang, J., Shi, S., Wang, Z., Li, H., & Qi, X. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3216606 · Cited by 107
“It also shares a similar spirit with curriculum learning [57], where confident easy boxes are learned at earlier stages and ambiguous hard examples are handled later after the model has been trained well enough to distinguish these examples.”
BackgroundMethodInstance-Specific and Model-Adaptive Supervision for Semi-Supervised Semantic Segmentation
Zhao, Z. W., Long, S., Pi, J., Wang, J., & Zhou, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02270 · Cited by 90
“Inspired by curriculum learning [2], we prioritize the training on easy samples over hard ones.”
MethodPseudo Labeling Methods for Semi-Supervised Semantic Segmentation: A Review and Future Perspectives
Ran, L., Li, Y., Liang, G., & Zhang, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3508768 · Cited by 92
Teaching Classification Boundaries to Humans
Basu, S., & Christensen, J. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8623 · Cited by 58
“…interpretation has found success in recent work on online learning, in which easier examples are presented before more difficult examples for online training of classifiers; this approach is known as “curriculum learning,” as explored in (Bengio et al. 2009) and (Kumar, Packer, and Koller 2010).”
Spitkovsky, V. I., Alshawi, H., & Jurafsky, D. S. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1204 · Cited by 51
“…named entity classification (Collins and Singer, 1999; Yarowsky, 1995), 11 parts-of-speech induction (Clark, 2000; 2003), and language modeling (Krueger and Dayan, 2009; Bengio et al., 2009), in addition to unsupervised parsing (Spitkovsky et al., 2009; Tu and Honavar, 2011; Cohn et al., 2011).”
BackgroundThe strategic student approach for life-long exploration and learning
Lopes, M., & Oudeyer, P.-Y. (2012) · 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL) · doi:10.1109/devlrn.2012.6400807 · Cited by 55
“A first family of approaches have considered mechanisms where ordered training examples are provided by an external teacher, such as in optimal teaching techniques [1] or in curriculum learning [2], [3].”
BackgroundD2C: Deep cumulatively and comparatively learning for human age estimation
Li, K., Xing, J., Hu, W., & Maybank, S. J. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2017.01.007 · Cited by 60
“Inspired by the concept of “curriculum learning” proposed in [45], we use the easy face pairs at the beginning and gradually increase the difficulty of the face 205 pairs.”
MethodScalable Logo Recognition Using Proxies
Fehérvári, I., & Appalaraju, S. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00081 · Cited by 62
“Curriculum learning [3] was explored by [1] where a probability distribution was used to sample image pairs online in order of their hardness, with easy to cluster image pairs sampled more earlier on in training and harder to identify pairs introduced later on in training.”
MethodBrain SegNet: 3D local refinement network for brain lesion segmentation
Hu, X., Luo, W., Hu, J., Guo, S., Huang, W., Scott, M. R., Wiest, R., Dahlweid, F.-M., & Reyes, M. (2020) · BMC Medical Imaging · doi:10.1186/s12880-020-0409-2 · Cited by 66
“Our training scheme is inspired from the intuition of curriculum learning [18], which encourages the learning to start from an easier task, and then takes more complex tasks gradually during the training process.”
MethodSmall Sample Learning in Big Data Era
Shu, J., Xu, Z., & Meng, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04572 · Cited by 50
“…are learning regimes inspired by the learning process of humans and animals that implements learning through gradually including samples into training process from easy to complex so as to increase the entropy of training samples (Bengio et al., 2009; Kumar et al., 2010; Jiang et al., 2014a,b).”
BackgroundMethodInfluentialAre all negatives created equal in contrastive instance discrimination?
Cai, T., Frankle, J., Schwab, D. J., & Morcos, A. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06682 · Cited by 53
“While developing a negative curriculum is beyond the scope of this work, curricula have shown utility in many other contexts (Bengio et al., 2009).”
BackgroundEfficient Convolutional Neural Networks for Depth-Based Multi-Person Pose Estimation
Martinez-Gonzalez, A. N., Villamizar, M., Canévet, O., & Odobez, J. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2952779 · Cited by 59
“One of them is curriculum learning [40], i.”
BackgroundMosaic flows: A transferable deep learning framework for solving PDEs on unseen domains
Wang, H., Planas, R., Chandramowlishwaran, A. M., & Bostanabad, R. (2021) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2021.114424 · Cited by 101
“Our task decomposition strategy is motivated by curriculum learning [66] which was originally developed for animal training [67] where difficult tasks are divided into simpler tasks which are easier to learn and collectively make up the original task.”
MethodMultibody dynamics and control using machine learning
Hashemi, A., Orzechowski, G., Mikkola, A., & McPhee, J. (2023) · Multibody System Dynamics · doi:10.1007/s11044-023-09884-x · Cited by 64
“In [101], DRL was coupled with curriculum learning [102] for simulating natural human gait.”
MethodXiao, J., Pisutsin, P., & Feroskhan, M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3331370 · Cited by 77
“In the standard curriculum learning methods [17], [34], the curriculum is manually designed with a static, discrete parameter sequence in which the task difficulty level is progressively increased with a pre-designed mode.”
Sub-domain Modelling for Dialogue Management with Hierarchical Reinforcement Learning
Budzianowski, P., Ultes, S., Su, P.-H., Mrkšić, N., Wen, T.-H., Casanueva, I., Rojas-Barahona, L. M., & Gašić, M. (2017) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w17-5512 · Cited by 64
“Following the idea of curriculum learning (Bengio et al., 2009), we test the adaptation capabilities of pre-trained policies to more complex situations.”
MethodPota, M., Marulli, F., Esposito, M., Pietro, G. D., & Fujita, H. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.11.003 · Cited by 59
Comparing Task Simplifications to Learn Closed-Loop Object Picking Using Deep Reinforcement Learning
Breyer, M., Furrer, F., Novković, T., Siegwart, R., & Nieto, J. I. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2896467 · Cited by 73
“methodology of curriculum learning [1], we guide the training of our models by progressively increasing the workspace with the agent’s performance and compare this method against reward function shaping and using a heuristic to bootstrap the full problem.”
BackgroundMethodSelf-Paced Collaborative and Adversarial Network for Unsupervised Domain Adaptation
Zhang, W., Xu, D., Ouyang, W., & Li, W. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2962476 · Cited by 56
“Different from those works, in this work, we propose a SPCAN model, which is inspired by the Curriculum Learning (CL) [57] and Self-Paced Learning [28], to select target pseudo-labeled samples in an easy-to-hard fashion, thus improving the stability of progressive training.”
MethodTriplet-Based Wireless Channel Charting: Architecture and Experiments
Ferrand, P., Decurninge, A., Ordóñez, L. G., & Guillaud, M. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3087251 · Cited by 54
“The selection of triplets can also be made increasingly challenging so that the embedding discriminates better between the classes; such an approach is called curriculum learning or shaping [27].”
BackgroundCurriculum Pre-training Heterogeneous Subgraph Transformer for Top-NRecommendation
Wang, H., Zhou, K., Zhao, W. X., Wang, J., & Wen, J.-R. (2023) · ACM Transactions on Information Systems · doi:10.1145/3528667 · Cited by 70
“Such an idea can be in essence characterized by curriculum learning [1], which starts from simple tasks or instances and gradually transforms tomore complex ones [30, 43].”
BackgroundSong, D., Gan, W., Yao, P., Zang, W., Zhang, Z., & Qu, X. (2022) · Ocean Engineering · doi:10.1016/j.oceaneng.2022.110947 · Cited by 54
The A2iA Multi-lingual Text Recognition System at the Second Maurdor Evaluation
Moysset, B., Bluche, T., Knibbe, M., BenZeghiba, M. F., Messina, R., Louradour, J., & Kermorvant, C. (2014) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2014.57 · Cited by 58
“Besides, we followed the principle of “Curriculum Learning” [16] to optimize the RNN by stochastic gradient descent.”
MethodSelf-citationLearning Representations for Weakly Supervised Natural Language Processing Tasks
Huang, F., Ahuja, A., Downey, D., Yang, Y., Guo, Y., & Yates, A. (2013) · Computational Linguistics · doi:10.1162/coli_a_00167 · Cited by 55
Learning Monocular Visual Odometry through Geometry-Aware Curriculum Learning
Saputra, M. R. U., de Gusmão, P. P. B., Wang, S., Markham, A., & Trigoni, N. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793581 · Cited by 56
“[8] proposed CL as a strategy to improve the convergence speed and generalization ability of a machine learning model by learning through highly organized or meaningful order of examples.”
BackgroundInfluentialShortformer: Better language modeling using shorter inputs
Press, O., Smith, N. A., & Lewis, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.427 · Cited by 55
“All models are trained in two stages; the second stage always uses a subsequence length of 3 , 072 , 6 Curriculum learning (Bengio et al., 2009) trains on easier inputs before progressing to harder ones.”
BackgroundMethod
Metadata
- Authors
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J.
- Affiliations
- Université de Montréal (CA)
- NEC Laboratories America, Princeton, NJ
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 41-48
- OpenAlex
- W2296073425
- Semantic Scholar
- 8de174ab5419b9d3127695405efd079808e956e8
- Cited by
- 5,366
- References
- 40
- Citations by year
- 2026: 526 · 2025: 636 · 2024: 551 · 2023: 591 · 2022: 536 · 2021: 711 · 2020: 565 · 2019: 442 · 2018: 293 · 2017: 198 · 2016: 103 · 2015: 63 · 2014: 41 · 2013: 40 · 2012: 33
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- Canadian Institute for Advanced Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- curriculum learning, training order, non-convex optimization, deep neural network, stochastic neural networks, deterministic neural networks, generalization, convergence rate, numerical continuation, global optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.