Reinforcement Learning in Autonomic Computing: A Manifesto and Case Studies
Abstract
Reinforcement learning is a promising new approach for automatically developing effective policies for real-time self-management. RL can achieve superior performance to traditional methods, while requiring less built-in domain knowledge. Several case studies from real and simulated systems management applications demonstrate RL's promises and challenges. These studies show that standard online RL can learn effective policies in feasible training times. Moreover, a Hybrid RL approach can profit from any knowledge contained in an existing policy by training on the policy's observable behavior, without needing to interface directly to such knowledge
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reinforcement learning is best suited for making cognitive choices, such as decision making, planning and scheduling.
“Hence, RL is best suited for making cognitive choices, such as decision making, planning and scheduling [441].”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [441]
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037 · see citing works
The vision of autonomic computing
Kephart, J. O., & Chess, D. M. (2003) · Computer · doi:10.1109/mc.2003.1160055 · Cited by 6,572 · see citing works
Temporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · see citing works
Self-citationFeedback Control of Computing Systems
Hellerstein, J. L., Diao, Y., Parekh, S. S., et al. (2004) · Wiley eBooks · doi:10.1002/047166880x · Cited by 974 · see citing works
Residual Algorithms: Reinforcement Learning with Function Approximation
Baird, L. C. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50013-x · Cited by 948 · see citing works
Temporal Difference Learning and TD-Gammon
Tesauro, G. (1995) · ICGA Journal · doi:10.3233/icg-1995-18207 · Cited by 770 · see citing works
Self-citationAutonomous Inverted Helicopter Flight via Reinforcement Learning
Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., & Liang, E. (2006) · Springer tracts in advanced robotics · doi:10.1007/11552246_35 · Cited by 557 · see citing works
Learning to trade via direct reinforcement
Moody, J. E., & Saffell, M. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935097 · Cited by 555 · see citing works
An analytical model for multi-tier internet services and its applications
Urgaonkar, B., Pacifici, G., Shenoy, P. J., et al. (2005) · ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS) · doi:10.1145/1064212.1064252 · Cited by 548 · see citing works
Performance by Design: Computer Capacity Planning By Example
Menascé, D. A., Dowdy, L. W., & Almeida, V. A. F. (2004) · CERN Document Server (European Organization for Nuclear Research) · Cited by 387 · see citing works
A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation
Tesauro, G., Jong, N. K., Das, R., & Bennani, M. N. (2006) · IEEE International Conference on Autonomic Computing (ICAC) · doi:10.1109/icac.2006.1662383 · Cited by 377 · see citing works
Self-citationResearch challenges of autonomic computing
Kephart, J. O. (2005) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/1062455.1062464 · Cited by 268 · see citing works
An analytical model for multi-tier internet services and its applications
Urgaonkar, B., Pacifici, G., Shenoy, P. J., et al. (2005) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/1071690.1064252 · Cited by 200 · see citing works
Online resource allocation using decompositional reinforcement learning
Tesauro, G. J. (2005) · Cited by 97 · see citing works
Self-citationAdaptive Action Selection in Autonomic Software Using Reinforcement Learning
Amoui, M., Salehie, M., Mirarab, S., & Tahvildari, L. (2008) · International Conference on Networking and Services · doi:10.1109/icas.2008.35 · Cited by 57 · see citing works
A reinforcement learning framework for utility-based scheduling in resource-constrained systems
Vengerov, D. (2008) · Future Generation Computer Systems · doi:10.1016/j.future.2008.02.006 · Cited by 42 · see citing works
A Reinforcement Learning Framework for Dynamic Resource Allocation: First Results.
Vengerov, D., & Iakovlev, N. (2005) · International Conference on Autonomic Computing · doi:10.1109/icac.2005.4 · Cited by 39 · see citing works
Adaptive job routing and scheduling
Whiteson, S., & Stone, P. (2004) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2004.08.027 · Cited by 34 · see citing works
A middleware for autonomic QoS management based on learning
Vienne, P., & Sourrouille, J.-L. (2005) · doi:10.1145/1108473.1108475 · Cited by 32 · see citing works
Internet Traffic: Periodicity, Tail Behavior, and Performance Implications
Gelenbe, E. (2000) · CRC Press eBooks · doi:10.1201/9781482274530-7 · Cited by 27 · see citing works
Towards autonomic computing:adaptive network routing and scheduling
Whiteson, S., & Stone, P. (2004) · International Conference on Autonomic Computing · doi:10.1109/icac.2004.1301381 · Cited by 16 · see citing works
A Weighted Voting Mechanism for Action Selection Problem in Self-Adaptive Software
Salehie, M., & Tahvildari, L. (2007) · doi:10.1109/saso.2007.4 · Cited by 15 · see citing works
Adaptive job routing and scheduling
Whiteson, S., & Stone, P. K. A. M. H. A. J. L. A. A. T. A. P. (2004) · Engineering Applications of Artificial Intelligence · doi:10.1016/s0952-1976(04)00109-5 · Cited by 15 · see citing works
Adaptive Utility-Based Scheduling in Resource-Constrained Systems
Vengerov, D. (2005) · Lecture notes in computer science · doi:10.1007/11589990_50 · Cited by 14 · see citing works
Collaborative reinforcement learning of autonomic behaviour
Dowling, J., Cunningham, R., Curran, E., et al. (2004) · Arrow@dit (Dublin Institute of Technology) · doi:10.1109/dexa.2004.47 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Boutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018) · Journal of Internet Services and Applications · doi:10.1186/s13174-018-0087-2 · Cited by 1,036 · see citing works
Towards energy-aware scheduling in data centers using machine learning
Berral, J. L., Goiri, Í., Nou, R., Julià, F., Guitart, J., Gavaldà, R., & Torres, J. (2010) · Proceedings of the 1st International Conference on Energy-Efficient Computing and Networking · doi:10.1145/1791314.1791349 · Cited by 262 · see citing works
Machine Learning for Cognitive Network Management
Ayoubi, S., Limam, N., Salahuddin, M. A., Shahriar, N., Boutaba, R., Estrada-Solano, F., & Caicedo, O. M. (2018) · IEEE Communications Magazine · doi:10.1109/mcom.2018.1700560 · Cited by 217 · see citing works
- Authors
- Tesauro, G.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- Corresponding author
- Tesauro, Gerald
- Venue
- IEEE Internet Computing
- Published
- 2007
- Volume
- 11
- Issue
- 1
- Pages
- 22-30
- ISSN
- 1089-7801 (ISSN-L)
- OpenAlex
- W2073105339
- Cited by
- 143
- References
- 29
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 3 · 2023: 1 · 2022: 6 · 2021: 6 · 2020: 7 · 2019: 8 · 2018: 11 · 2017: 3 · 2016: 8 · 2015: 1 · 2014: 4 · 2013: 15 · 2012: 9
- FWCI
- 14.98
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Internet Computing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- reinforcement learning, autonomic computing, self-management, policy learning, hybrid reinforcement learning, online learning, system management, policy transfer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
