Meta-learning within Projective Simulation
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Learning models of artificial intelligence can nowadays perform very well on a large variety of tasks. However, in practice, different task environments are best handled by different learning models, rather than a single universal approach. Most non-trivial models thus require the adjustment of several to many learning parameters, which is often done on a case-by-case basis by an external party. Meta-learning refers to the ability of an agent to autonomously and dynamically adjust its own learning parameters or meta-parameters. In this paper, we show how projective simulation, a recently developed model of artificial intelligence, can naturally be extended to account for meta-learning in reinforcement learning settings. The projective simulation approach is based on a random walk process over a network of clips. The suggested meta-learning scheme builds upon the same design and employs clip networks to monitor the agent's performance and to adjust its meta-parameters on the fly. We distinguish between reflex-type adaptation and adaptation through learning, and show the utility of both approaches. In addition, a trade-off between flexibility and learning-time is addressed. The extended model is examined on three different kinds of reinforcement learning tasks, in which the agent has different optimal values of the meta-parameters, and is shown to perform well, reaching near-optimal to optimal success rates in all of them, without ever needing to manually adjust any meta-parameter.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Aharonov, Y., Davidovich, L., & Zagury, N. (1993) · Physical Review A · doi:10.1103/physreva.48.1687 · Cited by 1,622
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359 · cited 1×
“The second problem in meta-learning, the tuning of hyperparameters, is usually solved by gradient-based optimization [19], grid search [20], random search [21], genetic algorithms [22] or Bayesian optimization [23, 25].”
BackgroundExponential algorithmic speedup by a quantum walk
Childs, A. M., Cleve, R., Deotto, E., Farhi, E., Gutmann, S., & Spielman, D. A. (2003) · Proceedings of the thirty-fifth annual ACM symposium on Theory of computing · doi:10.1145/780542.780552 · Cited by 825
Aharonov, D., Ambainis, A., Kempe, J., & Vazirani, U. (2001) · Proceedings of the thirty-third annual ACM symposium on Theory of computing · doi:10.1145/380752.380758 · Cited by 654 · cited 1×
“Moreover, the random walk can be extended to the quantum regime, leading to quantum walks [37]–[39], in which case polynomial and even exponential improvements have been reported in e.”
BackgroundMeta-learning in Reinforcement Learning
Schweighofer, N., & Doya, K. (2003) · Neural Networks · doi:10.1016/s0893-6080(02)00228-9 · Cited by 261 · cited 1×
“An example for such a reflex-type adaptation approach for meta-learning can be found in [15] where the fundamental RL parameters, namely, the learning rate α, the exploitationexploration parameter β, and the discount factor γ are tuned according to predefined equations; In contrast, an agent”
MethodQuantum Speedup for Active Learning Agents
Paparo, G. D., Dunjko, V., Makmal, A., Martin-Delgado, M. A., & Briegel, H. J. (2014) · Physical Review X · doi:10.1103/physrevx.4.031002 · Cited by 192 · cited 1×
“Recently, a quantum variant of PS (envisioned already in [29]) was indeed formalized and shown to exhibit a quadratic speed-up in deliberation time over its classical counterpart [43]–[45].”
BackgroundSelf-citationDiscrete Quantum Walks Hit Exponentially Faster
Kempe, J. (2005) · Probability Theory and Related Fields · doi:10.1007/s00440-004-0423-2 · Cited by 152
Cited by
Machine learning & artificial intelligence in the quantum domain: a review of recent progress
Dunjko, V., & Briegel, H. J. (2018) · Reports on Progress in Physics · doi:10.1088/1361-6633/aab406 · Cited by 1,055
Self-citationQuantum-Enhanced Machine Learning
Dunjko, V., Taylor, J. M., & Briegel, H. J. (2016) · Physical Review Letters · doi:10.1103/physrevlett.117.130501 · Cited by 460
Self-citationActive learning machine learns to create new quantum experiments
Melnikov, A. A., Poulsen Nautrup, H., Krenn, M., Dunjko, V., Tiersch, M., Zeilinger, A., & Briegel, H. J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714936115 · Cited by 296
Self-citationSystem Design Perspective for Human-Level Agents Using Deep Reinforcement Learning: A Survey
Nguyen, N. D., Nguyen, T., & Nahavandi, S. (2017) · IEEE Access · doi:10.1109/access.2017.2777827 · Cited by 100
“Can we create an agent that can think and be creative? Recent advances in hierarchical RL, intrinsic motivation, meta-learning [74] (ability to adjust itself to adapt with the environment) and inverse RL [75] (a process of inferring reward function via demonstrations) are the first steps in the progress of building a…”
BackgroundAdvances in quantum reinforcement learning
Dunjko, V., Taylor, J. M., & Briegel, H. J. (2017) · 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · doi:10.1109/smc.2017.8122616 · Cited by 81
“For example, in [17], the meta-parameter – the “forgetting” parameter, but the details are not relevant here – of the PS model is, is optimized.”
BackgroundMethodResultSelf-citation
Metadata
- Authors
- Makmal, A., Melnikov, A. A., Dunjko, V., & Briegel, H. J.
- Venue
- IEEE Access
- Published
- 2016
- Volume
- 4
- Pages
- 2110-2122
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2949945034
- Cited by
- 38
- References
- 7
- FWCI
- 4.15
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Meta-learning within Projective Simulation. https://app.cochat.ai/papers/10.1109/access.2016.2556579
- The paper
- Makmal, A., Melnikov, A. A., Dunjko, V., & Briegel, H. J. (2016). Meta-learning within Projective Simulation. IEEE Access. https://doi.org/10.1109/access.2016.2556579
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.