Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Quantum machine learning (QML) has been identified as one of the key fields that could reap advantages from near-term quantum devices, next to optimization and quantum chemistry. Research in this area has focused primarily on variational quantum algorithms (VQAs), and several proposals to enhance supervised, unsupervised and reinforcement learning (RL) algorithms with VQAs have been put forward. Out of the three, RL is the least studied and it is still an open question whether VQAs can be competitive with state-of-the-art classical algorithms based on neural networks (NNs) even on simple benchmark tasks. In this work, we introduce a training method for parametrized quantum circuits (PQCs) that can be used to solve RL tasks for discrete and continuous state spaces based on the deep Q-learning algorithm. We investigate which architectural choices for quantum Q-learning agents are most important for successfully solving certain types of environments by performing ablation studies for a number of different data encoding and readout strategies. We provide insight into why the performance of a VQA-based Q-learning algorithm crucially depends on the observables of the quantum model and show how to choose suitable observables based on the learning task at hand. To compare our model against the classical DQN algorithm, we perform an extensive hyperparameter search of PQCs and NNs with varying numbers of parameters. We confirm that similar to results in classical literature, the architectural choices and hyperparameters contribute more to the agents' success in a RL setting than the number of parameters used in the model. Finally, we show when recent separation results between classical and quantum agents for policy gradient RL can be extended to inferring optimal Q-values in restricted families of environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Quantum Computing in the NISQ era and beyond
Preskill, J. (2018) · Quantum · doi:10.22331/q-2018-08-06-79 · Cited by 8,830
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 3×
“Actor-critic methods are among the state-of-the-art in current RL literature [26], and therefore both value-based and policybased algorithms are areas of active research.”
BackgroundMethodInfluentialPolynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer
Shor, P. W. (1999) · SIAM Review · doi:10.1137/s0036144598347011 · Cited by 3,904
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 2×
“[24] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.”
BackgroundMethodHardware-efficient variational quantum eigensolver for small molecules and quantum magnets
Kandala, A., Mezzacapo, A., Temme, K., Takita, M., Brink, M., Chow, J. M., & Gambetta, J. M. (2017) · Nature · doi:10.1038/nature23879 · Cited by 2,982 · cited 2×
“For this, we use a hardware-efficient ansatz [60] as shown in fig.”
BackgroundMethodSupervised learning with quantum-enhanced feature spaces
Havlíček, V., Córcoles, A. D., Temme, K., Harrow, A. W., Kandala, A., Chow, J. M., & Gambetta, J. M. (2019) · Nature · doi:10.1038/s41586-019-0980-2 · Cited by 2,572
Barren plateaus in quantum neural network training landscapes
McClean, J. R., Boixo, S., Smelyanskiy, V. N., Babbush, R., & Neven, H. (2018) · Nature Communications · doi:10.1038/s41467-018-07090-4 · Cited by 2,329
Noisy intermediate-scale quantum algorithms
Bharti, K., Cervera-Lierta, A., Kyaw, T. H., Haug, T., Alperin-Lea, S., Anand, A., Degroote, M., Heimonen, H., Kottmann, J. S., Menke, T., et al. (2022) · Reviews of Modern Physics · doi:10.1103/revmodphys.94.015004 · Cited by 2,231 · cited 1×
“Quantum machine learning has emerged as one field that is amenable to applications of VQAs on noisy intermediatescale quantum (NISQ) devices [1, 2].”
MethodQuantum Machine Learning in Feature Hilbert Spaces
Schuld, M., & Killoran, N. (2019) · Physical Review Letters · doi:10.1103/physrevlett.122.040504 · Cited by 1,840 · cited 1×
“Many proposals for QML algorithms have been made in supervised [3, 4, 5, 6, 7] and unsupervised [8, 9, 10, 11, 12, 13] learning.”
BackgroundNeural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Mitarai, K., Negoro, M., Kitagawa, M., & Fujii, K. (2018) · Physical Review A · doi:10.1103/physreva.98.032309 · Cited by 1,639 · cited 1×
“[3] Kosuke Mitarai, Makoto Negoro, Masahiro Kitagawa, and Keisuke Fujii.”
BackgroundDota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · cited 2×
“These types of RL algorithms have been shown to play Atari arcade games as well as human players [21], and even reach super-human levels of performance on games as complex as Go [22], Dota [23] and StarCraft [24].”
BackgroundMethodCircuit-centric quantum classifiers
Schuld, M., Bocharov, A., Svore, K. M., & Wiebe, N. (2020) · Physical Review A · doi:10.1103/physreva.101.032308 · Cited by 1,057 · cited 1×
“[4] Maria Schuld, Alex Bocharov, Krysta M Svore, and Nathan Wiebe.”
BackgroundCost function dependent barren plateaus in shallow parametrized quantum circuits
Cerezo, M., Sone, A., Volkoff, T., Cincio, L., & Coles, P. J. (2021) · Nature Communications · doi:10.1038/s41467-021-21728-w · Cited by 1,047
Sim, S., Johnson, P. D., & Aspuru‐Guzik, A. (2019) · Advanced Quantum Technologies · doi:10.1002/qute.201900070 · Cited by 972 · cited 1×
“[45] Sukin Sim, Peter D Johnson, and Alán Aspuru-Guzik.”
BackgroundSmith, L. N. (2018) · arXiv · doi:10.48550/arxiv.1803.09820 · Cited by 790 · cited 1×
“” [28] Thus a whole field of heuristics and numerical studies has formed on finding good sets of hyperparameters like NN architectures [29–31], activation functions [32–34], or learning rates and batch sizes [28, 35].”
BackgroundNoise-induced barren plateaus in variational quantum algorithms
Wang, S., Fontana, E., Cerezo, M., Sharma, K., Sone, A., Cincio, L., & Coles, P. J. (2021) · Nature Communications · doi:10.1038/s41467-021-27045-6 · Cited by 773
Effect of data encoding on the expressive power of variational quantum-machine-learning models
Schuld, M., Sweke, R., & Meyer, J. J. (2021) · Physical Review A · doi:10.1103/physreva.103.032430 · Cited by 739 · cited 5×
“The fact that agents with trainable input weights and data re-uploading perform much better than those without, emphasizes the importance of matching the PQC’s expressivity to the learning task at hand, as described in [48].”
BackgroundMethodInfluentialData re-uploading for a universal quantum classifier
Pérez-Salinas, A., Cervera-Lierta, A., Gil-Fuster, E., & Latorre, J. I. (2020) · Quantum · doi:10.22331/q-2020-02-06-226 · Cited by 713
A rigorous and robust quantum speed-up in supervised machine learning
Liu, Y., Arunachalam, S., & Temme, K. (2021) · Nature Physics · doi:10.1038/s41567-021-01287-z · Cited by 649
Classification with Quantum Neural Networks on Near Term Processors
Farhi, E., & Neven, H. (2018) · arXiv · doi:10.48550/arxiv.1802.06002 · Cited by 598
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582
Quantum Generative Adversarial Learning
Lloyd, S., & Weedbrook, C. (2018) · Physical Review Letters · doi:10.1103/physrevlett.121.040502 · Cited by 576 · cited 1×
“[11] Seth Lloyd and Christian Weedbrook.”
Amin, M. H., Andriyash, E., Rolfe, J., Kulchytskyy, B., & Melko, R. (2018) · Physical Review X · doi:10.1103/physrevx.8.021050 · Cited by 520 · cited 1×
“[8] Mohammad H Amin, Evgeny Andriyash, Jason Rolfe,”
BackgroundQuantum Generative Adversarial Networks for learning and loading random distributions
Zoufal, C., Lucchi, A., & Woerner, S. (2019) · npj Quantum Information · doi:10.1038/s41534-019-0223-2 · Cited by 480
Layerwise learning for quantum neural networks
Skolik, A., McClean, J. R., Mohseni, M., van der Smagt, P., & Leib, M. (2021) · Quantum Machine Intelligence · doi:10.1007/s42484-020-00036-4 · Cited by 324 · cited 1×
“[43] Andrea Skolik, Jarrod R McClean, Masoud Mohseni, Patrick van der Smagt, and Martin Leib.”
BackgroundSelf-citationDifferentiable learning of quantum circuit Born machines
Liu, J.-G., & Wang, L. (2018) · Physical Review A · doi:10.1103/physreva.98.062324 · Cited by 279
Noisy intermediate-scale quantum (NISQ) algorithms
Bharti, K., Cervera-Lierta, A., Kyaw, T. H., Haug, T., Alperin-Lea, S., Anand, A., Degroote, M., Heimonen, H., Kottmann, J. S., Menke, T., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.08448 · Cited by 166
The Born supremacy: quantum advantage and training of an Ising Born machine
Coyle, B., Mills, D., Danos, V., & Kashefi, E. (2020) · npj Quantum Information · doi:10.1038/s41534-020-00288-9 · Cited by 155 · cited 1×
“[9] Brian Coyle, Daniel Mills, Vincent Danos, and Elham Kashefi.”
BackgroundVariational Quantum Boltzmann Machines
Zoufal, C., Lucchi, A., & Woerner, S. (2020) · arXiv · doi:10.48550/arxiv.2006.06004 · Cited by 99
Advances in quantum reinforcement learning
Dunjko, V., Taylor, J. M., & Briegel, H. J. (2017) · 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · doi:10.1109/smc.2017.8122616 · Cited by 81
Self-citationLayer VQE: A Variational Approach for Combinatorial Optimization on Noisy Quantum Computers
Liu, X., Angone, A., Shaydulin, R., Safro, I., Alexeev, Y., & Cincio, L. (2022) · IEEE Transactions on Quantum Engineering · doi:10.1109/tqe.2021.3140190 · Cited by 78
Quantum enhancements for deep reinforcement learning in large spaces
Jerbi, S., Trenkwalder, L. M., Nautrup, H. P., Briegel, H. J., & Dunjko, V. (2019) · arXiv · doi:10.48550/arxiv.1910.12760 · Cited by 69
Self-citationAdaptive pruning-based optimization of parameterized quantum circuits
Sim, S., Romero, J., Gonthier, J. F., & Kunitsa, A. A. (2020) · arXiv · doi:10.48550/arxiv.2010.00629 · Cited by 66
Entanglement-Induced Barren Plateaus
Ortiz Marrero, C., Kieferová, M., & Wiebe, N. (2021) · PRX Quantum · doi:10.1103/prxquantum.2.040316 · Cited by 39
Cited by
Challenges and opportunities in quantum machine learning
Cerezo, M., Verdon, G., Huang, H.-Y., Cincio, L., & Coles, P. J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00311-3 · Cited by 689
Challenges and opportunities in quantum optimization
Abbas, A., Ambainis, A., Augustino, B., Bärtschi, A., Buhrman, H., Coffrin, C., Cortiana, G., Dunjko, V., Egger, D. J., Elmegreen, B. G., et al. (2024) · Nature Reviews Physics · doi:10.1038/s42254-024-00770-9 · Cited by 201
“For example, it has been proved 154,155 that there exists task environments with a super-polynomial advantage for quantum reinforcement-learning algorithms (with respect to finding optimal policies and optimal q -functions), unless the discrete logarithm problem has an efficient classical solution.”
Self-citationLiu, C.-Y., Kuo, E.-J., Abraham Lin, C.-H., Gemsun Young, J., Chang, Y.-J., Hsieh, M.-H., & Goan, H.-S. (2025) · Quantum Machine Intelligence · doi:10.1007/s42484-025-00305-0 · Cited by 40
Uncovering instabilities in variational-quantum deep Q-networks
Franz, M., Wolf, L., Periyasamy, M., Ufrecht, C., Scherer, D. D., Plinge, A., Mutschler, C., & Mauerer, W. (2023) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2022.08.021 · Cited by 26
Metadata
- Authors
- Skolik, A., Jerbi, S., & Dunjko, V.
- Venue
- Quantum
- Published
- 2022-05-24
- Volume
- 6
- Pages
- 720
- ISSN
- 2521-327X (ISSN-L)
- OpenAlex
- W3141755656
- Cited by
- 191
- References
- 38
- FWCI
- 12.84
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.