CoChat
Sign Up
Cited by 751Magnetic confinement fusion researchFusion and Plasma Physics StudiesFusion materials and technologies

Magnetic control of tokamak plasmas through deep reinforcement learning

Open access
Degrave, Jonas, Felici, Federico, Buchli, Jonas, Neunert, Michael, Tracey, Brendan, Carpanese, Francesco, Ewalds, Timo, Hafner, Roland +23 more·2022·Nature
DOI 10.1038/s41586-021-04301-9PMC PMC8850200
Open this paper in CoChat — free accountOpen PDF

Abstract

Nuclear fusion using magnetic confinement, in particular in the tokamak configuration, is a promising path towards sustainable energy. A core challenge is to shape and maintain a high-temperature plasma within the tokamak vessel. This requires high-dimensional, high-frequency, closed-loop control using magnetic actuator coils, further complicated by the diverse requirements across a wide range of plasma configurations. In this work, we introduce a previously undescribed architecture for tokamak magnetic controller design that autonomously learns to command the full set of control coils. This architecture meets control objectives specified at a high level, at the same time satisfying physical and operational constraints. This approach has unprecedented flexibility and generality in problem specification and yields a notable reduction in design effort to produce new plasma configurations. We successfully produce and control a diverse set of plasma configurations on the Tokamak à Configuration Variable 1,2 , including elongated, conventional shapes, as well as advanced configurations, such as negative triangularity and ‘snowflake’ configurations. Our approach achieves accurate tracking of the location, current and shape for these configurations. We also demonstrate sustained ‘droplets’ on TCV, in which two separate plasmas are maintained simultaneously within the vessel. This represents a notable advance for tokamak feedback control, showing the potential of reinforcement learning to accelerate research in the fusion domain, and is one of the most challenging real-world systems to which reinforcement learning has been applied.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41586-021-04301-9

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Deep learning models are increasingly used to predict and control complex systems we do not yet fully understand.

    “Increasingly, deep learning models are used to predict and control complex systems we do not yet fully understand, such as inertially confined plasmas (Degrave et al., 2022) and biomoloecular systems (Abramson et al., 2024).”

    From Robust agents learn causal world models (2024) · cites this paper as Degrave et al., 2022

    The Discussion describes the work as a new paradigm for plasma magnetic confinement on tokamaks and reports successful hardware control experiments. That contradicts the citing sentence’s description of Degrave et al. as an example involving inertially confined plasmas, although the paper does support the broader point that deep learning can control a complex plasma system. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. [1]Hofmann F Creation and control of variably shaped plasmas in TCV Plasma Phys. Control. Fusion 1994 36 B277 10.1088/0741-3335/36/12B/023 · DOI
  2. [2]Coda S Physics research on the TCV tokamak facility: from conventional to alternative scenarios and beyond Nucl. Fusion 2019 59 112023 10.1088/1741-4326/ab25cb · DOI
  3. [3]Anand H Coda S Felici F Galperti C Moret J-M A novel plasma position and shape controller for advanced configuration development on the TCV tokamak Nucl. Fusion 2017 57 126026 10.1088/1741-4326/aa7f4d · DOI
  4. [4]Mele A MIMO shape control at the EAST tokamak: simulations and experiments Fusion Eng. Des. 2019 146 1282 1285 10.1016/j.fusengdes.2019.02.058 · DOI
  5. [5]Anand H Plasma flux expansion control on the DIII-D tokamak Plasma Phys. Control. Fusion 2020 63 015006 10.1088/1361-6587/abc457 · DOI
  6. [6]De Tommasi G Plasma magnetic control in tokamak devices J. Fusion Energy 2019 38 406 436 10.1007/s10894-018-0162-5 · DOI
  7. [7]Walker ML Humphreys DA Valid coordinate systems for linearized plasma shape response models in tokamaks Fusion Sci. Technol. 2006 50 473 489 10.13182/FST06-A1271 · DOI
  8. [8]Blum J Heumann H Nardon E Song X Automating the design of tokamak experiment scenarios J. Comput. Phys. 2019 394 594 614 10.1016/j.jcp.2019.05.046 · DOI
  9. [9]Ferron JR Real time equilibrium reconstruction for tokamak discharge control Nucl. Fusion 1998 38 1055 10.1088/0029-5515/38/7/308 · DOI
  10. [10]Moret J-M Tokamak equilibrium reconstruction code LIUQE and its real time implementation Fusion Eng. Des. 2015 91 1 15 10.1016/j.fusengdes.2014.09.019 · DOI
  11. [11]Xie, Z., Berseth, G., Clary, P., Hurst, J. & van de Panne, M. Feedback control for Cassie with deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 1241–1246 (IEEE, 2018).
  12. [12]Akkaya, I. et al. Solving Rubik’s cube with a robot hand. Preprint at https://arxiv.org/abs/1910.07113 (2019). · arXiv
  13. [13]Bellemare MG Autonomous navigation of stratospheric balloons using reinforcement learning Nature 2020 588 77 82 10.1038/s41586-020-2939-8 33268863 · DOI
  14. [14]Humphreys D Advancing fusion with machine learning research needs workshop report J. Fusion Energy 2020 39 123 155 10.1007/s10894-020-00258-1 · DOI · Paper page
  15. [15]Bishop CM Haynes PS Smith ME Todd TN Trotman DL Real time control of a tokamak plasma using neural networks Neural Comput. 1995 7 206 217 10.1162/neco.1995.7.1.206 · DOI
  16. [16]Joung S Deep neural network Grad-Shafranov solver constrained with measured magnetic signals Nucl. Fusion 2019 60 16034 10.1088/1741-4326/ab555f · DOI
  17. [17]van de Plassche KL Fast modeling of turbulent transport in fusion plasmas using neural networks Phys. Plasmas 2020 27 022310 10.1063/1.5134126 · DOI
  18. [18]Abbate J Conlin R Kolemen E Data-driven profile prediction for DIII-D Nucl. Fusion 2021 61 046027 10.1088/1741-4326/abe08d · DOI
  19. [20]Jardin, S. Computational Methods in Plasma Physics (CRC Press, 2010).
  20. [21]Grad H Rubin H Hydromagnetic equilibria and force-free fields J. Nucl. Energy (1954) 1958 7 284 285 10.1016/0891-3919(58)90139-6 · DOI
  21. [22]Carpanese, F. Development of Free-boundary Equilibrium and Transport Solvers for Simulation and Real-time Interpretation of Tokamak Experiments. PhD thesis, EPFL (2021).
  22. [23]Abdolmaleki, A. et al. Relative entropy regularized policy iteration. Preprint at https://arxiv.org/abs/1812.02256 (2018). · arXiv
  23. [24]Paley, J. I., Coda, S., Duval, B., Felici, F. & Moret, J.-M. Architecture and commissioning of the TCV distributed feedback control system. In 2010 17th IEEE-NPSS Real Time Conference 1–6 (IEEE, 2010).
  24. [25]Freidberg, J. P. Plasma Physics and Fusion Energy (Cambridge Univ. Press, 2008).
  25. [26]Hommen GD Real-time optical plasma boundary reconstruction for plasma position control at the TCV Tokamak Nucl. Fusion 2014 54 073018 10.1088/0029-5515/54/7/073018 · DOI
  26. [27]Austin ME Achievement of reactor-relevant performance in negative triangularity shape in the DIII-D tokamak Phys. Rev. Lett. 2019 122 115001 10.1103/PhysRevLett.122.115001 30951344 · DOI
  27. [28]Kolemen E Initial development of the DIII–D snowflake divertor control Nucl. Fusion 2018 58 066007 10.1088/1741-4326/aab0d3 · DOI
  28. [29]Anand H Real time magnetic control of the snowflake plasma configuration in the TCV tokamak Nucl. Fusion 2019 59 126032 10.1088/1741-4326/ab4440 · DOI
  29. [30]Wigbers, M. & Riedmiller, M. A new method for the analysis of neural reference model control. In Proc. International Conference on Neural Networks (ICNN’97) Vol. 2, 739–743 (IEEE, 1997).
  30. [31]Berkenkamp, F., Turchetta, M., Schoellig, A. & Krause, A. Safe model-based reinforcement learning with stability guarantees. In 2017 Advances in Neural Information Processing Systems 908–919 (ACM, 2017).
  31. [32]Wabersich KP Hewing L Carron A Zeilinger MN Probabilistic model predictive safety certification for learning-based control IEEE Tran. Automat. Control 2021 67 176 188 10.1109/TAC.2021.3049335 · DOI
  32. [33]Abdolmaleki, A. et al. On multi-objective policy optimization as a tool for reinforcement learning. Preprint at https://arxiv.org/abs/2106.08199 (2021). · arXiv
  33. [34]Coda S Overview of the TCV tokamak program: scientific progress and facility upgrades Nucl. Fusion 2017 57 102011 10.1088/1741-4326/aa6412 · DOI
  34. [35]Karpushov AN Neutral beam heating on the TCV tokamak Fusion Eng. Des. 2017 123 468 472 10.1016/j.fusengdes.2017.02.076 · DOI
  35. [36]Lister JB Plasma equilibrium response modelling and validation on JT-60U Nucl. Fusion 2002 42 708 10.1088/0029-5515/42/6/309 · DOI
  36. [37]Lister JB The control of tokamak configuration variable plasmas Fusion Technol. 1997 32 321 373 10.13182/FST97-A1 · DOI
  37. [38]Ulyanov, D., Vedaldi, A. & Lempitsky, V. Instance normalization: the missing ingredient for fast stylization. Preprint at https://arxiv.org/abs/1607.08022 (2016). · arXiv
  38. [39]Andrychowicz, M. et al. What matters in on-policy reinforcement learning? A large-scale empirical study. In ICLR 2021 Ninth International Conference on Learning Representations (2021).
  39. [40]Cassirer, A. et al. Reverb: a framework for experience replay. Preprint at https://arxiv.org/abs/2102.04736 (2021). · arXiv
  40. [41]Hoffman, M. et al. Acme: a research framework for distributed reinforcement learning. Preprint at https://arxiv.org/abs/2006.00979 (2020). · arXiv
  41. [42]Hofmann F FBT-a free-boundary tokamak equilibrium code for highly elongated and shaped plasmas Comput. Phys. Commun. 1988 48 207 221 10.1016/0010-4655(88)90041-0 · DOI
  42. [43]Abadi, M. et al. TensorFlow: a system for large-scale machine learning. In Proc. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16) 265–283 (2016).
  43. [44]De Tommasi G Model-based plasma vertical stabilization and position control at EAST Fusion Eng. Des. 2018 129 152 157 10.1016/j.fusengdes.2018.02.020 · DOI
  44. [45]Gerkšič, S. & De Tommasi, G. ITER plasma current and shape control using MPC. In 2016 IEEE Conference on Control Applications (CCA) 599–604 (IEEE, 2016).
  45. [46]Boncagni, L. et al. Performance-based controller switching: an application to plasma current control at FTU. In 2015 54th IEEE Conference on Decision and Control (CDC) 2319–2324 (IEEE, 2015).
  46. [47]Wakatsuki T Suzuki T Hayashi N Oyama N Ide S Safety factor profile control with reduced central solenoid flux consumption during plasma current ramp-up phase using a reinforcement learning technique Nucl. Fusion 2019 59 066022 10.1088/1741-4326/ab1571 · DOI
  47. [48]Wakatsuki T Suzuki T Oyama N Hayashi N Ion temperature gradient control using reinforcement learning technique Nucl. Fusion 2021 61 046036 10.1088/1741-4326/abe68d · DOI
  48. [49]Seo J Feedforward beta control in the KSTAR tokamak by deep reinforcement learning Nucl. Fusion 2021 61 106010 10.1088/1741-4326/ac121b · DOI
  49. [50]Yang, F. et al. Launchpad: a programming model for distributed machine learning research. Preprint at https://arxiv.org/abs/2106.04516 (2021). · arXiv
  50. [54]Hender, T. C. et al. Chapter 3: MHD stability, operational limits and disruptions. Nucl. Fusion47, S128–S202 (2007).

Cited by

Sorted by citations

Authors
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de las Casas, D., Donner, C., Fritz, L., Galperti, C., Huber, A., Keeling, J., Tsimpoukelli, M., Kay, J., Merle, A., Moret, J.-M., Noury, S., Pesamosca, F., Pfau, D., Sauter, O., Sommariva, C., Coda, S., Duval, B., Fasoli, A., Kohli, P., Kavukcuoglu, K., Hassabis, D., & Riedmiller, M.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • École Polytechnique Fédérale de Lausanne (CH)
Corresponding author
  • Felici, Federico
  • Buchli, Jonas
  • Tracey, Brendan
ORCID
Venue
Nature
Publisher
Springer Science and Business Media LLC
Published
2022-02-16
Volume
602
Issue
7897
Pages
414-419
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
DOI
10.1038/s41586-021-04301-9
PMC
PMC8850200
OpenAlex
W4213377513
Cited by
751
References
1
Citations by year
2026: 120 · 2025: 168 · 2024: 190 · 2023: 165 · 2022: 105 · 2019: 1
FWCI
19.67
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
Funders
  • National Science Foundation
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
Topics
  • Physical Sciences › Physics and Astronomy › Nuclear and High Energy Physics › Magnetic confinement fusion research
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Fusion and Plasma Physics Studies
  • Physical Sciences › Materials Science › Materials Chemistry › Fusion materials and technologies
Keywords
tokamak plasma control, magnetic confinement fusion, deep reinforcement learning, plasma shape control, negative triangularity, closed-loop feedback control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review