CoChat
Sign Up
Cited by 128Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

Robot Learning From Randomized Simulations: A Review

Open access
Muratore, Fabio, Ramos, Fabio, Turk, Greg, Yu, Wenhao, Gienger, Michael, Peters, Jan·2022·Frontiers in Robotics and AI
DOI 10.3389/frobt.2022.799893PMC PMC9038844
Open this paper in CoChat — free accountOpen PDF

Abstract

The rise of deep learning has caused a paradigm shift in robotics research, favoring methods that require large amounts of data. Unfortunately, it is prohibitively expensive to generate such data sets on a physical platform. Therefore, state-of-the-art approaches learn in simulation where data generation is fast as well as inexpensive and subsequently transfer the knowledge to the real robot (sim-to-real). Despite becoming increasingly realistic, all simulators are by construction based on models, hence inevitably imperfect. This raises the question of how simulators can be modified to facilitate learning robot control policies and overcome the mismatch between simulation and reality, often called the “reality gap.” We provide a comprehensive review of sim-to-real research for robotics, focusing on a technique named “domain randomization” which is a method for learning from randomized simulations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/frobt.2022.799893

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The substantial domain gap between synthetic and real-world data presents significant challenges in transferring models trained in simulated environments to real-world applications.

    “While simulation provides a means to generate extensive datasets, the substantial domain gap between synthetic and real-world data presents significant challenges in transferring models trained in simulated environments to real-world applications (Muratore et al., 2022).”

    From Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control (2025) · cites this paper as Muratore et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 165

  1. Abdulsamad H. Dorau T. Belousov B. Zhu J. Peters J. (2021). Distributionally Robust Trajectory Optimization under Uncertain Dynamics via Relative-Entropy Trust Regions. arXiv 2103. 15388
  2. Alghonaim R. Johns E. (2020). Benchmarking Domain Randomisation for Visual Sim-To-Real Transfer. arXiv 2011.07112 · arXiv
  3. Allevato A. Short E. S. Pryor M. Thomaz A. (2019). Tunenet: One-Shot Residual Tuning for System Identification and Sim-To-Real Robot Task Transfer. In Conference on Robot Learning (CoRL), Osaka, Japan, October 30 - November 1 (PMLR), vol. 100 of Proc. Machine Learn. Res., 445– 455.
  4. Amari S.-i. (1977). Dynamics of Pattern Formation in Lateral-Inhibition Type Neural fields. Biol. Cybern. 27, 77– 87. 10.1007/bf00337259 911931 · DOI
  5. Andrychowicz M. Crow D. Ray A. Schneider J. Fong R. Welinder P. (2017). “ Hindsight Experience Replay,” in Conference on Neural Information Processing Systems (NIPS), December 4-9 (Long Beach, CA, USA, 5048– 5058.
  6. Andrychowicz O. M. Baker B. Chociej M. Józefowicz R. McGrew B. Pachocki J. (2020). Learning Dexterous In-Hand Manipulation. Int. J. Robotics Res. 39, 3– 20. 10.1177/0278364919887447 · DOI · Paper page
  7. Antonova R. Rai A. Li T. Kragic D. (2019). “ Bayesian Optimization in Variational Latent Spaces with Dynamic Compression,” in Conference on Robot Learning (CoRL), October 30 - November 1 (Osaka, Japan: PMLR), 456– 465. of Proceedings of Machine Learning Research. 100
  8. Asada M. Noda S. Tawaratsumida S. Hosoda K. (1996). Purposive Behavior Acquisition for a Real Robot by Vision-Based Reinforcement Learning. Mach. Learn. 23, 279– 303. 10.1023/A:101823700882310.1007/bf00117447 · DOI
  9. Atkeson C. G. H C. A. An C. H. (1986). Estimation of Inertial Parameters of Manipulator Loads and Links. Int. J. Robotics Res. 5, 101– 119. 10.1177/027836498600500306 · DOI · Paper page
  10. Barcelos L. Lambert A. Oliveira R. Borges P. Boots B. Ramos F. (2021). “ Dual Online Stein Variational Inference for Control and Dynamics,” in Robotics: Science and Systems (RSS), July 12-16. Virtual Event. 10.15607/RSS.2021.XVII.068 · DOI
  11. Barcelos L. Oliveira R. Possas R. Ott L. Ramos F. (2020). “ DISCO: Double Likelihood-free Inference Stochastic Control,” in International Conference on Robotics and Automation (ICRA), May 31 - August 31 (Paris, France: IEEE), 10969– 10975. 10.1109/ICRA40945.2020.9196931 · DOI
  12. Battaglia P. W. Hamrick J. B. Tenenbaum J. B. (2013). Simulation as an Engine of Physical Scene Understanding. Proc. Natl. Acad. Sci. 110, 18327– 18332. 10.1073/pnas.1306572110 24145417 PMC3831455 · DOI · Paper page
  13. Battaglia P. W. Pascanu R. Lai M. Rezende D. J. Kavukcuoglu K. (2016). “ Interaction Networks for Learning about Objects, Relations and Physics,” in Conference on Neural Information Processing Systems (NIPS), December 5-10 (Barcelona, Spain, 4502– 4510.
  14. Bayraksan G. Morton D. P. (2006). Assessing Solution Quality in Stochastic Programs. Math. Program 108, 495– 514. 10.1007/s10107-006-0720-x · DOI
  15. Bengio Y. Louradour J. Collobert R. Weston J. (2009). “ Curriculum Learning,” in International Conference on Machine Learning (ICML), June 14-18 (Montreal, Quebec, Canada: ACM), 41– 48. of ACM International Conference Proceeding Series. 10.1145/1553374.1553380 382 · DOI · Paper page
  16. Bongard J. Zykov V. Lipson H. (2006). Resilient Machines through Continuous Self-Modeling. Science 314, 1118– 1121. 10.1126/science.1133687 17110570 · DOI · Paper page
  17. Bousmalis K. Irpan A. Wohlhart P. Bai Y. Kelcey M. Kalakrishnan M. (2018). “ Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping,” in International Conference on Robotics and Automation, May 21-25 (Brisbane, Australia: ICRA), 4243– 4250. 10.1109/ICRA.2018.8460875 · DOI · Paper page
  18. Brockman G. Cheung V. Pettersson L. Schneider J. Schulman J. Tang J. (2016). Openai Gym. arXiv 1606.01540 · arXiv
  19. Brogliato B. Maschke B. Lozano R. Egeland O. (2007). Dissipative Systems Analysis and Control. Theor. Appl. 2. 10.1007/978-1-84628-517-2 · DOI
  20. Brooks R. A. (1992). “ Artificial Life and Real Robots,” in European Conference on Artificial Life (ECAL), December 11-13 (Paris, France, 3– 10.
  21. Calandra R. Ivaldi S. Deisenroth M. P. Rueckert E. Peters J. (2015). “ Learning Inverse Dynamics Models with Contacts,” in International Conference on Robotics and Automation (ICRA), 26-30 May (Seattle, WA, USA: IEEE), 3186– 3191. 10.1109/ICRA.2015.7139638 · DOI
  22. Carlini N. Wagner D. (2017). “ Adversarial Examples Are Not Easily Detected,” in Workshop on Artificial Intelligence and Security (AISec), November 3 (Dallas, TX, USA: ACM), 3– 14. 10.1145/3128572.3140444 · DOI
  23. Chatzilygeroudis K. Mouret J.-B. (2018). “ Using Parameterized Black-Box Priors to Scale up Model-Based Policy Search for Robotics,” in International Conference on Robotics and Automation (ICRA), May 21-25 (Brisbane, Australia: IEEE), 1– 9. 10.1109/ICRA.2018.8461083 · DOI
  24. Chatzilygeroudis K. Rama R. Kaushik R. Goepp D. Vassiliades V. Mouret J.-B. (2017). “ Black-box Data-Efficient Policy Search for Robotics,” in International Conference on Intelligent Robots and Systems (IROS), September 24-28 (Vancouver, BC: CanadaIEEE), 51– 58. 10.1109/IROS.2017.8202137 · DOI
  25. Chatzilygeroudis K. Vassiliades V. Stulp F. Calinon S. Mouret J.-B. (2020). A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials. IEEE Trans. Robot. 36, 328– 347. 10.1109/TRO.2019.2958211 · DOI · Paper page
  26. Chebotar Y. Handa A. Makoviychuk V. Macklin M. Issac J. Ratliff N. (2019). “ Closing the Sim-To-Real Loop: Adapting Simulation Randomization with Real World Experience,” in International Conference on Robotics and Automation (ICRA), May 20-24 (Montreal, QC, Canada, 8973– 8979. 10.1109/ICRA.2019.8793789 · DOI
  27. Chen G. Choi W. Yu X. Han T. X. Chandraker M. (2017). “ Learning Efficient Object Detection Models with Knowledge Distillation,” in Conference on Neural Information Processing Systems (NIPS), December 4-9 (Long Beach, CA, USA, 742– 751.
  28. Chow Y. S. Robbins H. (1963). On Optimal Stopping Rules. Z. Wahrscheinlichkeitstheorie Verw Gebiete 2, 33– 49. 10.1007/bf00535296 · DOI
  29. Christiano P. F. Shah Z. Mordatch I. Schneider J. Blackwell T. Tobin J. (2016). Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model. arXiv 1610.03518 · arXiv
  30. Chung S.-J. Pollard N. (2016). Predictable Behavior during Contact Simulation: a Comparison of Selected Physics Engines. Comp. Anim. Virtual Worlds 27, 262– 270. 10.1002/cav.1712 · DOI
  31. Ciresan D. Meier U. Schmidhuber J. (2012). “ Multi-column Deep Neural Networks for Image Classification,” in Conference on Computer Vision and Pattern Recognition (CVPR), June 16-21 (RI, USA: IEEE Computer Society), 3642– 3649. 10.1109/CVPR.2012.6248110 · DOI
  32. Collins J. Howard D. Leitner J. (2019). “ Quantifying the Reality gap in Robotic Manipulation Tasks,” in International Conference on Robotics and Automation (ICRA), May 20-24 (Montreal, QC, Canada: IEEE), 6706– 6712. 10.1109/ICRA.2019.8793591 · DOI
  33. Cranmer K. Brehmer J. Louppe G. (2020). The Frontier of Simulation-Based Inference. Proc. Natl. Acad. Sci. USA 117, 30055– 30062. 10.1073/pnas.1912789117 32471948 PMC7720103 · DOI · Paper page
  34. Cui J. Kingsbury B. Ramabhadran B. Saon G. Sercu T. Audhkhasi K. (2017). “ Knowledge Distillation across Ensembles of Multilingual Models for Low-Resource Languages,” in Knowledge distillation across ensembles of multilingual models for low-resource languages, March 5-9 (ICASSP, New Orleans, LA, USA: IEEE), 4825– 4829. 10.1109/ICASSP.2017.7953073 · DOI
  35. Cully A. Clune J. Tarapore D. Mouret J.-B. (2015). Robots that Can Adapt like Animals. Nature 521, 503– 507. 10.1038/nature14422 26017452 · DOI · Paper page
  36. Cutler M. How J. P. (2016). “ Autonomous Drifting Using Simulation-Aided Reinforcement Learning,” in International Conference on Robotics and Automation (ICRA), May 16-21 (Stockholm, Sweden: IEEE), 5442– 5448. 10.1109/ICRA.2016.7487756 · DOI
  37. Cutler M. How J. P. (2015). “ Efficient Reinforcement Learning for Robots Using Informative Simulated Priors,” in International Conference on Robotics and Automation (ICRA), 26-30 May (Seattle, WA, USA: IEEE), 2605– 2612. 10.1109/ICRA.2015.7139550 · DOI
  38. Czarnecki W. M. Pascanu R. Osindero S. Jayakumar S. M. Swirszcz G. Jaderberg M. (2019). “ Distilling Policy Distillation,” in International Conference on Artificial Intelligence and Statistics (AISTATS) (Naha, Okinawa, Japan16-18 April: PMLR), 1331– 1340. of Proceedings of Machine Learning Research. 89
  39. Dai H. Landry B. Yang L. Pavone M. Tedrake R. (2021). “ Lyapunov-stable Neural-Network Control,” in Robotics: Science and Systems (RSS), July 12-16. Virtual Event. 10.15607/RSS.2021.XVII.063 · DOI
  40. Dai T. Arulkumaran K. Tukra S. Behbahani F. Bharath A. A. (2019). Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation. arXiv 1912. 08324
  41. Degrave J. Hermans M. Dambre J. Wyffels F. (2019). A Differentiable Physics Engine for Deep Learning in Robotics. Front. Neurorobot. 13, 6. 10.3389/fnbot.2019.00006 30899218 PMC6416213 · DOI · Paper page
  42. Deisenroth M. P. Neumann G. Peters J. (2011). A Survey on Policy Search for Robotics. FNT in Robotics 2, 1– 142. 10.1561/2300000021 · DOI
  43. Deisenroth M. P. Rasmussen C. E. (2011). “ PILCO: a Model-Based and Data-Efficient Approach to Policy Search,” in International Conference on Machine Learning (ICML), June 28 - July 2 (Bellevue, Washington, USA, 465– 472.
  44. Delage E. Ye Y. (2010). Distributionally Robust Optimization under Moment Uncertainty with Application to Data-Driven Problems. Operations Res. 58, 595– 612. 10.1287/opre.1090.0741 · DOI
  45. Dennett D. C. (1975). Why the Law of Effect Will Not Go Away. J. Theor. Soc. Behav. 10.1111/j.1468-5914.1975.tb00350.x · DOI
  46. Du Y. Watkins O. Darrell T. Abbeel P. Pathak D. (2021). Auto-tuned Sim-To-Real Transfer. arXiv 2104.07662. · arXiv
  47. Durkan C. Murray I. Papamakarios G. (2020). “ On Contrastive Learning for Likelihood-free Inference,” in International Conference on Machine Learning (ICML), July 13-18 (PMLR), 2771– 2781. Virtual Eventof Proceedings of Machine Learning Research. 119
  48. Erez T. Smart W. D. (2008). “ What Does Shaping Mean for Computational Reinforcement Learning?,” in International Conference on Development and Learning (ICDL) (Monterey, CA, USA: IEEE), 215– 219.
  49. Erez T. Tassa Y. Todorov E. (2015). “ Simulation Tools for Model-Based Robotics: Comparison of Bullet, Havok, Mujoco, ODE and Physx,” in International Conference on Robotics and Automation (ICRA), May 26-30 (Seattle, WA, USA, 4397– 4404. 10.1109/ICRA.2015.7139807 · DOI
  50. Fawzi A. Fawzi O. Frossard P. (2015). “ Fundamental Limits on Adversarial Robustness,” in International Conference on Machine Learning (ICML), Workshop on Deep Learning.
  51. Fearnhead P. Prangle D. (2012). Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic Approximate Bayesian Computation. J. R. Stat. Soc. 74, 419– 474. 10.1111/j.1467-9868.2011.01010.x · DOI
  52. Finn C. Abbeel P. Levine S. (2017). “ Model-agnostic Meta-Learning for Fast Adaptation of Deep Networks,” in International Conference on Machine Learning, 6-11 August (Sydney, NSW, Australia: ICML), 1126– 1135.
  53. Forrester A. I. J. Sóbester A. Keane A. J. (2007). Multi-fidelity Optimization via Surrogate Modelling. Proc. R. Soc. A. 463, 3251– 3269. 10.1098/rspa.2007.1900 · DOI
  54. Grant E. Finn C. Levine S. Darrell T. Griffiths T. L. (2018). “ Recasting Gradient-Based Meta-Learning as Hierarchical Bayes,” in International Conference on Learning Representations (ICLR), April 30 - May 3, 2018 (Vancouver, BC, Canada. Conference Track (OpenReview.net).
  55. Greenberg D. S. Nonnenmacher M. Macke J. H. (2019). “ Automatic Posterior Transformation for Likelihood-free Inference,” in International Conference on Machine Learning (ICML), 9-15 June (Long Beach, California, USA: PMLR), 2404– 2414. of Proceedings of Machine Learning Research. 97
  56. Greydanus S. Dzamba M. Yosinski J. (2019). “ Hamiltonian Neural Networks,” in Conference on Neural Information Processing Systems (NeurIPS), December 8-14 (Vancouver, BC, Canada, 15353– 15363.
  57. Höfer S. Bekris K. E. Handa A. Higuera J. C. G. Golemo F. Mozifian M. (2020). Perspectives on Sim2real Transfer for Robotics: A Summary of the R:SS 2020 Workshop. arXiv 2012.03806. · arXiv
  58. Haefner B. Queau Y. Mollenhoff T. Cremers D. (2018). “ Fight Ill-Posedness with Ill-Posedness: Single-Shot Variational Depth Super-resolution from Shading,” in Conference on Computer Vision and Pattern Recognition (CVPR), June 18-22 (Salt Lake City, UT, USA: IEEE Computer Society), 164– 174. 10.1109/CVPR.2018.00025 · DOI
  59. Hanna J. P. Stone P. (2017). “ Grounded Action Transformation for Robot Learning in Simulation,” in AAAI Conference on Artificial Intelligence, February 4-9 (San Francisco, California, USA, 3834– 3840.
  60. Heiden E. Millard D. Coumans E. Sheng Y. Sukhatme G. S. (2021). “ NeuralSim: Augmenting Differentiable Simulators with Neural Networks,” in International Conference on Robotics and Automation (ICRA), May 30 - June 5 (Xi’an, China. 10.1109/icra48506.2021.9560935 · DOI
  61. Hermans J. Begy V. Louppe G. (2020)., 119. PMLR, 4239– 4248. of Proceedings of Machine Learning Research. Likelihood-free MCMC with Amortized Approximate Ratio Estimators International Conference on Machine Learning (ICML), Virtual Event 13-18 July
  62. Hinton G. E. Vinyals O. Dean J. (2015). Distilling the Knowledge in a Neural Network. arXiv 1503.02531. · arXiv
  63. Hu Y. Liu J. Spielberg A. Tenenbaum J. B. Freeman W. T. Wu J. (2019). “ Chainqueen: A Real-Time Differentiable Physical Simulator for Soft Robotics,” in International Conference on Robotics and Automation (ICRA), May 20-24 (Montreal, QC, Canada: IEEE), 6265– 6271. 10.1109/ICRA.2019.8794333 · DOI
  64. Huang S. H. Papernot N. Goodfellow I. J. Duan Y. Abbeel P. (2017). “ Adversarial Attacks on Neural Network Policies, Workshop Track,” in International Conference on Learning Representations (ICLR) Toulon, April 24-26 (France. OpenReview.net).
  65. Ilyas A. Santurkar S. Tsipras D. Engstrom L. Tran B. Madry A. (2019). “ Adversarial Examples Are Not Bugs, They Are Features,” in Conference on Neural Information Processing Systems (NeurIPS), December 8-14 (Vancouver, BC, Canada, 125– 136.
  66. Ivaldi S. Peters J. Padois V. Nori F. (2014). “ Tools for Simulating Humanoid Robot Dynamics: A Survey Based on User Feedback,” in Tools for simulating humanoid robot dynamics: A survey based on user feedback, November 18-20 (Humanoids, Madrid, Spain: IEEE), 842– 849. 10.1109/HUMANOIDS.2014.7041462 · DOI
  67. Jakobi N. (1997). Evolutionary Robotics and the Radical Envelope-Of-Noise Hypothesis. Adaptive Behav. 6, 325– 368. 10.1177/105971239700600205 · DOI
  68. Jakobi N. Husbands P. Harvey I. (1995). “ Noise and the Reality gap: The Use of Simulation in Evolutionary Robotics,” in Advances in Artificial Life, June 4-6 (Granada, Spain, 704– 720. 704–720. 10.1007/3-540-59496-5_337 · DOI · Paper page
  69. James S. Davison A. J. Johns E. (2017). “ Transferring End-To-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task,” in Conference on Robot Learning (CoRL), November 13-15 (Mountain View, California, USA: PMLR), 334– 343. of Proceedings of Machine Learning Research. 78
  70. Jiang Y. Zhang T. Ho D. Bai Y. Liu C. K. Levine S. (2021). Simgan: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning. arXiv 2101.06005 · arXiv
  71. Körber M. Lange J. Rediske S. Steinmann S. Glück R. (2021). Comparing Popular Simulation Environments in the Scope of Robotics and Reinforcement Learning. arXiv 2103. 04616
  72. Kadian A. Truong J. Gokaslan A. Clegg A. Wijmans E. Lee S. (2020). Sim2real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? IEEE Robot. Autom. Lett. 5, 6670– 6677. 10.1109/LRA.2020.3013848 · DOI · Paper page
  73. Kahn H. Marshall A. W. (1953). Methods of Reducing Sample Size in Monte Carlo Computations. Or 1, 263– 278. 10.1287/opre.1.5.263 · DOI
  74. Kakade S. M. (2001). VancouverBritish Columbia, Canada, 1531– 1538. A Natural Policy Gradient Conference on Neural Information Processing Systems (NIPS) December 3-8.
  75. Kaspar M. Munoz Osorio J. D. Bock J. (2020). “ Sim2real Transfer for Reinforcement Learning without Dynamics Randomization,” in International Conference on Intelligent Robots and Systems (IROS), October 24 - January 24 (Las Vegas, NV, USA: IEEE), 4383– 4388. 10.1109/IROS45743.2020.9341260 · DOI · Paper page
  76. Kishimoto K. Amari S. (1979). Existence and Stability of Local Excitations in Homogeneous Neural fields. J. Math. Biol. 7, 303– 318. 10.1007/bf00275151 224127 · DOI
  77. Klink P. Abdulsamad H. Belousov B. D’Eramo C. Peters J. Pajarinen J. (2021). A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning, 13176. arXiv 2102.
  78. Klink P. Abdulsamad H. Belousov B. Peters J. (2019). “ Self-paced Contextual Reinforcement Learning,” in Conference on Robot Learning (CoRL), October 30 - November 1 (Osaka, Japan: PMLR), 513– 529. of Proceedings of Machine Learning Research. 100.
  79. Kolev K. Klodt M. Brox T. Cremers D. (2009). Continuous Global Optimization in Multiview 3d Reconstruction. Int. J. Comput. Vis. 84, 80– 96. 10.1007/s11263-009-0233-1 · DOI
  80. Koos S. Mouret J.-B. Doncieux S. (2010). “ Crossing the Reality gap in Evolutionary Robotics by Promoting Transferable Controllers,” in Genetic and Evolutionary Computation Conference (GECCO), July 7-11 (Portland, Oregon, USA: ACM), 119– 126. 10.1145/1830483.1830505 · DOI
  81. Koos S. Mouret J.-B. Doncieux S. (2013). The Transferability Approach: Crossing the Reality gap in Evolutionary Robotics. IEEE Trans. Evol. Computat. 17, 122– 145. 10.1109/TEVC.2012.2185849 · DOI
  82. Krizhevsky A. Sutskever I. Hinton G. E. (2012). “ Imagenet Classification with Deep Convolutional Neural Networks,” in Conference on Neural Information Processing Systems (NIPS), 1106– 1114. Lake Tahoe, Nev. United States December 3-6
  83. Krotkov E. Hackett D. Jackel L. Perschbacher M. Pippine J. Strauss J. (2017). The DARPA Robotics challenge Finals: Results and Perspectives. J. Field Robotics 34, 229– 240. 10.1002/rob.21683 · DOI · Paper page
  84. Kumar A. Fu Z. Pathak D. Malik J. (2021). “ RMA: Rapid Motor Adaptation for Legged Robots,” in Robotics: Science and Systems (RSS), Virtual Event, July 12-16. 10.15607/RSS.2021.XVII.011 · DOI
  85. Kumar M. P. Packer B. Koller D. (2010). “ Self-paced Learning for Latent Variable Models,” in Conference on Neural Information Processing Systems (NIPS), 6-9 December (Vancouver, British Columbia, Canada, 1189– 1197.
  86. Kurakin A. Goodfellow I. J. Bengio S. (2017). “ Adversarial Examples in the Physical World,” in International Conference on Learning Representations (ICLR) Toulon, April 24-26 (France. Workshop Track (OpenReview.net).
  87. Landau I. D. Lozano R. M’Saad M. Karimi A. (2011). Adaptive Control: Algorithms, Analysis and Applications. 2 edn. Springer Science & Business Media.
  88. Levine S. Koltun V. (2013). “ Variational Policy Search via Trajectory Optimization,” in Conference on Neural Information Processing Systems (NIPS), December 5-8 (Lake Tahoe, Nevada, USA, 207– 215.
  89. Levine S. Pastor P. Krizhevsky A. Ibarz J. Quillen D. (2018). Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. Int. J. Robotics Res. 37, 421– 436. 10.1177/0278364917710318 · DOI · Paper page
  90. Lillicrap T. P. Hunt J. J. Pritzel A. Heess N. Erez T. Tassa Y. (2016). “ Continuous Control with Deep Reinforcement Learning,” in International Conference on Learning Representations (ICLR), May 2-4 (San Juan, Puerto Rico. Conference Track (OpenReview.net).
  91. Liu Y. Ramachandran P. Liu Q. Peng J. (2017). “ Stein Variational Policy Gradient,” in Association for Uncertainty in Artificial Intelligence (UAI) (Sydney, Australia, August 11– 15.
  92. Lowrey K. Kolev S. Dao J. Rajeswaran A. Todorov E. (2018). “ Reinforcement Learning for Non-prehensile Manipulation: Transfer from Simulation to Physical System,” in Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), May 16-19 (Brisbane, Australia, 35– 42. 10.1109/SIMPAR.2018.8376268 · DOI
  93. Lueckmann J. Gonçalves P. J. Bassetto G. Öcal K. Nonnenmacher M. Macke J. H. (2017). “ Flexible Statistical Inference for Mechanistic Models of Neural Dynamics,” in Conference on Neural Information Processing Systems, December 4-9 (Long Beach, CA, USA: NIPS), 1289– 1299.
  94. Luksch T. Gienger M. Mühlig M. Yoshiike T. (2012). “ Adaptive Movement Sequences and Predictive Decisions Based on Hierarchical Dynamical Systems,” in International Conference on Intelligent Robots and Systems (IROS), October 7-12 (Vilamoura, Algarve, Portugal: IEEE), 2082– 2088. 10.1109/iros.2012.6385651 · DOI
  95. Lutter M. Mannor S. Peters J. Fox D. Garg A. (2021a). Robust Value Iteration for Continuous Control Tasks. arXiv 2105.12189. 36260585 · arXiv
  96. Lutter M. Ritter C. Peters J. (2019). “ Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning,” in International Conference on Learning Representations (ICLR), May 6-9 (New Orleans, LA, USA. Conference Track (OpenReview.net).
  97. Lutter M. Silberbauer J. Watson J. Peters J. (2021b). Differentiable Physics Models for Real-World Offline Model-Based Reinforcement Learning. arXiv 2011.01734 · arXiv
  98. Mak W.-K. Morton D. P. Wood R. K. (1999). Monte Carlo Bounding Techniques for Determining Solution Quality in Stochastic Programs. Operations Res. Lett. 24, 47– 56. 10.1016/S0167-6377(98)00054-6 · DOI
  99. Mandlekar A. Zhu Y. Garg A. Fei-Fei L. Savarese S. (2017). Vancouver, BC: Canada. September 24-28. 3932–3939. 10.1109/IROS.2017.8206245 Adversarially Robust Policy Learning: Active Construction of Physically-Plausible Perturbations International Conference on Intelligent Robots and Systems (IROS) · DOI · Paper page
  100. Marco A. Berkenkamp F. Hennig P. Schoellig A. P. Krause A. Schaal S. (2017). “ Virtual vs. Real: Trading off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization,” in International Conference on Robotics and Automation (ICRA), May 29 - Jun 3 (Marina Bay Sands, Singapore. 10.1109/icra.2017.7989186 · DOI
  101. Marjoram P. Molitor J. Plagnol V. Tavare S. (2003). Markov Chain Monte Carlo without Likelihoods. Proc. Natl. Acad. Sci. 100, 15324– 15328. 10.1073/pnas.0306899100 14663152 PMC307566 · DOI · Paper page
  102. Martin Martin R. Brock O. (2014). “ Online Interactive Perception of Articulated Objects with Multi-Level Recursive Estimation Based on Task-specific Priors,” in International Conference on Intelligent Robots and Systems (IROS), September 14-18 (Chicago, IL, USA: IEEE), 2494– 2501. 10.1109/IROS.2014.6942902 · DOI · Paper page
  103. Matas J. James S. Davison A. J. (2018). “ Sim-to-real Reinforcement Learning for Deformable Object Manipulation,” in Conference on Robot Learning (CoRL), October 29-31 (Zürich, Switzerland: PMLR), 734– 743. of Proceedings of Machine Learning Research. 87
  104. Matl C. Narang Y. Bajcsy R. Ramos F. Fox D. (2020). “ Inferring the Material Properties of Granular media for Robotic Tasks,” in International Conference on Robotics and Automation (ICRA) (Paris, FranceMay 31 - August 31: IEEE), 2770– 2777. 10.1109/ICRA40945.2020.9197063 · DOI
  105. Mehta B. Diaz M. Golemo F. Pal C. J. Paull L. (2019). “ Active Domain Randomization,” in Conference on Robot Learning (CoRL), October 30 - November 1 (Osaka, Japan: PMLR), 1162– 1176. of Proceedings of Machine Learning Research. 100
  106. Mehta B. Handa A. Fox D. Ramos F. (2020). “ A User‘s Guide to Calibrating Robotics Simulators,” in Conference on Robot Learning (CoRL), Virtual Event, November 16 - 18 (PMLR). Proceedings of Machine Learning Research.
  107. Mnih V. Kavukcuoglu K. Silver D. Rusu A. A. Veness J. Bellemare M. G. (2015). Human-level Control through Deep Reinforcement Learning. Nature 518, 529– 533. 10.1038/nature14236 25719670 · DOI · Paper page
  108. Molchanov A. Chen T. Honig W. Preiss J. A. Ayanian N. Sukhatme G. S. (2019). “ Sim-to-(multi)-real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors,” in International Conference on Intelligent Robots and Systems (IROS), November 3-8 (Macau, SAR, China: IEEE), 59– 66. 10.1109/IROS40897.2019.8967695 · DOI
  109. Mordatch I. Lowrey K. Todorov E. (2015). “ Ensemble-cio: Full-Body Dynamic Motion Planning that Transfers to Physical Humanoids,” in International Conference on Intelligent Robots and Systems (IROS), September 28 - October 2 (Hamburg, Germany, 5307– 5314. 10.1109/IROS.2015.7354126 · DOI · Paper page
  110. Morere P. Ott L. Ramos F. (2019). Learning to Plan Hierarchically from Curriculum. IEEE Robot. Autom. Lett. 4, 2815– 2822. 10.1109/LRA.2019.2920285 · DOI
  111. Mozian M. Camilo Gamboa Higuera J. Meger D. Dudek G. (2020). “ Learning Domain Randomization Distributions for Training Robust Locomotion Policies,” in International Conference on Intelligent Robots and Systems (IROS) Las Vegas, October 24 - January 24 (NV, USA: IEEE), 6112– 6117. 10.1109/IROS45743.2020.9341019 · DOI
  112. Muratore F. Eilers C. Gienger M. Peters J. (2021a). Data-efficient Domain Randomization with Bayesian Optimization. IEEE Robot. Autom. Lett. 6, 911– 918. 10.1109/LRA.2021.3052391 · DOI · Paper page
  113. Muratore F. Gienger M. Peters J. (2021b). Assessing Transferability from Simulation to Reality for Reinforcement Learning. IEEE Trans. Pattern Anal. Mach. Intell. 43, 1172– 1183. 10.1109/TPAMI.2019.2952353 31722475 · DOI
  114. Muratore F. Gruner T. Wiese F. Gienger B. B. M. Peters J. (2021c). “ Neural Posterior Domain Randomization,” in Conference on Robot Learning (CoRL), Virtual Event, November 8-11 (London, England.
  115. Muratore F. Treede F. Gienger M. Peters J. (2018). “ Domain Randomization for Simulation-Based Policy Optimization with Transferability Assessment,” in Conference on Robot Learning (CoRL) (Zürich, SwitzerlandOctober 29-31: PMLR), 700– 713. of Proceedings of Machine Learning Research. 87
  116. Ng A. Y. Jordan M. I. (2000). “ PEGASUS: a Policy Search Method for Large Mdps and Pomdps,” in UAI, June 30 - July 3 (Stanford, California, USA: Morgan Kaufmann), 406– 415.
  117. OpenAI Akkaya I. Andrychowicz M. Chociej M. Litwin M. McGrew B. (2019). Solving Rubik’s Cube with a Robot Hand. arXiv 1910.07113 · arXiv
  118. Pan S. J. Yang Q. (2010). A Survey on Transfer Learning. IEEE Trans. Knowl. Data Eng. 22, 1345– 1359. 10.1109/TKDE.2009.191 · DOI · Paper page
  119. Papamakarios G. Murray I. (2016). “ Fast ϵ-free Inference of Simulation Models with Bayesian Conditional Density Estimation,” in Conference on Neural Information Processing Systems (NIPS), December 5-10 (Barcelona, Spain, 1028– 1036.
  120. Papamakarios G. Sterratt D. C. Murray I. (2019). “ Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows,” in International Conference on Artificial Intelligence and Statistics (AISTATS), April 16-18 (Naha, Okinawa, Japan: PMLR), 837– 848. of Proceedings of Machine Learning Research. 89
  121. Parisotto E. Ba L. J. Salakhutdinov R. (2016). “ Actor-mimic: Deep Multitask and Transfer Reinforcement Learning,” in International Conference on Learning Representations (ICLR) San Juan, May 2-4 (Puerto Rico. Conference Track.
  122. Paul S. Osborne M. A. Whiteson S. (2019). Fingerprint Policy Optimisation for Robust Reinforcement Learning. In International Conference on Machine Learning (ICML), Long Beach California, USA, 9-15 June (PMLR), vol. 97
  123. Peng X. B. Andrychowicz M. Zaremba W. Abbeel P. (2018). “ Sim-to-real Transfer of Robotic Control with Dynamics Randomization,” in International Conference on Robotics and Automation (ICRA), May 21-25 (Brisbane, Australia, 1– 8. 10.1109/ICRA.2018.8460528 · DOI · Paper page
  124. Perlin K. (2002). Improving Noise. ACM Trans. Graph. 21, 681– 682. 10.1145/566654.566636 · DOI
  125. Peters J. Mülling K. Altun Y. (2010). “ Relative Entropy Policy Search,” in AAAI Conference on Artificial Intelligence, July 11-15 (Atlanta, Georgia, USA.
  126. Pinto L. Andrychowicz M. Welinder P. Zaremba W. Abbeel P. (2018). “ Asymmetric Actor Critic for Image-Based Robot Learning,” in Robotics: Science and Systems (RSS), June 26-30 (Pittsburgh, Pennsylvania, USA. 10.15607/RSS.2018.XIV.008 · DOI
  127. Polvara R. Patacchiola M. Hanheide M. Neumann G. (2020). Sim-to-real Quadrotor landing via Sequential Deep Q-Networks and Domain Randomization. Robotics 9, 8. 10.3390/robotics9010008 · DOI
  128. Possas R. Barcelos L. Oliveira R. Fox D. Ramos F. (2020). “ Online Bayessim for Combined Simulator Parameter Inference and Policy Improvement,” in International Conference on Intelligent Robots and Systems (IROS) Las Vegas, October 24 - January 24 (NV, USA: IEEE), 5445– 5452. 10.1109/IROS45743.2020.9341401 · DOI
  129. Rajeswaran A. Ghotra S. Ravindran B. Levine S. (2017). “ Epopt: Learning Robust Neural Network Policies Using Model Ensembles,” in International Conference on Learning Representations (ICLR), Toulon, April 24-26 (France. Conference Track (OpenReview.net).
  130. Rajeswaran A. Mordatch I. Kumar V. (2020). “ A Game Theoretic Framework for Model Based Reinforcement Learning,” in International Conference on Machine Learning (ICML), Virtual Event, 13-18 July (PMLR), 7953– 7963. of Proceedings of Machine Learning Research. 119
  131. Ramos F. Possas R. Fox D. (2019). “ Bayessim: Adaptive Domain Randomization via Probabilistic Inference for Robotics Simulators,” in Robotics: Science and Systems (RSS), June 22-26 (Germany: Freiburg im Breisgau). 10.15607/RSS.2019.XV.029 · DOI
  132. Rasmussen C. E. Williams C. K. I. (2006). Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press.
  133. Rawlik K. Toussaint M. Vijayakumar S. (2012). Sydney, NSW, Australia: RSS. 10.15607/RSS.2012.VIII.045 On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference Robotics: Science and Systems July 9-13 · DOI
  134. Ruiz N. Schulter S. Chandraker M. (2019). “ Learning to Simulate,” in International Conference on Learning Representations (ICLR), May 6-9 (New Orleans, LA, USA. (OpenReview.net).
  135. Russakovsky O. Deng J. Su H. Krause J. Satheesh S. Ma S. (2015). Imagenet Large Scale Visual Recognition challenge. Int. J. Comput. Vis. 115, 211– 252. 10.1007/s11263-015-0816-y · DOI · Paper page
  136. Rusu A. A. Colmenarejo S. G. Gülçehre Ç. Desjardins G. Kirkpatrick J. Pascanu R. (2016a). “ Policy Distillation,” in (San Juan, Puerto Rico. Conference Track. International Conference on Learning Representations (ICLR) May 2-4
  137. Rusu A. A. Rabinowitz N. C. Desjardins G. Soyer H. Kirkpatrick J. Kavukcuoglu K. (2016b). Progressive Neural Networks. arXiv 1606.04671 · arXiv
  138. Sadeghi F. Levine S. (2017). “ CAD2RL: Real Single-Image Flight without a Single Real Image,” in Robotics: Science and Systems (RSS), July 12-16 (Cambridge, Massachusetts, USA. 10.15607/RSS.2017.XIII.034 · DOI · Paper page
  139. Santoro A. Bartunov S. Botvinick M. Wierstra D. Lillicrap T. P. (2016). “ Meta-learning with Memory-Augmented Neural Networks,” in International Conference on Machine Learning (ICML), June 19-24 (New York City, NY, USA: JMLR.org), 1842– 1850. 48
  140. Siekmann J. Green K. Warila J. Fern A. Hurst J. (2021). “ Blind Bipedal Stair Traversal via Sim-To-Real Reinforcement Learning,” in Robotics: Science and Systems (RSS), Virtual Event, July 12-16. 10.15607/RSS.2021.XVII.061 · DOI · Paper page
  141. Silver D. Huang A. Maddison C. J. Guez A. Sifre L. van den Driessche G. (2016). Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature 529, 484– 489. 10.1038/nature16961 26819042 · DOI · Paper page
  142. Sunnåker M. Busetto A. G. Numminen E. Corander J. Foll M. Dessimoz C. (2013). Approximate Bayesian Computation. Plos Comput. Biol. 9, e1002803. 10.1371/journal.pcbi.1002803 23341757 PMC3547661 · DOI
  143. Sutanto G. Wang A. S. Lin Y. Mukadam M. Sukhatme G. S. Rai A. (2020). “ Encoding Physical Constraints in Differentiable newton-euler Algorithm,” in L4DC, Virtual Event, 11-12 June (Berkeley, CA, USA: PMLR), 804– 813. of Proceedings of Machine Learning Research. 120
  144. Szegedy C. Zaremba W. Sutskever I. Bruna J. Erhan D. Goodfellow I. J. (2014). “ Intriguing Properties of Neural Networks,” in (Banff, Canada. Conference Track. International Conference on Learning Representations (ICLR) April 14-16
  145. Tan J. Zhang T. Coumans E. Iscen A. Bai Y. Hafner D. (2018). “ Sim-to-real: Learning Agile Locomotion for Quadruped Robots,” in Robotics: Science and Systems (RSS), June 26-30 (Pittsburgh, Pennsylvania, USA. 10.15607/RSS.2018.XIV.010 · DOI · Paper page
  146. Teh Y. W. Bapst V. Czarnecki W. M. Quan J. Kirkpatrick J. Hadsell R. (2017). “ Distral: Robust Multitask Reinforcement Learning,” in Conference on Neural Information Processing Systems (NIPS) (Long Beach, CA, USA, 4496– 4506.
  147. Ting J.-A. D’Souza A. Schaal S. (2011). Bayesian Robot System Identification with Input and Output Noise. Neural Networks 24, 99– 108. 10.1016/j.neunet.2010.08.011 20863655 · DOI
  148. Ting J. Mistry M. Peters J. Schaal S. Nakanishi J. (2006). “ A Bayesian Approach to Nonlinear Parameter Identification for Rigid Body Dynamics,” in Robotics: Science and Systems (RSS), August 16-19 (Philadelphia, Pennsylvania, USA: The MIT Press). 10.15607/RSS.2006.II.032 · DOI
  149. Van Parys B. Kuhn D. Goulart P. Morari M. (2015). Distributionally Robust Control of Constrained Stochastic Systems. IEEE Trans. Automat. Contr. 61, 1. 10.1109/TAC.2015.2444134 · DOI
  150. Wang J. M. Fleet D. J. Hertzmann A. (2010). Optimizing Walking Controllers for Uncertain Inputs and Environments. ACM Trans. Graphics 29, 73– 78. 10.1145/1833351.177881010.1145/1778765.1778810 · DOI
  151. Watson J. Abdulsamad H. Findeisen R. Peters J. (2021). Stochastic Control through Approximate Bayesian Input Inference. arXiv 2105. 07693
  152. Williams R. J. (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Mach Learn. 8, 229– 256. 10.1007/BF00992696 · DOI
  153. Wittenmark B. (1995). “ Adaptive Dual Control Methods: An Overview,” in Adaptive Systems in Control and Signal Processing 1995 (Elsevier), 67– 72. 10.1016/b978-0-08-042375-3.50010-x · DOI
  154. Wright R. D. Ramsay T. E. (1979). On the Effectiveness of Common Random Numbers. Manag. Sci. 25, 649– 656. 10.1287/mnsc.25.7.649 · DOI
  155. Wu J. Yildirim I. Lim J. J. Freeman B. Tenenbaum J. B. (2015). “ Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning,” in Conference on Neural Information Processing Systems (NIPS), December 7-12 (Montreal, Quebec, Canada, 127– 135.
  156. Xie Z. Clary P. Dao J. Morais P. Hurst J. W. van de Panne M. (2019). “ Learning Locomotion Skills for Cassie: Iterative Design and Sim-To-Real,” in Conference on Robot Learning (CoRL), October 30 - November 1 (Osaka, Japan: PMLR), 317– 329. of Proceedings of Machine Learning Research. 100
  157. Xie Z. Da X. van de Panne M. Babich B. Garg A. (2020). Dynamics Randomization Revisited: A Case Study for Quadrupedal Locomotion. arXiv 2011.02404 · arXiv
  158. Yan W. Vangipuram A. Abbeel P. Pinto L. (2020). “ Learning Predictive Representations for Deformable Objects Using Contrastive Estimation,” in Conference on Robot Learning (CoRL), Virtual Event, November 16 - 18 (Virtual Event/Cambridge, MA, USA: PMLR), 564– 574. of Proceedings of Machine Learning Research. 155.
  159. Yang C. Sun J. Zhang Q. Ma X. (2013). Lyapunov Stability and strong Passivity Analysis for Nonlinear Descriptor Systems. IEEE Trans. Circuits Syst. 60, 1003– 1012. 10.1109/TCSI.2012.2215396 · DOI
  160. Yu W. Kumar V. C. Turk G. Liu C. K. (2019a). “ Sim-to-real Transfer for Biped Locomotion,” in International Conference on Intelligent Robots and Systems (IROS), November 3-8 (Macau, SAR, China: IEEE), 3503– 3510. 10.1109/IROS40897.2019.8968053 · DOI
  161. Yu W. Liu C. K. Turk G. (2019b). “ Policy Transfer with Strategy Optimization,” in International Conference on Learning Representations (ICLR), May 6-9 (New Orleans, LA, USA. Conference Track (OpenReview.net).
  162. Zhang H. Chen H. Boning D. S. Hsieh C. (2021). “ Robust Reinforcement Learning on State Observations with Learned Optimal Adversary,” in International Conference on Learning Representations (ICLR), Virtual Event, May 3-7 (Austria. OpenReview.net).
  163. Zhang L. M. Plappert M. Zaremba W. (2020). Predicting Sim-To-Real Transfer with Probabilistic Dynamics Models, 12864. arXiv 2009.
  164. Zhen J. Kuhn D. Wiesemann W. (2021). Mathematical Foundations of Robust and Distributionally Robust Optimization. arXiv 2105.00760 · arXiv
  165. Zhuang F. Qi Z. Duan K. Xi D. Zhu Y. Zhu H. (2021). A Comprehensive Survey on Transfer Learning. Proc. IEEE 109, 43– 76. 10.1109/JPROC.2020.3004555 · DOI · Paper page

Cited by

Sorted by citations

Authors
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J.
Affiliations
  • Technische Universität Darmstadt (DE)
  • The University of Sydney (AU)
  • Georgia Institute of Technology (US)
  • Google (United States) (US)
  • Honda (Germany) (DE)
Corresponding author
Muratore, Fabio
ORCID
Venue
Frontiers in Robotics and AI
Publisher
Frontiers Media SA
Published
2022-04-11
Volume
9
Pages
799893-799893
ISSN
2296-9144 (ISSN-L)
DOI
10.3389/frobt.2022.799893
PMC
PMC9038844
OpenAlex
W3212380917
Cited by
128
References
17
Citations by year
2026: 40 · 2025: 38 · 2024: 27 · 2023: 14 · 2022: 9
FWCI
1.53
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Robotics and AI (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Technischen Universität Darmstadt (acceptedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • TUbilio (Technical University of Darmstadt) (submittedVersion)
  • Technischen Universität Darmstadt (cc-by)
Funders
  • National Science Foundation (1514258)
  • European Commission (HORIZON2020)
  • H2020 European Research Council (640554)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
sim-to-real transfer, domain randomization, robot learning, reality gap, data-efficient learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review