CoChat

Domain randomization for transferring deep neural networks from simulation to the real world

Open access
Tobin, Josh, Fong, Rachel, Ray, Alex, Schneider, Jonas, Zaremba, Wojciech, Abbeel, Pieter · 2017 · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.

Abstract

Bridging the `reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to 1.5 cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.

Overview

CoChat Summary

TL;DR

This paper introduces domain randomization: training an object detector on simulated RGB images with randomized rendering, textures, lighting, camera pose, and distractors so that the real world appears as another variation. The authors show that a detector trained only in simulation can localize objects in the real world accurately enough to grasp them in clutter.

Key findings

  • Real-world object detectors trained only on simulated images localized objects to within about 1.5 cm on average, including with distractors and partial occlusions; per-object errors in Table I range from 0.6 ± 0.3 cm to 3.2 ± 5.8 cm depending on object and condition.
  • The pipeline enabled a Fetch robot to detect and pick up the target object in 38 out of 40 trials in increasingly cluttered scenes, and a detector trained on a Spam can succeeded in 9 of 10 trials with previously unseen YCB food distractors.
  • Ablations found sensitivity to number of training images, number of unique textures, presence of distractors in training, and camera randomization; random noise had negligible effect. Performance improved up to around 50,000 training samples, and degraded significantly with fewer than 1,000 textures.
  • ImageNet pretraining was not essential with enough data: random weight initialization achieved nearly the same real-world transfer performance as pretrained initialization, though pretraining helped in the low-data regime.

Methods

  • The authors trained a modified VGG-16 convolutional neural network to map a single 224×224 monocular webcam image to Cartesian coordinates of object(s) of interest, using simulated images rendered by MuJoCo.
  • Domain randomization varied number/shape of distractors, object positions and textures, table/floor/skybox/robot textures, camera position/orientation/field of view, number and properties of lights, and image noise. Textures were random RGB values, gradients, or checker patterns; 0–10 distractor objects were added per scene.
  • Training used stochastic gradient descent with Adam on an L2 loss, learning rates around 1e-4, and batch sizes of 25, 50, or 100. Evaluation used 480 webcam images of eight geometric objects at 70–105 cm from a fixed camera, with 60 labeled images per object: 20 alone, 20 with distractors, and 20 partially occluded.

Limitations

  • The paper states that the detectors still overfit simulated training data: simulated error was 0.3–0.5 cm versus about 1.5 cm in the real world, and no real-world validation could be done during training because no real data were assumed available.
  • The paper states future work is needed for contact-rich manipulation or higher precision, and that the method fixes table height, effectively making the task 2D pose estimation from a single monocular uncalibrated camera.
  • Evident from the text: evaluation is limited to eight geometric objects plus one Spam can, the test camera position remains constant across all images, and lighting and surrounding scene were not controlled.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/iros.2017.8202133

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 8 Not valid · 7 Not checked · 32 Own results and interpretations

29 more claims

Create a free account to see all 49 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 11
  • It may state two separate claims as one. 8

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

  • Very Deep Convolutional Networks for Large-Scale Image Recognition

    Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×

    “In particular, we use a modified version the VGG-16 architecture [42] shown in Figure 2.”
    Method
  • Adam: A Method for Stochastic Optimization

    Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×

    “We found that using a learning rate of around 1e − 4 (as opposed to the standard 1e − 3 for Adam) improved convergence and helped avoid a common local optimum, mapping all objects to the center of the table.”
    Method
  • Deep learning

    LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126

  • Human-level control through deep reinforcement learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 3×

    “Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [27] and robotic control [21], [38].”
    BackgroundResult
  • How transferable are features in deep neural networks?

    Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×

    “A variety of approaches have been proposed, including re-training the model in the target domain (e.g., [52]), adapting the weights of the model based on the statistics of the source and target domains (e.g., [22]), learning invariant features between domains (e.g., [47]), and learning a mapping…”
    Background
  • MuJoCo: A physics engine for model-based control

    Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 1×

    “We render images using the MuJoCo Physics Engine’s [45] built-in renderer.”
    Method
  • Trust region policy optimization

    Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · cited 1×

    “Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [29] and robotic control [26], [41].”
    ResultSelf-citation
  • Learning Transferable Features with Deep Adaptation Networks

    Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505

  • End-to-End Training of Deep Visuomotor Policies

    Levine, S., Finn, C., Darrell, T., et al. (2015) · Journal of machine learning research · doi:10.5555/2946645.2946684 · Cited by 3,987

  • Describing Textures in the Wild

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438 · cited 1×

    “To examine how our detectors would perform on objects with complex textures, we created simulated scenes in which the object of interest is given a texture from the Describable Textures Dataset (DTD) [4].”
    Method

Show all 71 references

Research works printed but not linked: 2

  1. wise, 2016, Fetch and freight: Standard platforms for service robot applications, Workshop on Autonomous Mobile Service Robots
  2. sucan, 0, MoveIt!

Cited by

See all 3,124

Metadata

Authors
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P.
Affiliations
  • University of California, Berkeley (US)
  • OpenAI (United States) (US)
  • International Computer Science Institute (US)
Venue
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Publisher
IEEE
Published
2017-09
Pages
23-30
ISSN
2153-0858 (ISSN-L)
OpenAlex
W2605102758
Cited by
3,124
References
69
Citations by year
2026: 413 · 2025: 433 · 2024: 336 · 2023: 403 · 2022: 374 · 2021: 422 · 2020: 352 · 2019: 256 · 2018: 106 · 2017: 25
FWCI
92.18
Type
conference paper
Language
en
Open access
Locations
  • IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
domain randomization, sim-to-real transfer, reality gap, object detection, object localization, robotic grasping, cluttered environments, distractor robustness, partial occlusion, robotic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Domain randomization for transferring deep neural networks from simulation to the real world. https://app.cochat.ai/papers/10.1109/iros.2017.8202133
The paper
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). https://doi.org/10.1109/iros.2017.8202133

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.