CoChat

Domain randomization for transferring deep neural networks from simulation to the real world

Open access
Tobin, Josh, Fong, Rachel, Ray, Alex, Schneider, Jonas, Zaremba, Wojciech, Abbeel, Pieter · 2017 · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.

Abstract

Bridging the `reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to 1.5 cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.

Overview

CoChat Summary

TL;DR

This paper introduces domain randomization: training an object detector on simulated RGB images with randomized rendering, textures, lighting, camera pose, and distractors so that the real world appears as another variation. The authors show that a detector trained only in simulation can localize objects in the real world accurately enough to grasp them in clutter.

Key findings

  • Real-world object detectors trained only on simulated images localized objects to within about 1.5 cm on average, including with distractors and partial occlusions; per-object errors in Table I range from 0.6 ± 0.3 cm to 3.2 ± 5.8 cm depending on object and condition.
  • The pipeline enabled a Fetch robot to detect and pick up the target object in 38 out of 40 trials in increasingly cluttered scenes, and a detector trained on a Spam can succeeded in 9 of 10 trials with previously unseen YCB food distractors.
  • Ablations found sensitivity to number of training images, number of unique textures, presence of distractors in training, and camera randomization; random noise had negligible effect. Performance improved up to around 50,000 training samples, and degraded significantly with fewer than 1,000 textures.
  • ImageNet pretraining was not essential with enough data: random weight initialization achieved nearly the same real-world transfer performance as pretrained initialization, though pretraining helped in the low-data regime.

Methods

  • The authors trained a modified VGG-16 convolutional neural network to map a single 224×224 monocular webcam image to Cartesian coordinates of object(s) of interest, using simulated images rendered by MuJoCo.
  • Domain randomization varied number/shape of distractors, object positions and textures, table/floor/skybox/robot textures, camera position/orientation/field of view, number and properties of lights, and image noise. Textures were random RGB values, gradients, or checker patterns; 0–10 distractor objects were added per scene.
  • Training used stochastic gradient descent with Adam on an L2 loss, learning rates around 1e-4, and batch sizes of 25, 50, or 100. Evaluation used 480 webcam images of eight geometric objects at 70–105 cm from a fixed camera, with 60 labeled images per object: 20 alone, 20 with distractors, and 20 partially occluded.

Limitations

  • The paper states that the detectors still overfit simulated training data: simulated error was 0.3–0.5 cm versus about 1.5 cm in the real world, and no real-world validation could be done during training because no real data were assumed available.
  • The paper states future work is needed for contact-rich manipulation or higher precision, and that the method fixes table height, effectively making the task 2D pose estimation from a single monocular uncalibrated camera.
  • Evident from the text: evaluation is limited to eight geometric objects plus one Spam can, the test camera position remains constant across all images, and lighting and surrounding scene were not controlled.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/iros.2017.8202133

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Illustration of our approach. An object detector is trained on hundreds of thousands of low-fidelity rendered images with random camera positions, lighting conditions, object positions, and non-realistic textures. At test time, the same detector is used in the real world with no additional training.
Fig. 1 Illustration of our approach. An object detector is trained on hundreds of thousands of low-fidelity rendered images with random camera positions, lighting conditions, object positions, and non-realistic textures. At test time, the same detector is used in the real world with no additional training.From the PDF
Fig. 2 The model architecture used in our experiments. Each vertical bar corresponds to a layer of the model. ReLU nonlinearities are used throughout, and max pooling occurs between each of the groupings of convolutional layers. The input is an image from an external webcam downsized to (224 × 224) and the output of the network predicts the ( x, y, z ) coordinates of object(s) of interest.
Fig. 2 The model architecture used in our experiments. Each vertical bar corresponds to a layer of the model. ReLU nonlinearities are used throughout, and max pooling occurs between each of the groupings of convolutional layers. The input is an image from an external webcam downsized to (224 × 224) and the output of the network predicts the ( x, y, z ) coordinates of object(s) of interest.From the PDF
Fig. 3 The geometric objects used in our experiments.
Fig. 3 The geometric objects used in our experiments.From the PDF
Fig. 4 Sensitivity of test error on real images to the number of simulated training examples used. Each training example corresponds to a single labeled example of an object on the table with between 0 and 10 distractor objects. Lighting and all textures are randomized between iterations.
Fig. 4 Sensitivity of test error on real images to the number of simulated training examples used. Each training example corresponds to a single labeled example of an object on the table with between 0 and 10 distractor objects. Lighting and all textures are randomized between iterations.From the PDF
Fig. 5 Sensitivity to amount of texture randomization. In each case, the detector was trained using 10 , 000 random object positions and combinations of distractors, but only the given number of unique texturizations and lighting conditions were used.
Fig. 5 Sensitivity to amount of texture randomization. In each case, the detector was trained using 10 , 000 random object positions and combinations of distractors, but only the given number of unique texturizations and lighting conditions were used.From the PDF
Fig. 6 Two representative executions of grasping objects using vision learned in simulation only. The object detector network estimates the positions of the object of interest, and then a motion planner plans a simple sequence of motions to grasp the object at that location.
Fig. 6 Two representative executions of grasping objects using vision learned in simulation only. The object detector network estimates the positions of the object of interest, and then a motion planner plans a simple sequence of motions to grasp the object at that location.From the PDF
Fig. 7 A selection of randomly textured scenes used in the training phase of our method
Fig. 7 A selection of randomly textured scenes used in the training phase of our methodFrom the PDF

Verification

Claims the paper makes

2 Valid · 8 Not valid · 7 Not checked · 32 Own results and interpretations

29 more claims

Create a free account to see all 49 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 11
  • It may state two separate claims as one. 8

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 2

  1. wise, 2016, Fetch and freight: Standard platforms for service robot applications, Workshop on Autonomous Mobile Service Robots
  2. sucan, 0, MoveIt!

Cited by

See all 3,124

Metadata

Authors
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P.
Affiliations
  • University of California, Berkeley (US)
  • OpenAI (United States) (US)
  • International Computer Science Institute (US)
Venue
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Publisher
IEEE
Published
2017-09
Pages
23-30
ISSN
2153-0858 (ISSN-L)
OpenAlex
W2605102758
Cited by
3,124
References
69
Citations by year
2026: 413 · 2025: 433 · 2024: 336 · 2023: 403 · 2022: 374 · 2021: 422 · 2020: 352 · 2019: 256 · 2018: 106 · 2017: 25
FWCI
92.18
Type
conference paper
Language
en
Open access
Locations
  • IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
domain randomization, sim-to-real transfer, reality gap, object detection, object localization, robotic grasping, cluttered environments, distractor robustness, partial occlusion, robotic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.