Domain randomization for transferring deep neural networks from simulation to the real world
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.
Abstract
Bridging the `reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to 1.5 cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.
Overview
CoChat Summary
TL;DR
This paper introduces domain randomization: training an object detector on simulated RGB images with randomized rendering, textures, lighting, camera pose, and distractors so that the real world appears as another variation. The authors show that a detector trained only in simulation can localize objects in the real world accurately enough to grasp them in clutter.
Key findings
- Real-world object detectors trained only on simulated images localized objects to within about 1.5 cm on average, including with distractors and partial occlusions; per-object errors in Table I range from 0.6 ± 0.3 cm to 3.2 ± 5.8 cm depending on object and condition.
- The pipeline enabled a Fetch robot to detect and pick up the target object in 38 out of 40 trials in increasingly cluttered scenes, and a detector trained on a Spam can succeeded in 9 of 10 trials with previously unseen YCB food distractors.
- Ablations found sensitivity to number of training images, number of unique textures, presence of distractors in training, and camera randomization; random noise had negligible effect. Performance improved up to around 50,000 training samples, and degraded significantly with fewer than 1,000 textures.
- ImageNet pretraining was not essential with enough data: random weight initialization achieved nearly the same real-world transfer performance as pretrained initialization, though pretraining helped in the low-data regime.
Methods
- The authors trained a modified VGG-16 convolutional neural network to map a single 224×224 monocular webcam image to Cartesian coordinates of object(s) of interest, using simulated images rendered by MuJoCo.
- Domain randomization varied number/shape of distractors, object positions and textures, table/floor/skybox/robot textures, camera position/orientation/field of view, number and properties of lights, and image noise. Textures were random RGB values, gradients, or checker patterns; 0–10 distractor objects were added per scene.
- Training used stochastic gradient descent with Adam on an L2 loss, learning rates around 1e-4, and batch sizes of 25, 50, or 100. Evaluation used 480 webcam images of eight geometric objects at 70–105 cm from a fixed camera, with 60 labeled images per object: 20 alone, 20 with distractors, and 20 partially occluded.
Limitations
- The paper states that the detectors still overfit simulated training data: simulated error was 0.3–0.5 cm versus about 1.5 cm in the real world, and no real-world validation could be done during training because no real data were assumed available.
- The paper states future work is needed for contact-rich manipulation or higher precision, and that the method fixes table height, effectively making the task 2D pose estimation from a single monocular uncalibrated camera.
- Evident from the text: evaluation is limited to eight geometric objects plus one Spam can, the test camera position remains constant across all images, and lighting and surrounding scene were not controlled.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 8 Not valid · 7 Not checked · 32 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although previous work demonstrated the ability to perform robotic control using a neural network pretrained on ImageNet and fine-tuned on randomized rendered pixels [37], this paper provides the first demonstration that domain randomization can be useful for robotic tasks requiring precision.”
What the sources say
- Partly supportedcites [37] [37] Fereshteh Sadeghi and Sergey Levine. (cad) 2 RL: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201 , 2016. - sourceExperimental — accuracy not yet measured
The citation [37] is attached to the preceding claim about prior work, not to the citing authors’ claim that their own paper is the first to demonstrate precision robotics. Section 3.1 (p. 4) says LSDA pretrains a CNN on ImageNet classification data and fine-tunes it, but for object detection. It does not report robotic control or fine-tuning on randomized rendered pixels. (LLM, conf=0.84)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Rather than relying on controller robustness, Yu et al. [53] use a model trained on varied physics to perform system identification using online trajectory data, but their approach is not shown to succeed in the real world.”
What the sources say
- Could not verifycites [53] [53] Wenhao Yu, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. arXiv preprint arXiv:1702.02453 , 2017. - sourceExperimental — accuracy not yet measured
The cited paper discusses image classification and localisation with convolutional networks (A.1, p. 10; A.2, p. 11), not Yu et al.'s approach. Searches for varied-physics training, online trajectory-based system identification, and real-world success found no passage addressing the claim. Our copy therefore cannot verify it. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Though using realistic RGB rendering alone has had limited success for transferring to real robotic tasks [16], incorporating realistic simulation of depth information can allow models trained on rendered images to transfer reasonably well to the real world [32].”
What the sources say
- Could not verifycites [16] [16] Stephen James and Edward Johns. 3d simulation for robot arm control with deep q-learning. arXiv preprint arXiv:1609.03759 , 2016. - sourceExperimental — accuracy not yet measured
The Introduction (§1) and Conclusion (§8) describe Adam as a method for stochastic optimization, while the CNN experiment (§6.3) concerns training on image datasets. Searches for realistic RGB rendering and sim-to-real robotic transfer found no passage addressing whether RGB rendering alone has had limited success on real robotic tasks. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, their experiments - collision avoidance in hallways and open spaces - do not demonstrate the ability to deal with high-precision tasks.”
What the sources say
- Could not verifycites [37] [37] Fereshteh Sadeghi and Sergey Levine. (cad) 2 RL: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201 , 2016. - sourceExperimental — accuracy not yet measured
The Experiments section (p. 5) evaluates LSDA on an object-detection dataset, not Sadeghi and Levine’s collision-avoidance experiments. Searches for Sadeghi and Levine, hallway or open-space collision avoidance, and high-precision tasks found no passage addressing the claim. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, using a pre-trained model can significantly improve performance when less training data is used.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
29 more claims
Create a free account to see all 49 claims and to verify the 2 references still open.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 8
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most policies trained in simulation generalize poorly to the real-world due to agents adapting to specific visual details of the simulator.”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In robotic vision tasks including object localization [53], object detection [54], pose estimation [55], and semantic segmentation [56], the training data from simulator always have different textures, lighting, and camera positions from the realistic environments.”
From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These artifacts limit the applicability of generative videos in physics-sensitive domains such as robotics [25,26], autonomous driving [27], simulationbased design, and scientific visualization [28].”
From PhyStruct-Video: Structured Physics State Spaces for Physically Consistent Video Generation (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In domain randomization, the inputs during learning are “randomized” in the sense of being warped or corrupted in particular ways.”
From The overfitted brain: Dreams evolved to assist generalization (2021) · cites this paper as [144]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Domain randomization is a domain generalization technique that introduces drastic changes in the training image appearances, which forces the network to learn domain-invariant features between training and target domains [39].”
From A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers (2024) · cites this paper as [39]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A. Experimental Setup · p. 4
“Each training sample consists of (a) a rendered image of the object and one or more distractors (also from among the geometric object set) on a simulated tabletop”
§ A. Domain randomization · p. 4
“Between 0 and 10 distractor objects are added to the table in each scene.”
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“In particular, we use a modified version the VGG-16 architecture [42] shown in Figure 2.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We found that using a learning rate of around 1e − 4 (as opposed to the standard 1e − 3 for Adam) improved convergence and helped avoid a common local optimum, mapping all objects to the center of the table.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 3×
“Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [27] and robotic control [21], [38].”
BackgroundResultHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×
“A variety of approaches have been proposed, including re-training the model in the target domain (e.g., [52]), adapting the weights of the model based on the statistics of the source and target domains (e.g., [22]), learning invariant features between domains (e.g., [47]), and learning a mapping…”
BackgroundMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 1×
“We render images using the MuJoCo Physics Engine’s [45] built-in renderer.”
MethodTrust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · cited 1×
“Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [29] and robotic control [26], [41].”
ResultSelf-citationLearning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T., et al. (2015) · Journal of machine learning research · doi:10.5555/2946645.2946684 · Cited by 3,987
Describing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438 · cited 1×
“To examine how our detectors would perform on objects with complex textures, we created simulated scenes in which the object of interest is given a texture from the Describable Textures Dataset (DTD) [4].”
Method
Research works printed but not linked: 2
- wise, 2016, Fetch and freight: Standard platforms for service robot applications, Workshop on Autonomous Mobile Service Robots
- sucan, 0, MoveIt!
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
“helps a lot in closing this gap [70, 71].”
BackgroundAndrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
Self-citationSim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309
Self-citation
Metadata
- Authors
- Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- OpenAI (United States) (US)
- International Computer Science Institute (US)
- ORCID
- Abbeel, Pieter
- Venue
- 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2017-09
- Pages
- 23-30
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2605102758
- Semantic Scholar
- 32ceb28e45a445df4d89df281bb0e3ab5aab1a2a
- Cited by
- 3,124
- References
- 69
- Citations by year
- 2026: 413 · 2025: 433 · 2024: 336 · 2023: 403 · 2022: 374 · 2021: 422 · 2020: 352 · 2019: 256 · 2018: 106 · 2017: 25
- FWCI
- 92.18
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- domain randomization, sim-to-real transfer, reality gap, object detection, object localization, robotic grasping, cluttered environments, distractor robustness, partial occlusion, robotic manipulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Domain randomization for transferring deep neural networks from simulation to the real world. https://app.cochat.ai/papers/10.1109/iros.2017.8202133
- The paper
- Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). https://doi.org/10.1109/iros.2017.8202133
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.