Domain randomization for transferring deep neural networks from simulation to the real world
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.
Abstract
Bridging the `reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to 1.5 cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.
Overview
CoChat Summary
TL;DR
This paper introduces domain randomization: training an object detector on simulated RGB images with randomized rendering, textures, lighting, camera pose, and distractors so that the real world appears as another variation. The authors show that a detector trained only in simulation can localize objects in the real world accurately enough to grasp them in clutter.
Key findings
- Real-world object detectors trained only on simulated images localized objects to within about 1.5 cm on average, including with distractors and partial occlusions; per-object errors in Table I range from 0.6 ± 0.3 cm to 3.2 ± 5.8 cm depending on object and condition.
- The pipeline enabled a Fetch robot to detect and pick up the target object in 38 out of 40 trials in increasingly cluttered scenes, and a detector trained on a Spam can succeeded in 9 of 10 trials with previously unseen YCB food distractors.
- Ablations found sensitivity to number of training images, number of unique textures, presence of distractors in training, and camera randomization; random noise had negligible effect. Performance improved up to around 50,000 training samples, and degraded significantly with fewer than 1,000 textures.
- ImageNet pretraining was not essential with enough data: random weight initialization achieved nearly the same real-world transfer performance as pretrained initialization, though pretraining helped in the low-data regime.
Methods
- The authors trained a modified VGG-16 convolutional neural network to map a single 224×224 monocular webcam image to Cartesian coordinates of object(s) of interest, using simulated images rendered by MuJoCo.
- Domain randomization varied number/shape of distractors, object positions and textures, table/floor/skybox/robot textures, camera position/orientation/field of view, number and properties of lights, and image noise. Textures were random RGB values, gradients, or checker patterns; 0–10 distractor objects were added per scene.
- Training used stochastic gradient descent with Adam on an L2 loss, learning rates around 1e-4, and batch sizes of 25, 50, or 100. Evaluation used 480 webcam images of eight geometric objects at 70–105 cm from a fixed camera, with 60 labeled images per object: 20 alone, 20 with distractors, and 20 partially occluded.
Limitations
- The paper states that the detectors still overfit simulated training data: simulated error was 0.3–0.5 cm versus about 1.5 cm in the real world, and no real-world validation could be done during training because no real data were assumed available.
- The paper states future work is needed for contact-rich manipulation or higher precision, and that the method fixes table height, effectively making the task 2D pose estimation from a single monocular uncalibrated camera.
- Evident from the text: evaluation is limited to eight geometric objects plus one Spam can, the test camera position remains constant across all images, and lighting and surrounding scene were not controlled.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
2 Valid · 8 Not valid · 7 Not checked · 32 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although previous work demonstrated the ability to perform robotic control using a neural network pretrained on ImageNet and fine-tuned on randomized rendered pixels [37], this paper provides the first demonstration that domain randomization can be useful for robotic tasks requiring precision.”
What the sources say
- Partly supportedcites [37] [37] Fereshteh Sadeghi and Sergey Levine. (cad) 2 RL: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201 , 2016. - sourceExperimental — accuracy not yet measured
The citation [37] is attached to the preceding claim about prior work, not to the citing authors’ claim that their own paper is the first to demonstrate precision robotics. Section 3.1 (p. 4) says LSDA pretrains a CNN on ImageNet classification data and fine-tunes it, but for object detection. It does not report robotic control or fine-tuning on randomized rendered pixels. (LLM, conf=0.84)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Rather than relying on controller robustness, Yu et al. [53] use a model trained on varied physics to perform system identification using online trajectory data, but their approach is not shown to succeed in the real world.”
What the sources say
- Could not verifycites [53] [53] Wenhao Yu, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. arXiv preprint arXiv:1702.02453 , 2017. - sourceExperimental — accuracy not yet measured
The cited paper discusses image classification and localisation with convolutional networks (A.1, p. 10; A.2, p. 11), not Yu et al.'s approach. Searches for varied-physics training, online trajectory-based system identification, and real-world success found no passage addressing the claim. Our copy therefore cannot verify it. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Though using realistic RGB rendering alone has had limited success for transferring to real robotic tasks [16], incorporating realistic simulation of depth information can allow models trained on rendered images to transfer reasonably well to the real world [32].”
What the sources say
- Could not verifycites [16] [16] Stephen James and Edward Johns. 3d simulation for robot arm control with deep q-learning. arXiv preprint arXiv:1609.03759 , 2016. - sourceExperimental — accuracy not yet measured
The Introduction (§1) and Conclusion (§8) describe Adam as a method for stochastic optimization, while the CNN experiment (§6.3) concerns training on image datasets. Searches for realistic RGB rendering and sim-to-real robotic transfer found no passage addressing whether RGB rendering alone has had limited success on real robotic tasks. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, their experiments - collision avoidance in hallways and open spaces - do not demonstrate the ability to deal with high-precision tasks.”
What the sources say
- Could not verifycites [37] [37] Fereshteh Sadeghi and Sergey Levine. (cad) 2 RL: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201 , 2016. - sourceExperimental — accuracy not yet measured
The Experiments section (p. 5) evaluates LSDA on an object-detection dataset, not Sadeghi and Levine’s collision-avoidance experiments. Searches for Sadeghi and Levine, hallway or open-space collision avoidance, and high-precision tasks found no passage addressing the claim. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, using a pre-trained model can significantly improve performance when less training data is used.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
29 more claims
Create a free account to see all 49 claims and to verify the 2 references still open.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 8
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most policies trained in simulation generalize poorly to the real-world due to agents adapting to specific visual details of the simulator.”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In robotic vision tasks including object localization [53], object detection [54], pose estimation [55], and semantic segmentation [56], the training data from simulator always have different textures, lighting, and camera positions from the realistic environments.”
From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These artifacts limit the applicability of generative videos in physics-sensitive domains such as robotics [25,26], autonomous driving [27], simulationbased design, and scientific visualization [28].”
From PhyStruct-Video: Structured Physics State Spaces for Physically Consistent Video Generation (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In domain randomization, the inputs during learning are “randomized” in the sense of being warped or corrupted in particular ways.”
From The overfitted brain: Dreams evolved to assist generalization (2021) · cites this paper as [144]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Domain randomization is a domain generalization technique that introduces drastic changes in the training image appearances, which forces the network to learn domain-invariant features between training and target domains [39].”
From A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers (2024) · cites this paper as [39]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A. Experimental Setup · p. 4
“Each training sample consists of (a) a rendered image of the object and one or more distractors (also from among the geometric object set) on a simulated tabletop”
§ A. Domain randomization · p. 4
“Between 0 and 10 distractor objects are added to the table in each scene.”
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“In particular, we use a modified version the VGG-16 architecture [42] shown in Figure 2.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We found that using a learning rate of around 1e − 4 (as opposed to the standard 1e − 3 for Adam) improved convergence and helped avoid a common local optimum, mapping all objects to the center of the table.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 3×
“Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [27] and robotic control [21], [38].”
BackgroundResultHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×
“A variety of approaches have been proposed, including re-training the model in the target domain (e.g., [52]), adapting the weights of the model based on the statistics of the source and target domains (e.g., [22]), learning invariant features between domains (e.g., [47]), and learning a mapping…”
BackgroundMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 1×
“We render images using the MuJoCo Physics Engine’s [45] built-in renderer.”
MethodTrust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · cited 1×
“Learning in simulation is especially promising for building on recent results using deep reinforcement learning to achieve human-level performance on tasks like Atari [29] and robotic control [26], [41].”
ResultSelf-citationLearning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505 · cited 1×
“omputer vision community has devoted significant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]). A variety of approaches have been proposed, including re-training the model in the target domain (e.…”
BackgroundEnd-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T., et al. (2015) · Journal of machine learning research · doi:10.5555/2946645.2946684 · Cited by 3,987
Describing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438 · cited 1×
“To examine how our detectors would perform on objects with complex textures, we created simulated scenes in which the object of interest is given a texture from the Describable Textures Dataset (DTD) [4].”
MethodDeep domain confusion Maximizing for domain invariance
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., & Darrell, T. (2014) · arXiv · doi:10.48550/arxiv.1412.3474 · Cited by 2,285 · cited 1×
“, [50]), and learning a mapping from the target domain to the source domain (e.”
BackgroundPlaying for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,815 · cited 1×
“Combining data from high-quality simulators with other approaches like fine-tuning can also reduce the number of labeled samples required in the real world [34].”
BackgroundEnd-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
Self-citationRusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · cited 1×
“Researchers in the reinforcement learning community have also studied the problem of domain adaptation by learning invariant feature representations [13], adapting pretrained networks [35], and other methods.”
BackgroundRobots that can adapt like animals
Cully, A., Clune, J., Tarapore, D., & Mouret, J.-B. (2015) · Nature · doi:10.1038/nature14422 · Cited by 979 · cited 1×
“[8] demonstrates that exploration in the real world can be more efficient with prior knowledge from a simulator.”
BackgroundThe YCB object and model set: Towards common benchmarks for manipulation research
Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., & Dollar, A. M. (2015) · 2015 International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2015.7251504 · Cited by 838 · cited 2×
“To test the performance of our object detectors on real-world objects with non-uniform textures, we trained an object detector to localize a can of Spam from the YCB Dataset [3].”
MethodSelf-citationWhat you saw is not what you get: Domain adaptation using asymmetric kernel transforms
Kulis, B., Saenko, K., & Darrell, T. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995702 · Cited by 688 · cited 1×
“The computer vision community has devoted significant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]).”
BackgroundRender for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
Su, H., Qi, C. R., Li, Y., & Guibas, L. J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.308 · Cited by 680 · cited 1×
“More recently, 3D models have been used to augment training data to aid transferring deep neural networks between datasets and prevent over-fitting on small datasets for tasks like viewpoint estimation [40] and object detection [42], [29].”
MethodCross-domain video concept detection using adaptive svms
Yang, J., Yan, R., & Hauptmann, A. G. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291276 · Cited by 667
CAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618 · cited 2×
“Sadeghi and Levine’s work [37] is the most similar to our own.”
BackgroundMethodAdaptive batch normalization for practical domain adaptation
Li, Y., Wang, N., Shi, J., Hou, X., & Liu, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.03.005 · Cited by 601
Domain Adaptation: Learning Bounds and Algorithms
Mansour, Y., Mohri, M., & Rostamizadeh, A. (2009) · arXiv · doi:10.48550/arxiv.0902.3430 · Cited by 504 · cited 1×
“The computer vision community has devoted significant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]).”
BackgroundModel learning for robot control: A survey
Nguyen-Tuong, D., & Peters, J. (2011) · Cognitive Processing · doi:10.1007/s10339-011-0404-1 · Cited by 503 · cited 1×
“Even with strong system identification, the real world has unmodeled physical effects like nonrigidity, gear backlash, wear-and-tear, and fluid dynamics that are not captured by current physics simulators (though learning techniques may help bridge this gap [33]).”
BackgroundDescription and recognition of curved objects☆
Nevatia, R., & Binford, T. O. (1977) · Artificial Intelligence · doi:10.1016/0004-3702(77)90006-6 · Cited by 446 · cited 1×
“Researchers in computer vision have used 3D models as a tool to improve performance on real images since the earliest days of the field (e.g., [30], [24]).”
MethodThe MOPED framework: Object recognition and pose estimation for manipulation
Collet, A., Martinez, M., & Srinivasa, S. S. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911401765 · Cited by 421 · cited 2×
“, a full 3D mesh model [47] or a 3D metric feature representation [6]).”
BackgroundSim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · cited 1×
“Rusu et al. [36] explore using the progressive network architecture to adapt a model that is pre-trained on simulated pixels, and find it has better sample efficiency than fine-tuning or training in the real-world alone.”
BackgroundUnsupervised Cross-Domain Image Generation
Taigman, Y., Polyak, A., & Wolf, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02200 · Cited by 359
Revisiting Batch Normalization For Practical Domain Adaptation
Li, Y., Wang, N., Shi, J., Liu, J., & Hou, X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.04779 · Cited by 340 · cited 1×
“…the model in the target domain (e.g., [52]), adapting the weights of the model based on the statistics of the source and target domains (e.g., [22]), learning invariant features between domains (e.g., [47]), and learning a mapping from the target domain to the source domain (e.g., [43]).”
BackgroundLearning Deep Object Detectors from 3D Models
Peng, X., Sun, B., Ali, K., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.151 · Cited by 338 · cited 1×
“In [31], the authors find that by pretraining a network on ImageNet and fine-tuning on synthetic data created from 3D models, better detection performance on the PASCAL dataset can be achieved than training with only a few labeled examples from the real dataset.”
BackgroundLearning with Augmented Features for Heterogeneous Domain Adaptation
Duan, L., Xu, D., & Tsang, I. (2012) · arXiv · doi:10.48550/arxiv.1206.4660 · Cited by 320 · cited 1×
“The computer vision community has devoted significant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]).”
BackgroundLSDA: Large Scale Detection Through Adaptation
Hoffman, J., Guadarrama, S., Tzeng, E., Hu, R., Donahue, J., Girshick, R., Darrell, T., & Saenko, K. (2014) · arXiv · doi:10.48550/arxiv.1407.5035 · Cited by 284 · cited 1×
“The computer vision community has devoted significant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]).”
BackgroundObject recognition and full pose registration from a single image for robotic manipulation
Collet, A., Berenson, D., Srinivasa, S. S., & Ferguson, D. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152739 · Cited by 275 · cited 2×
“There has also been some success with only monocular camera images [5].”
BackgroundMethodThe Transferability Approach: Crossing the Reality Gap in Evolutionary Robotics
Koos, S., Mouret, J.-B., & Doncieux, S. (2013) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2012.2185849 · Cited by 272 · cited 1×
“[22].”
Evolutionary Robotics and the Radical Envelope-of-Noise Hypothesis
Jakobi, N. (1997) · Adaptive Behavior · doi:10.1177/105971239700600205 · Cited by 264 · cited 1×
“In [18], the authors suggested that transferability can be achieved by randomly varying the items in the implementation set – model parameters that are not essential to the controller achieving near-optimal performance.”
BackgroundPreparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237 · cited 1×
“Rather than relying on controller robustness, Yu et al. [53] use a model trained on varied physics to perform system identification using online trajectory data, but their approach is not shown to succeed in the real world.”
BackgroundPreparing for the Unknown: Learning a Universal Policy with Online System Identification
Yu, W., Tan, J., Liu, C. K., & Turk, G. (2017) · arXiv · doi:10.48550/arxiv.1702.02453 · Cited by 237 · cited 1×
“Rather than relying on controller robustness, Yu et al. [53] use a model trained on varied physics to perform system identification using online trajectory data, but their approach is not shown to succeed in the real world.”
Backgroundvan den Berg, J., Miller, S., Duckworth, D., Hu, H., Wan, A., Fu, X.-Y., Goldberg, K., & Abbeel, P. (2010) · 2010 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2010.5509621 · Cited by 233 · cited 1×
“Iterative learning control has been applied to a variety of robotic control problems, from model car control (e.g., [1] and [8]) to surgical robotics (e.g., [48]).”
BackgroundSelf-citationEmpagliflozin for Type 2 Diabetes Mellitus: An Overview of Phase 3 Clinical Trials
Levine, M. J. (2017) · Current Diabetes Reviews · doi:10.2174/1573399812666160613113556 · Cited by 230
Using inaccurate models in reinforcement learning
Abbeel, P., Quigley, M., & Ng, A. Y. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143845 · Cited by 207
Self-citationFrom Virtual to Reality: Fast Adaptation of Virtual Object Detectors to Real Domains
Sun, B., & Saenko, K. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.82 · Cited by 157
What and Where: 3D Object Recognition with Accurate Pose
Gordon, I., & Lowe, D. G. (2006) · Lecture notes in computer science · doi:10.1007/11957959_4 · Cited by 151 · cited 1×
“, Scale-Invariant Feature Transform [SIFT] features [14] or color co-occurrence histograms [12]) are matched with the 3D models (or features from the 3D models).”
MethodEfficient Learning of Domain-invariant Image Representations
Hoffman, J., Rodner, E., Donahue, J., Darrell, T. J., & Saenko, K. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3224 · Cited by 141 · cited 1×
“The computer vision community has devoted significant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]).”
BackgroundEnsemble-CIO: Full-body dynamic motion planning that transfers to physical humanoids
Mordatch, I., Lowrey, K., & Todorov, E. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354126 · Cited by 135 · cited 1×
“In the context of physics adaptation, Mordatch and collaborators [30] show that training a policy on an ensemble of dynamics models can make the controller robust to modeling error and improve transfer to a real robot.”
BackgroundA textured object recognition pipeline for color and depth image data
Tang, J., Miller, S., Singh, A., & Abbeel, P. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6224891 · Cited by 122 · cited 4×
“Most successful approaches rely on using multiple camera frames [6] or depth information [44].”
BackgroundInfluentialSelf-citationEPOpt: Learning Robust Neural Network Policies Using Model Ensembles
Rajeswaran, A., Ghotra, S., Ravindran, B., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01283 · Cited by 120 · cited 1×
“[36] explore different training strategies for learning from an ensemble of models, including adversarial training and adapting the ensemble distribution using data from the target domain, but also do not demonstrate successful real-world transfer.”
BackgroundMitash, C., Bekris, K. E., & Boularias, A. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202206 · Cited by 120 · cited 1×
“…of the preceding approaches require reward functions or labeled data, which can be difficult to obtain in the real world, Mitash and collaborators [26] explore pretraining an object detector using realistic rendered images with randomized lighting from 3D models to bootstrap an automated learning…”
BackgroundHow Useful Is Photo-Realistic Rendering for Visual Learning?
Movshovitz-Attias, Y., Kanade, T., & Sheikh, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_18 · Cited by 118 · cited 1×
“More recently, 3D models have been used to augment training data to aid transferring deep neural networks between datasets and prevent over-fitting on small datasets for tasks like viewpoint estimation [43] and object detection [45], [31].”
MethodPlaying for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.02192 · Cited by 118
Deep predictive policy training using reinforcement learning
Ghadirzadeh, A., Maki, A., Kragic, D., & Bjorkman, M. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206046 · Cited by 113 · cited 1×
“In [13], the authors use a variational autoencoder trained on simulated data to encode trajectories as a lowdimensional latent code.”
MethodAdapting Deep Visuomotor Representations with Weak Pairwise Constraints
Tzeng, E., Devin, C., Hoffman, J., Finn, C., Abbeel, P., Levine, S., Saenko, K., & Darrell, T. (2020) · Springer Proceedings in Advanced Robotics · doi:10.1007/978-3-030-43089-4_44 · Cited by 113 · cited 1×
“In [49], the authors learn a correspondence”
Self-citationLearning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
Gupta, A., Devin, C., Liu, Y., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02949 · Cited by 98 · cited 2×
“See [15] for a more complete treatment of domain adaptation in the reinforcement learning literature.”
BackgroundSelf-citationReal-time visual tracking of 3d objects with dynamic handling of occlusion
Wunsch, P., & Hirzinger, G. (2002) · Proceedings of International Conference on Robotics and Automation · doi:10.1109/robot.1997.606722 · Cited by 93 · cited 1×
“Object detection and pose estimation for robotics is a well-studied problem in the literature (see, e.g., [4], [5], [6], [10], [44], [50], [54]).”
BackgroundDepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition
Planche, B., Wu, Z., Ma, K., Sun, S., Kluckner, S., Lehmann, O., Chen, T., Hutter, A., Zakharov, S., Kosch, H., et al. (2017) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00011 · Cited by 83 · cited 1×
“Though using realistic RGB rendering alone has had limited success for transferring to real robotic tasks [16], incorporating realistic simulation of depth information can allow models trained on rendered images to transfer reasonably well to the real world [32].”
BackgroundDepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition
Planche, B., Wu, Z.-Y., Ma, K., Sun, S., Kluckner, S., Lehmann, O., Chen, T., Hutter, A., Zakharov, S., Kosch, H., et al. (2017) · International Conference on 3D Vision · doi:10.48550/arxiv.1702.08558 · Cited by 77
Efficient reinforcement learning for robots using informative simulated priors
Cutler, M., & How, J. P. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139550 · Cited by 66 · cited 1×
“It is often faster to fine-tune a controller learned in simulation than to learn from scratch in the real world [7], [18].”
BackgroundEfficient multi-view object recognition and full pose estimation
Collet, A., & Srinivasa, S. S. (2010) · 2010 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2010.5509615 · Cited by 64 · cited 3×
“Most successful approaches rely on using multiple camera frames [6] or depth information [44].”
Background3D Simulation for Robot Arm Control with Deep Q-Learning
James, S., & Johns, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03759 · Cited by 62 · cited 1×
“Though using realistic RGB rendering alone has had limited success for transferring to real robotic tasks [19], incorporating realistic simulation of depth information can allow models trained on rendered images to transfer well to the real world [35].”
BackgroundReinforcement learning with multi-fidelity simulators
Cutler, M., Walsh, T. J., & How, J. P. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6907423 · Cited by 56 · cited 1×
“Iterative learning control has been applied to a variety of robotic control problems, from model car control (e.g., [1] and [8]) to surgical robotics (e.g., [48]).”
BackgroundObject recognition and pose estimation using color cooccurrence histograms and geometric modeling
Ekvall, S., Kragic, D., & Hoffmann, F. (2005) · Image and Vision Computing · doi:10.1016/j.imavis.2005.05.006 · Cited by 52 · cited 1×
“, Scale-Invariant Feature Transform [SIFT] features [14] or color co-occurrence histograms [12]) are matched with the 3D models (or features from the 3D models).”
MethodModular Deep Q Networks for Sim-to-real Transfer of Visuo-motor PoliciesPreprint
Zhang, F., Leitner, J., Milford, M. J., & Corke, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.06781 · Cited by 47
Detection and Localization of Multiple Objects
Zickler, S., & Veloso, M. (2006) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2006.321358 · Cited by 40
Reinforcement Learning for Pivoting Task
Antonova, R., Cruciani, S., Smith, C., & Kragić, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00472 · Cited by 32 · cited 1×
“Similarly, in [2], the authors use a simulator with randomized friction and action delay and find behaviors transfer to the real world.”
BackgroundObject recognition and pose estimation for robotic manipulation using color cooccurrence histograms
Ekvall, S., Frank Hoffmann, B., & Kragić, D. (2004) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2003.1248822 · Cited by 32
Learning omnidirectional path following using dimensionality reduction
Kolter, J. Z., & Ng, A. Y. (2007) · Robotics: Science and Systems · doi:10.15607/rss.2007.iii.033 · Cited by 31
How useful is photo-realistic rendering for visual learning?Preprint
Movshovitz-Attias, Y., Takeo Kanade, A., & Sheikh, Y. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08152 · Cited by 24
Deep Predictive Policy Training using Reinforcement LearningPreprint
Ghadirzadeh, A., Maki, A., Kragić, D., & Björkman, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00727 · Cited by 21
Leitner, J., Harding, S., Frank, M., Förster, A., & Schmidhuber, J. (2013) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2013.6706819 · Cited by 11 · cited 1×
“Neural network-based perception has also been explored [25], but collecting a large enough training set can be challenging.”
BackgroundMitash, C., Bekris, K. E., & Boularias, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03347 · Cited by 11
Vision-Based Reaching Using Modular Deep Networks: from Simulation to the Real World
Zhang, F., Leitner, J., Upcroft, B., et al. (2016) · Cited by 8
Learning Omnidirectional Path Following Using Dimensionality Reduction
Kolter, J. Z., & Ng, A. Y. (2008) · The MIT Press eBooks · doi:10.7551/mitpress/7830.003.0034 · Cited by 5
CAD2RL: Real Single-Image Flight without a Single Real Image
Sadeghi, F., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1611.04201
Research works printed but not linked: 2
- wise, 2016, Fetch and freight: Standard platforms for service robot applications, Workshop on Autonomous Mobile Service Robots
- sucan, 0, MoveIt!
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
“helps a lot in closing this gap [70, 71].”
BackgroundAndrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
Self-citationSim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309
Self-citation
Metadata
- Authors
- Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- OpenAI (United States) (US)
- International Computer Science Institute (US)
- ORCID
- Abbeel, Pieter
- Venue
- 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2017-09
- Pages
- 23-30
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2605102758
- Semantic Scholar
- 32ceb28e45a445df4d89df281bb0e3ab5aab1a2a
- Cited by
- 3,124
- References
- 69
- Citations by year
- 2026: 413 · 2025: 433 · 2024: 336 · 2023: 403 · 2022: 374 · 2021: 422 · 2020: 352 · 2019: 256 · 2018: 106 · 2017: 25
- FWCI
- 92.18
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- domain randomization, sim-to-real transfer, reality gap, object detection, object localization, robotic grasping, cluttered environments, distractor robustness, partial occlusion, robotic manipulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.