Robobarista: Object part-based transfer of manipulation trajectories from crowd-sourcing in 3d pointclouds

“Is segmentation required for the system? In vision community, even with the state-of-the-art techniques [12, 27], detection of ‘manipulatable’ object parts such as ‘handle’”

“On the other hand, direct perception approach skips the intermediate representation of object labels and directly perceives affordance based on the shape of the object [14, 28].”

“regression problem [5] with a uni-modal input.”

“Is segmentation required for the system? In vision community, even with the state-of-the-art techniques [14, 29], detection of ‘manipulatable’ object parts such as ‘handle’ and ‘lever’ in point-cloud is by itself a challenging problem [31].”

“Then, we introduce the crowd-sourcing platform Robobarista for easily scaling the collection of manipulation demonstrations to non-expert public on the web.”

“We use the stacked sparse de-noising autoencoder (SSDA) to train weights W i and bias b for each layer [59, 63].”

“tuned by minimizing the negative log-likelihood with the stochastic gradient method with mini-batches: NLL = − ∑|D| i=0 log(P (Y = y |x,W, b)) To prevent over-fitting to the training data, we used dropout [17], which randomly drops a specified percentage of the output of every layer.”

“intermediate step of detecting object part does not guarantee successful transfers, we built an object part classifier using multiclass SVM [56] on point-cloud features including local shape features [26], histogram of curvatures [40], and distribution of points.”

“based learning [2, 13] replicates one of the demonstrations.”

“Teaching robots how to manipulate different objects has often relied on experts [4, 1].”

“The first h(1) layer learns a separate layer of features for each modality of x (= h) [36].”

“is interpolated using spherical linear interpolation (Slerp) [44].”

“For each object, we took raw RGB-D images with the Microsoft Kinect sensor and stitched them using Kinect Fusion [20] to form a denser point-cloud in order to incorporate”

“In robotics, deep learning has previously been successfully used for detecting grasps on multi-channel input of RGB-D images [30] and for classifying terrain from long-range vision [16].”

“3) Structured support vector machine (SSVM): It is a traditional approach to use hand-coded features with SSVM [57], which can be solved with cutting plane method [22] along with our loss function (Sec.”

“Borrowing from Sturm et al. [50], we encode joint type, center of the joint, and axis of the joint as the latent variable h ∈ H in Latent SSVM [62].”

“on high level task composition [32, 10] or is often limited to previously seen objects [38, 37].”

“Teaching robots how to manipulate different objects has often relied on experts [4, 1].”

“[25]) and not their rotations and gripper status.”

“There has been great advancement in scene understanding [31, 41, 26, 61], in human activity detection [51, 19], and in features for RGB-D images and point-clouds [47, 29].”

“Thus, rather than relying on scene understanding techniques [7, 31, 15], we”

“Every layer h uses the rectified linear unit [63] as the”

“The most successful approach for teaching robots, such as performing complex helicopter maneuvers [1] or playing table tennis [35], has been based on LfD [4].”

“Thus, rather than relying on scene understanding techniques [7, 31, 15], we”

“There has been great advancement in scene understanding [31, 41, 26, 61], in human activity detection [51, 19], and in features for RGB-D images and point-clouds [47, 29].”

“ferent controllers to accomplish complicated tasks such as baking cookies [8] and folding the laundry [33], or focus on specific motions such as learning to grasp [24] and open doors [11].”

“the x-axis along the principal axis of the object part using PCA [18].”

“Among previous efforts to scale teaching to the crowd [9, 52, 21], Forbes”

“feedbacks [60, 58] to adapt exactly to the object as well as find a collision free path [49].”

“In future work, we plan to share the learned model using knowledge-engine RoboBrain [43].”

“Manipulating new instances of objects relies on finding articulation [50, 39] or using interaction [23] but requires tracking with a vision algorithm.”

“Among previous efforts to scale teaching to the crowd [9, 52, 21], Forbes”

“Another approach using part-based transfer between objects has been shown to be successful for grasping [10, 12].”

“Another approach using part-based transfer between objects has been shown to be successful for grasping [10, 12].”

“as quaternions (rx, ry, rz, rw) instead of the more compact Euler angles to prevent problems such as the gimbal lock [42].”

“In robotics, deep learning has previously been successfully used for detecting grasps on multi-channel input of RGB-D images [30] and for classifying terrain from long-range vision [16].”

“On the other hand, direct perception approach skips the intermediate representation of object labels and directly perceives affordance based on the shape of the object [14, 28].”

“Manipulating new instances of objects relies on finding articulation [50, 39] or using interaction [23] but requires tracking with a vision algorithm.”

“There has been great advancement in scene understanding [31, 41, 26, 61], in human activity detection [51, 19], and in features for RGB-D images and point-clouds [47, 29].”

“[13] employs similar approach towards crowd-sourcing but collects multiple instances of similar table-top manipulation with same object, and others builds web-based platform for crowd-sourcing manipulation [54, 55].”

“Among previous efforts to scale teaching to the crowd [9, 52, 21], Forbes”

“Manipulating new instances of objects relies on finding articulation [50, 39] or using interaction [23] but requires tracking with a vision algorithm.”

“on high level task composition [32, 10] or is often limited to previously seen objects [38, 37].”

“Our Robobarista platform borrows some components of [3] but works on any standard web browser with OpenGL support and incorporates real point-clouds of various scenes.”

“on high level task composition [32, 10] or is often limited to previously seen objects [38, 37].”

“[13] introduces a score function for weighting demonstrations based on weighted distance to “seed” (expert) demonstration.”

“ferent controllers to accomplish complicated tasks such as baking cookies [8] and folding the laundry [33], or focus on specific motions such as learning to grasp [24] and open doors [11].”

“There has been great advancement in scene understanding [31, 41, 26, 61], in human activity detection [51, 19], and in features for RGB-D images and point-clouds [47, 29].”

“feedbacks [60, 58] to adapt exactly to the object as well as find a collision free path [49].”

“Others focus on learning to sequence different movements [53, 36] but assume that there exist perfect controllers such as grasp and pour.”

“feedbacks [60, 58] to adapt exactly to the object as well as find a collision free path [49].”

“on high level task composition [32, 10] or is often limited to previously seen objects [38, 37].”

“In recent years, LfD was successfully used in the field of robotics for applications as varied as autonomous helicopter maneuvers [1], playing table tennis [2], object manipulation [3], making coffee [4], grasping novel objects [5], carrot grating [6], and eggplant cutting [7].”

“Neural networks are also highly effective at combining data from multiple sensor modalities or information sources [119, 233, 236, 370, 369, 240, 173].”

“, Rubik’s cube [1]) or by incorporating explicit human demonstrations [46], [12], [39], [15], [57], [47], [38].”

“Deep learning has been used in a range of applications within robotic perception and learning [29, 23, 6].”

“3D deep learning is receiving attention and recognition at an accelerated rate due to its high relevance in complex tasks such as robotics [22, 42, 36, 43], self-driving cars [33, 29, 8], and augmented and virtual reality [13, 3].”

“We focus on sensory-rich robotics applications, and our architecture extends previous works doing sensory-fusion with feed-forward networks [52, 61] to the fusion of temporal streams.”

“Several works adopt a behavioral cloning approach, where the model is trained using supervised learning with language-action sequences pairs (Anderson et al., 2018; Mei et al., 2016; Sung et al., 2015).”

“Alternatively, teleoperation-based techniques for collecting task demonstrations have been employed for over two decades [23, 24, 25].”

“We also remark that separating decision making into a high-level trajectory computation step and a low-level action execution is standard in motion planning [22], and has been explored in several recent studies on robotic manipulation [40, 41].”

“Such networks can be used for learning task plans from demonstrations to build upon a robot’s knowledge base [147, 148]; these learned plans can be used for predicting future activities to anticipate the consequences and forces exerted from those actions [149].”

“tructions directly to an action sequence to be executed. Tellex et al. [9] dynamically instantiate a graphical model given a language command, from which a plan for the agent is inferred. Sung et al. [10] learn a neural network to predict relatedness between htrajectory, languageipairs and htrajectory, point cloudipai…”

“xtend these demonstrations to unseen situations. Successful LfD applications include autonomous helicopter maneuvers [13], playing table tennis [14], [15], object manipulation [16], and making coffee [17]. Acquiring a sufficient number of demonstrations is the common challenge of LfD systems. To overcome this problem,…”

“Other works in this area have considered mapping text to a variety of manipulation actions (Sung et al., 2015).”

“Sub-goal trees can be seen as a form of trajectory representation (as we discuss in Section 5.2), which was investigated for learning robotic skills (M¨ulling et al., 2013; Sung et al., 2018) and navigation (Qureshi et al., 2018).”

“level tasks [16] and trajectory selection [17].”

“Each object is provided with a nominal plan with multiple phases, each defined as a sequence of smoothly interpolated waypoints consisting of end-effector position and orientation along with gripper actions of grasping similar to [6].”

“[33] proposed a manipulation planning approach according to the assumption that many household items share similar operational components.”

“As an example application, we consider manipulating novel appliances [1].”

“When instance-level knowledge is used alone, general rules in the domain are induced solely from raw data and no prior knowledge is incorporated [61, 62, 56].”

“Sung et al. [32] proposed a manipulation planning approach according to the assumption that many household items share similar operational components.”

“, Rubik’s cube [1]) or by incorporating explicit human demonstrations [46], [12], [39], [15], [57], [47], [38].”

“Another subcategory of instruction-following involves instructing an embodied agent using natural language [32, 15, 5, 29, 30, 22, 31].”

“…2016) to learning language-guided autonomous policies for robotic navigation and obstacle avoidance (Mei et al., 2016; Arumugam et al., 2017; Matuszek et al., 2013; Blukis et al., 2018; Sung et al., 2018; Shah et al., 2018; MacGlashan et al., 2015; Artzi & Zettlemoyer, 2013; Chen & Mooney, 2011).”

“No-table examples include protein folding [1, 2], RNA synthesis [3, 4], genome sequence alignment [5], robot trajectory planning [6], and others [7–9].”

“Manipulations are usually associated with specific affordance-bearing parts of objects [29], [30], [31].”

“Generalizing between objects is often easier when the representation explicitly models the manipulation-relevant parts of the objects [51, 24].”

“Although previous work [61] was able to visualize several nodes in the top layer, most were difficult to interpret.”

“Natural language utterances offer a flexible, bandwidth-efficient medium that humans can readily use to convey knowledge of an object’s operation [46].”

“Examples of successful application include autonomous helicopter maneuvers [Abbeel et al., 2010], playing table tennis [Kober et al., 2011][Calinon et al., 2010], object manipulation [Pastor et al., 2009], and making coffee [Sung et al., 2015].”

“Various trajectory representations have been investigated for learning robotic skills [21, 31] and navigation [24].”

“[40] crowdsourced multimodal affordance demonstrations using real robot data, enabling a robot to execute affordances on novel objects.”

“reinforcement learning [14], [15], [16], [17].”

“tions and crowdsourcing have been used for large-scale affordance learning [6, 8, 16, 17, 19], but only for a fixed set of manipulation actions.”

“A few pioneering achievements that have applied crowdsourced learning to robot physical skills include picking and grasping novel objects [14][15][16], machine operation [17], and assembly [18].”

“In a large class of approaches, which can broadly be termed as instructionfollowing, language is used to communicate the task to a learning agent, wherein, the agent is given a natural language command for a task, and is trained to take a sequence of actions that complete the task (Anderson et al., 2018; Fried et al.,…”

“No-table examples include protein folding [1, 2], RNA synthesis [3, 4], genome sequence alignment [5], robot trajectory planning [6], and others [7–9].”

“We focus on sensory-rich robotics applications, and our architecture extends previous works doing sensory-fusion with feed-forward networks [166, 212] to the fusion of temporal streams.”

“Similar research for using natural language as an additional source of information along with perception and human demonstrations is proposed earlier [77].”

“Visual perception plays a crucial role in robotic manipulation by providing environmental information and estimating object properties such as position, pose, and geometry [5]-[10].”

“Annotations have been used in a wide variety of object-related robotic tasks, such as shape reconstruction (Berger et al. 2017), rigid object grasping (Leeper et al. 2012), and learning articulated models (Sung et al. 2015).”

“and cheap alternative that has been applied to a wide variety of robotics applications like robot navigation [88], grasping [78, 112, 114], or object manipulation [68, 116].”

“Unlike our previous work [148], each cell count is also distributed to the neighboring cells with an exponential distribution.”