Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
• We aim at fooling the discriminator with the generated trajectory (with adversarial loss)
Discriminator• Discriminates real samples and generated samples
DatasetWe use rich sensory inputs to collect both
successful and failure pouring sequences.
Variations of Pouring Sequences
Our Method
Multimodal Data Fusion
Initial State ClassificationWe predict the initial state of containers: container
type and liquid amount.Forecasting 3D Trajectory
We forecast the one-step future 3D trajectory of thehand with an adversarial training procedure.Generator• We generate trajectories that are close to the ground
truth demonstration. (with regression loss)
Cross User
IntroductionHumans have the amazing ability to perform very
subtle manipulation tasks using a closed-loop controlsystem with imprecise mechanics (i.e., our body parts)but rich sensory information (e.g., vision, tactile, etc.).
In this work, we take liquid pouring as a concreteexample and aim at learning to continuously monitorwhether liquid pouring is successful (e.g., no spilling)or not via rich sensory inputs.
Liquid Pouring Monitoring via Rich Sensory InputsTz-Ying Wu1,*, Juan-Ting Lin1,*, Tsun-Hsuang Wang1, Chan-Wei Hu1,
Juan Carlos Niebles2, Min Sun1 (*indicate equal contribution)1 National Tsing Hua University 2 Stanford University
Experiment
Translation Error
Acknowledgement
Vanilla RNN: Our fusion RNN without any auxiliary tasks.RNN w/ IOSC: Our fusion RNN with an aux. task, initial object state classification.RNN w/ TF: Our fusion RNN with an aux. task, trajectory forecastingOurs w/o adv.: Our fusion RNN with two aux. tasks. In this setting, we treat one-step trajectory forecasting as a regression task.Ours: Our fusion RNN with two aux. tasks. In this setting, we introduce theadversarial training loss to generate more diverse trajectories.
Project Page: http://aliensunmin.github.io/project/monitoring/
Rotation Error
Cross Container
Ablation Study
𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 = 𝐿𝐿𝑟𝑟𝐺𝐺𝑟𝑟 + 𝐿𝐿𝑎𝑎𝑎𝑎𝑎𝑎
𝐿𝐿𝑟𝑟𝐺𝐺𝑟𝑟 =1
𝑇𝑇 − 1�𝑡𝑡=1
𝑇𝑇−1
𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝑋𝑋𝑡𝑡+1,𝐺𝐺𝜃𝜃𝐺𝐺 ℎ𝑡𝑡 )
𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 𝑋𝑋𝑡𝑡+1,𝐺𝐺𝜃𝜃𝐺𝐺 ℎ𝑡𝑡 = 𝑀𝑀𝑀𝑀𝑀𝑀 𝑃𝑃,𝑃𝑃𝑃 + �𝑘𝑘=𝑥𝑥,𝑦𝑦,𝑧𝑧
1 − 𝑐𝑐𝑐𝑐𝑑𝑑 𝑟𝑟𝑘𝑘 − 𝑟𝑟𝑘𝑘′
𝐿𝐿𝑎𝑎𝑎𝑎𝑎𝑎 =1
𝑇𝑇 − 1�𝑡𝑡=1
𝑇𝑇−1
−𝑙𝑙𝑐𝑐𝑙𝑙𝐷𝐷𝜃𝜃𝐷𝐷 ℎ𝑡𝑡,𝐺𝐺𝜃𝜃𝐺𝐺 ℎ𝑡𝑡
𝐿𝐿𝐷𝐷𝐷𝐷𝐷𝐷 =1
𝑇𝑇 − 1�𝑡𝑡=1
𝑇𝑇−1
−𝑙𝑙𝑐𝑐𝑙𝑙 𝐷𝐷𝜃𝜃𝐷𝐷 ℎ𝑡𝑡 ,𝑋𝑋𝑡𝑡+1 − 𝑙𝑙𝑐𝑐𝑙𝑙 1 − 𝐷𝐷𝜃𝜃𝐷𝐷 ℎ𝑡𝑡 ,𝐺𝐺𝜃𝜃𝐺𝐺 ℎ𝑡𝑡