UTCS Artificial Intelligence
courses
talks/events
demos
people
projects
publications
software/data
labs
areas
admin
An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch (2020)
Siddarth Desai,
Ishan Durugkar
, Haresh Karnan,
Garrett Warnell
,
Josiah Hanna
, and
Peter Stone
We examine the problem of transferring a policy learned in a source environment to a target environment with different dynamics, particularly in the case where it is critical to reduce the amount of interaction with the target environment during learning. This problem is particularly important in sim-to-real transfer because simulators inevitably model real-world dynamics imperfectly. In this paper, we show that one existing solution to this transfer problem—grounded action transformation—is closely related to the problem of imitation from observation (IfO): learning behaviors that mimic the observations of behavior demonstrations. After establishing this relationship, we hypothesize that recent state-of-the-art approaches from the IfO literature can be effectively repurposed for grounded transfer learning. To validate our hypothesis we derive a new algorithm—generative adversarial reinforced action transformation (GARAT)—based on adversarial imitation from observation techniques. We run experiments in several domains with mismatched dynamics, and find that agents trained with GARAT achieve higher returns in the target environment compared to existing black-box transfer methods.
View:
PDF
Citation:
In
Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 2020)
, Virtual, December 2020.
Bibtex:
@inproceedings{NEURIPS20-Karnan, title={An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch}, author={Siddarth Desai and Ishan Durugkar and Haresh Karnan and Garrett Warnell and Josiah Hanna and Peter Stone}, booktitle={Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 2020)}, month={December}, address={Virtual}, url="http://www.cs.utexas.edu/users/ai-lab?NEURIPS20-Karnan", year={2020} }
People
Ishan Durugkar
Ph.D. Student
ishand [at] cs utexas edu
Josiah Hanna
Ph.D. Student
jphanna [at] cs utexas edu
Peter Stone
Faculty
pstone [at] cs utexas edu
Garrett Warnell
Research Scientist
warnellg [at] cs utexas edu
Areas of Interest
Imitation Learning
Machine Learning
Reinforcement Learning
Robotics
Transfer Learning
Labs
Learning Agents