Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog (2018)
In this work, we present methods for parsing natural language to underlying meanings, and using robotic sensors to create multi-modal models of perceptual concepts. We combine these steps towards language understanding into a holistic agent for jointly improving parsing and perception on a robotic platform through human-robot dialog. We train and evaluate this agent on Amazon Mechanical Turk, then demonstrate it on a robotic platform initialized from that conversational data. Our experiments show that improving both parsing and perception components from conversations improves communication quality and human ratings of the agent.
In Late-breaking Track at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDIAL-18), Melbourne, Australia, July 2018.

Justin Hart Postdoctoral Fellow hart [at] cs utexas edu
Yuqian Jiang Ph.D. Student
Raymond J. Mooney Faculty mooney [at] cs utexas edu
Aishwarya Padmakumar Ph.D. Student aish [at] cs utexas edu
Jivko Sinapov Postdoctoral Alumni jsinapov [at] cs utexas edu
Peter Stone Faculty pstone [at] cs utexas edu
Jesse Thomason Ph.D. Alumni thomason DOT jesse AT gmail
Harel Yedidsion Postdoctoral Fellow harel [at] cs utexas edu