Language and Robotics
Embodied robots have the potential to better understand and communicate with humans in natural language due to their ability to sense their environment through multiple modalities such as vision, audio, haptics and proprioception. They can also move and influence the world through their actions, enabling more active exploration of the environment. Our research explores how multimodal perceptual information can be used to better understand language, and motor skills can be used to actively engage with humans to learn natural language through interaction, particularly through dialog and games such as "I Spy."
Prasoon Goyal Ph.D. Student pgoyal [at] cs utexas edu
Prasoon Goyal Ph.D. Student pgoyal [at] cs utexas edu
Aishwarya Padmakumar Ph.D. Student aish [at] cs utexas edu
Jesse Thomason Ph.D. Alumni thomason DOT jesse AT gmail
Harel Yedidsion Postdoctoral Fellow harel [at] cs utexas edu
     [Expand to show all 22][Minimize]
Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog 2018
Jesse Thomason, PhD Thesis, Department of Computer Science, The University of Texas at Austin.
Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions 2018
Jesse Thomason, Jivko Sinapov, Raymond Mooney, Peter Stone, In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) , February 2018.
BWIBots: A platform for bridging the gap between AI and human--robot interaction research 2017
Piyush Khandelwal, Shiqi Zhang, Jivko Sinapov, Matteo Leonetti, Jesse Thomason, Fangkai Yang, Ilaria Gori, Maxwell Svetlik, Priyanka Khante, Vladimir Lifschitz, J. K. Aggarwal, Raymond Mooney, and Peter Stone, The International Journal of Robotics Research (2017).
Guiding Interaction Behaviors for Multi-modal Grounded Language Learning 2017
Jesse Thomason, Jivko Sinapov, and Raymond J. Mooney, In Proceedings of the Workshop on Language Grounding for Robotics at ACL 2017 (RoboNLP-17), Vancouver, Canada, August 2017.
Integrated Learning of Dialog Strategies and Semantic Parsing 2017
Aishwarya Padmakumar, Jesse Thomason, and Raymond J. Mooney, In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), pp. 547--557, Valencia, Spain, April 2017.
Opportunistic Active Learning for Grounding Natural Language Descriptions 2017
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Justin Hart, Peter Stone, and Raymond J. Mooney, In Proceedings of the 1st Annual Conference on Robot Learning (CoRL-17), Sergey Levine and Vincent Vanhoucke and Ken Goldberg (Eds.), pp. 67--76, Mountain View, California, November 2017. PMLR.
Continuously Improving Natural Language Understanding for Robotic Systems through Semantic Parsing, Dialog, and Multi-modal Perception 2016
Jesse Thomason, PhD proposal, Department of Computer Science, The University of Texas at Austin.
Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy" 2016
Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond J. Mooney, In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI-16), pp. 3477--3483, New York City 2016.
Learning to Interpret Natural Language Commands through Human-Robot Dialog 2015
Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone, In Proceedings of the 2015 International Joint Conference on Artificial Intelligence (IJCAI), pp. 1923--1929, Buenos Aires, Argentina, July 2015.
Adapting Discriminative Reranking to Grounded Language Learning 2013
Joohyun Kim and Raymond J. Mooney, In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL-2013), pp. 218--227, Sofia, Bulgaria, August 2013.
Grounded Language Learning Models for Ambiguous Supervision 2013
Joo Hyun Kim, PhD Thesis, Department of Computer Science, University of Texas at Austin.
Fast Online Lexicon Learning for Grounded Language Acquisition 2012
David L. Chen, Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL-2012) (2012), pp. 430--439.
Generative Models of Grounded Language Learning with Ambiguous Supervision 2012
Joohyun Kim, Technical Report, PhD proposal, Department of Computer Science, The University of Texas at Austin.
Learning Language from Ambiguous Perceptual Context 2012
David L. Chen, PhD Thesis, Department of Computer Science, University of Texas at Austin. 196.
Unsupervised PCFG Induction for Grounded Language Learning with Highly Ambiguous Supervision 2012
Joohyun Kim and Raymond J. Mooney, In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Natural Language Learning (EMNLP-CoNLL '12), pp. 433--444, Jeju Island, Korea, July 2012.
Learning to Interpret Natural Language Navigation Instructions from Observations 2011
David L. Chen and Raymond J. Mooney, Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011) (2011), pp. 859-865.
Panning for Gold: Finding Relevant Semantic Content for Grounded Language Learning 2011
David L. Chen and Raymond J. Mooney, In Proceedings of Symposium on Machine Learning in Speech and Language Processing (MLSLP 2011), June 2011.
Generative Alignment and Semantic Parsing for Learning from Ambiguous Supervision 2010
Joohyun Kim and Raymond J. Mooney, In Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), pp. 543--551, Beijing, China, August 2010.
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language 2010
David L. Chen, Joohyun Kim, Raymond J. Mooney, Journal of Artificial Intelligence Research, Vol. 37 (2010), pp. 397--435.
Learning Language from Perceptual Context 2009
David L. Chen, unpublished. Ph.D. proposal, Department of Computer Sciences, University of Texas at Austin.
Learning to Sportscast: A Test of Grounded Language Acquisition 2008
David L. Chen and Raymond J. Mooney, In Proceedings of the 25th International Conference on Machine Learning (ICML), Helsinki, Finland, July 2008.
Guiding a Reinforcement Learner with Natural Language Advice: Initial Results in RoboCup Soccer 2004
Gregory Kuhlmann, Peter Stone, Raymond J. Mooney, and Jude W. Shavlik, In The AAAI-2004 Workshop on Supervisory Control of Learning and Adaptive Systems, July 2004.