Language and Robotics
Embodied robots have the potential to better understand and communicate with humans in natural language due to their ability to sense their environment through multiple modalities such as vision, audio, haptics and proprioception. They can also move and influence the world through their actions, enabling more active exploration of the environment. Our research explores how multimodal perceptual information can be used to better understand language, and motor skills can be used to actively engage with humans to learn natural language through interaction, particularly through dialog and games such as "I Spy."
People
Prasoon Goyal Ph.D. Alumni pgoyal [at] cs utexas edu
Prasoon Goyal Ph.D. Alumni pgoyal [at] cs utexas edu
Aishwarya Padmakumar Ph.D. Alumni aish [at] cs utexas edu
Jesse Thomason Ph.D. Alumni thomason DOT jesse AT gmail
Harel Yedidsion Postdoctoral Fellow harel [at] cs utexas edu
Publications
     [Expand to show all 40][Minimize]
CAPE: Corrective Actions from Precondition Errors using Large Language Models 2024
Shreyas Sundara Raman, Vanya Cohen, Ifrah Idrees, Eric Rosen, Raymond Mooney, Stefanie Tellex, and David Paulius, International Conference on Robotics and Automation (ICRA) (2024).
Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks 2023
Albert Yu, Raymond J. Mooney, International Conference on Learning Representations (2023).
End-to-End Learning to Follow Language Instructions with Compositional Policies 2022
Vanya Cohen, Geraud Nangue Tasse, Nakul Gopalan, Steven James, Ray Mooney, Benjamin Rosman, Workshop on Language and Robot Learning at CoRL 2022 (2022).
Planning with Large Language Models via Corrective Re-prompting 2022
Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, Stefanie Tellex, Foundation Models for Decision Making Workshop at NeurIPS 2022 (2022).
Supervised Attention from Natural Language Feedback for Reinforcement Learning 2021
Clara Cecilia Cannon, Masters Thesis, Department of Computer Science, The University of Texas at Austin.
Using Natural Language to Aid Task Specification in Sequential Decision Making Problems 2021
Prasoon Goyal, Ph.D. Proposal.
Zero-shot Task Adaptation using Natural Language 2021
Prasoon Goyal, Raymond J. Mooney, Scott Niekum, Arxiv (2021).
Dialog as a Vehicle for Lifelong Learning of Grounded Language Understanding Systems 2020
Aishwarya Padmakumar, PhD Thesis, Department of Computer Science, The University of Texas at Austin.
Evaluating the Robustness of Natural Language Reward Shaping Models to Spatial Relations 2020
Antony Yun, Undergraduate Honors Thesis, Computer Science Department, University of Texas at Austin.
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog 2020
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, Raymond J. Mooney, The Journal of Artificial Intelligence Research (JAIR), Vol. 67 (2020), pp. 327-374.
PixL2R: Guiding Reinforcement Learning using Natural Language by Mapping Pixels to Rewards 2020
Prasoon Goyal, Scott Niekum, Raymond J. Mooney, In 4th Conference on Robot Learning (CoRL), November 2020. Also presented on the 1st Language in Reinforcement Learning (LaReL) Workshop at ICML, July 2020 (Best Paper Award), the 6th Deep Rein...
Improving Grounded Natural Language Understanding through Human-Robot Dialog 2019
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J. Mooney, In IEEE International Conference on Robotics and Automation (ICRA), Montreal, Canada, May 2019.
Optimal Use Of Verbal Instructions For Multi-Robot Human Navigation Guidance 2019
Harel Yedidsion, Jacqueline Deans, Connor Sheehan, Mahathi Chillara, Justin Hart, Peter Stone, and Raymond J. Mooney, In Proceedings of the Eleventh International Conference on Social Robotics, pp. 133-143 2019. Springer.
Using Natural Language for Reward Shaping in Reinforcement Learning 2019
Prasoon Goyal, Scott Niekum, Raymond J. Mooney, In Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, August 2019.
Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog 2018
Jesse Thomason, PhD Thesis, Department of Computer Science, The University of Texas at Austin.
Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions 2018
Jesse Thomason, Jivko Sinapov, Raymond Mooney, Peter Stone, In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) , February 2018.
Improved Models and Queries for Grounded Human-Robot Dialog 2018
Aishwarya Padmakumar, PhD Proposal, Department of Computer Science, The University of Texas At Austin.
Interaction and Autonomy in RoboCup@Home and Building-Wide Intelligence 2018
Justin Hart, Harel Yedidsion, Yuqian Jiang, Nick Walker, Rishi Shah, Jesse Thomason, Aishwarya Padmakumar, Rolando Fernandez, Jivko Sinapov, Raymond Mooney, Peter Stone, In Artificial Intelligence (AI) for Human-Robot Interaction (HRI) symposium, AAAI Fall Symposium Series, Arlington, Virginia, October 2018.
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog 2018
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J. Mooney, In RSS Workshop on Models and Representations for Natural Human-Robot Communication (MRHRC-18). Robotics: Science and Systems (RSS), June 2018.
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog 2018
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J. Mooney, In Late-breaking Track at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDIAL-18), Melbourne, Australia, July 2018.
BWIBots: A platform for bridging the gap between AI and human--robot interaction research 2017
Piyush Khandelwal, Shiqi Zhang, Jivko Sinapov, Matteo Leonetti, Jesse Thomason, Fangkai Yang, Ilaria Gori, Maxwell Svetlik, Priyanka Khante, Vladimir Lifschitz, J. K. Aggarwal, Raymond Mooney, and Peter Stone, The International Journal of Robotics Research (2017).
Guiding Interaction Behaviors for Multi-modal Grounded Language Learning 2017
Jesse Thomason, Jivko Sinapov, and Raymond J. Mooney, In Proceedings of the Workshop on Language Grounding for Robotics at ACL 2017 (RoboNLP-17), Vancouver, Canada, August 2017.
Integrated Learning of Dialog Strategies and Semantic Parsing 2017
Aishwarya Padmakumar, Jesse Thomason, and Raymond J. Mooney, In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), pp. 547--557, Valencia, Spain, April 2017.
Opportunistic Active Learning for Grounding Natural Language Descriptions 2017
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Justin Hart, Peter Stone, and Raymond J. Mooney, In Proceedings of the 1st Annual Conference on Robot Learning (CoRL-17), Sergey Levine and Vincent Vanhoucke and Ken Goldberg (Eds.), pp. 67--76, Mountain View, California, November 2017. PMLR.
Continuously Improving Natural Language Understanding for Robotic Systems through Semantic Parsing, Dialog, and Multi-modal Perception 2016
Jesse Thomason, PhD proposal, Department of Computer Science, The University of Texas at Austin.
Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy" 2016
Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond J. Mooney, In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI-16), pp. 3477--3483, New York City 2016.
Learning to Interpret Natural Language Commands through Human-Robot Dialog 2015
Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone, In Proceedings of the 2015 International Joint Conference on Artificial Intelligence (IJCAI), pp. 1923--1929, Buenos Aires, Argentina, July 2015.
Adapting Discriminative Reranking to Grounded Language Learning 2013
Joohyun Kim and Raymond J. Mooney, In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL-2013), pp. 218--227, Sofia, Bulgaria, August 2013.
Grounded Language Learning Models for Ambiguous Supervision 2013
Joo Hyun Kim, PhD Thesis, Department of Computer Science, University of Texas at Austin.
Fast Online Lexicon Learning for Grounded Language Acquisition 2012
David L. Chen, Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL-2012) (2012), pp. 430--439.
Generative Models of Grounded Language Learning with Ambiguous Supervision 2012
Joohyun Kim, Technical Report, PhD proposal, Department of Computer Science, The University of Texas at Austin.
Learning Language from Ambiguous Perceptual Context 2012
David L. Chen, PhD Thesis, Department of Computer Science, University of Texas at Austin. 196.
Unsupervised PCFG Induction for Grounded Language Learning with Highly Ambiguous Supervision 2012
Joohyun Kim and Raymond J. Mooney, In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Natural Language Learning (EMNLP-CoNLL '12), pp. 433--444, Jeju Island, Korea, July 2012.
Learning to Interpret Natural Language Navigation Instructions from Observations 2011
David L. Chen and Raymond J. Mooney, Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011) (2011), pp. 859-865.
Panning for Gold: Finding Relevant Semantic Content for Grounded Language Learning 2011
David L. Chen and Raymond J. Mooney, In Proceedings of Symposium on Machine Learning in Speech and Language Processing (MLSLP 2011), June 2011.
Generative Alignment and Semantic Parsing for Learning from Ambiguous Supervision 2010
Joohyun Kim and Raymond J. Mooney, In Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), pp. 543--551, Beijing, China, August 2010.
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language 2010
David L. Chen, Joohyun Kim, Raymond J. Mooney, Journal of Artificial Intelligence Research, Vol. 37 (2010), pp. 397--435.
Learning Language from Perceptual Context 2009
David L. Chen, unpublished. Ph.D. proposal, Department of Computer Sciences, University of Texas at Austin.
Learning to Sportscast: A Test of Grounded Language Acquisition 2008
David L. Chen and Raymond J. Mooney, In Proceedings of the 25th International Conference on Machine Learning (ICML), Helsinki, Finland, July 2008.
Guiding a Reinforcement Learner with Natural Language Advice: Initial Results in RoboCup Soccer 2004
Gregory Kuhlmann, Peter Stone, Raymond J. Mooney, and Jude W. Shavlik, In The AAAI-2004 Workshop on Supervisory Control of Learning and Adaptive Systems, July 2004.
Labs
Machine Learning