Publications: Connecting Language and Perception
To truly understand language, an intelligent system must be able to connect
words, phrases, and sentences to its perception of objects and events in the
world. Ideally, an AI system would be able to learn language like a human
child, by being exposed to utterances in a rich perceptual environment. The
perceptual context would provide the necessary supervisory information, and
learning the connection between language and perception would ground the
system's semantic representations in its perception of the world. As a step in
this direction, our research is developing systems that learn
semantic parsers and language generators from sentences
paired only with their perceptual context. It is part of our research on
natural language learning. Our research on this topic is
supported by the National Science Foundation through grants
IIS-0712097 and IIS-1016312.
- Learning Language from its Perceptual Context [Video Lecture]
Raymond J. Mooney, Invited Talk, ECML-PKDD, 2008.
- Learning to Interpret Natural Language Navigation Instructions from Observations
[Details] [PDF] [Slides]
David L. Chen and Raymond J. Mooney
In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011), 859-865, August 2011.
- Panning for Gold: Finding Relevant Semantic Content for Grounded Language Learning
[Details] [PDF] [Slides]
David L. Chen and Raymond J. Mooney
In Proceedings of Symposium on Machine Learning in Speech and Language Processing (MLSLP 2011), June 2011.
- Generative Alignment and Semantic Parsing for Learning from Ambiguous Supervision
[Details] [PDF]
Joohyun Kim and Raymond J. Mooney
In Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), 543--551, Beijing, China, August 2010.
- Using Closed Captions as Supervision for Video Activity Recognition
[Details] [PDF]
Sonal Gupta, Raymond J. Mooney
In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2010), 1083--1088, Atlanta, GA, July 2010.
- Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language
[Details] [PDF]
David L. Chen, Joohyun Kim, Raymond J. Mooney
Journal of Artificial Intelligence Research, 37:397--435, 2010.
- Learning Language from Perceptual Context
[Details] [PDF] [Slides]
David L. Chen
December 2009. Ph.D. proposal, Department of Computer Sciences, University of Texas at Austin.
- Activity Retrieval in Closed Captioned Videos
[Details] [PDF]
Sonal Gupta
Masters Thesis, Department of Computer Sciences, University of Texas at Austin, August 2009. 64 pages.
- Using Closed Captions to Train Activity Recognizers that Improve Video Retrieval
[Details] [PDF]
Sonal Gupta and Raymond Mooney
In Proceedings of the CVPR-09 Workshop on Visual and Contextual Learning from Annotated Images and Videos (VCL), Miami, FL, June 2009.
- Watch, Listen & Learn: Co-training on Captioned Images and Videos
[Details] [PDF]
Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney
In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD), 457--472, Antwerp Belgium, September 2008.
- Learning to Sportscast: A Test of Grounded Language Acquisition
[Details] [PDF] [Slides] [Video]
David L. Chen and Raymond J. Mooney
In Proceedings of the 25th International Conference on Machine Learning (ICML), Helsinki, Finland, July 2008.
- Learning to Connect Language and Perception
[Details] [PDF]
Raymond J. Mooney
In Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), 1598--1601, Chicago, IL, July 2008. Senior Member Paper.
- Learning Language Semantics from Ambiguous Supervision
[Details] [PDF]
Rohit J. Kate and Raymond J. Mooney
In Proceedings of the 22nd Conference on Artificial Intelligence (AAAI-07), 895-900, Vancouver, Canada, July 2007.
- Learning Language from Perceptual Context: A Challenge Problem for AI
[Details] [PDF]
Raymond J. Mooney
In Proceedings of the 2006 AAAI Fellows Symposium, Boston, MA, July 2006.
