UT ML Group: Connecting Language and Perception

To truly understand language, an intelligent system must be able to connect words, phrases, and sentences to its perception of objects and events in the world. Ideally, an AI system would be able to learn language like a human child, by being exposed to utterances in a rich perceptual environment. The perceptual context would provide the necessary supervisory information, and learning the connection between language and perception would ground the system's semantic representations in its perception of the world. As a step in this direction, our research is developing systems that learn semantic parsers and language generators from sentences paired only with their perceptual context. It is part of our research on natural language learning. Our research on this topic is currently supported by the National Science Foundation through grant IIS-0712097.

Related Demos:

Publications

  1. Activity Retrieval in Closed Captioned Videos [Abstract] [PDF]
    Sonal Gupta
    M.A. thesis, Department of Computer Sciences, University of Texas at Austin, August 2009.
    64 pages.

  2. Using Closed Captions to Train Activity Recognizers that Improve Video Retrieval [Abstract] [PDF]
    Sonal Gupta and Raymond Mooney
    In Proceedings of the CVPR-09 Workshop on Visual and Contextual Learning from Annotated Images and Videos (VCL), Miami, FL, June 2009

  3. Watch, Listen & Learn: Co-training on Captioned Images and Videos [Abstract] [PDF]
    Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney
    In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD) , Antwerp Belgium, pp. 457-472, Sept. 2008.

  4. Learning to Sportscast: A Test of Grounded Language Acquisition [Abstract] [PDF]
    David L. Chen and Raymond J. Mooney
    In Proceedings of the 25th International Conference on Machine Learning (ICML) , Helsinki, Finland, July 2008.

  5. Learning to Connect Language and Perception [Abstract] [PDF]
    Raymond J. Mooney
    In Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), Senior Member Paper, Chicago, IL, pp. 1598-1601, July 2008.

  6. Learning Language Semantics from Ambiguous Supervision [Abstract] [PDF]
    Rohit J. Kate and Raymond J. Mooney
    In Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI-2007), Vancouver, BC, pp. 895-900, July 2007.

  7. Learning Language from Perceptual Context: A Challenge Problem for AI [Abstract] [PDF]
    Raymond J. Mooney
    In Proceedings of the 2006 AAAI Fellows Symposium, Boston, MA, July 2006.

mooney@cs.utexas.edu