CS 395T: Grounded Natural Language Processing

CS 395T:
Grounded Natural Language Processing

Instructor

Raymond J. Mooney

Teaching Assistant

Priyanka Mandikal

Time and Place

Spring 2024, Mon & Wed, 2-3:30pm, GDC 2.210

Office Hours

Instructor: Mon 3:30-4:30pm & Wed, 11am-noon, GDC 3.806
TA: Mondays from 12 - 2 pm on Zoom via email appointment.

General Course Information

This course will be a graduate research seminar in grounded natural language processing (GNLP), a subarea of AI that studies the connection between natural language and perception and action in the world. It makes connections between natural language processing (NLP) and computer vision, robotics, and computer graphics. Almost all work in the area uses machine learning to learn the connection between language and perception and/or action from some form of multi-modal training data.

The class will mostly revolve around reading both seminal and recent research papers in the area. Each student will be required to prepare organized oral presentations of papers and participate in class discussion about these papers. Finally, it will involve conducting an original research project in the area, writing a paper on this project suitable for submitting to a research conference, and presenting an oral presentation on the project.

Course Details

Lecture Slides

Watch my AAAI 2013 Keynote talk "Grounded Language Learning" on video also see the slides since these are not always visible in the video.
Visually Grounded Language: Past, Present, and Future, NIPS-2017 Workshop on Visually-Grounded Interaction and Language (ViGIL), Long Beach, CA, Dec. 8, 2017.
Dialog with Robots: Perceptually Grounded Communication with Lifelong Learning, SIGDIAL-2021 invited keynote talk, July 29, 2021.
Using Natural Language Processing to Aid Computer Vision, University of Washington and Microsoft Research Workshop on Understanding Situated Language in Everyday Life, Union, WA, July 22-25, 2013
A Review of Work on Natural Language Navigation Instructions, Combined Workshop on Spatial Language Understanding (SpLU) & Grounded Communication for Robotics (RoboNLP), NAACL 2019.