CS388: Natural Language Processing (Fall 2019)

Instructor: Greg Durrett, gdurrett@cs.utexas.edu
Lecture: Tuesday and Thursday 12:30pm - 2:00pm, GDC 4.302
Instructor Office Hours: Wednesday 4pm-5pm, Thursday 2pm-3pm, GDC 3.812
TA: Uday Kusupati, kusupatiuday@gmail.com
TA Office Hours: Monday 12pm-1pm, Tuesday 11am-12pm, GDC 1.302

Piazza Piazza

Description

This class is a graduate-level introduction to Natural Language Processing (NLP), the study of computing systems that can process, understand, or communicate in human language. The course covers fundamental approaches, largely machine learning and deep learning, used across the field of NLP as well as a comprehensive set of NLP tasks both historical and contemporary. Techniques studied include Bayes nets, conditional random fields, structured max-margin, tree-structured models, recurrent and convolutional neural networks, and attention mechanisms. Problems range from syntax (part-of-speech tagging, parsing) to semantics (lexical semantics, question answering, grounding) and include various applications such as summarization, machine translation, information extraction, and dialogue systems. Programming assignments throughout the semester involve building scalable machine learning systems for various NLP tasks.

Requirements

Syllabus

Detailed syllabus with course policies

Assignments: There are four programming assignments that require implementing models discussed in class. Framework code in Python and datasets will be provided. In addition, there is an open-ended final project to be done in teams of 2 (preferred) or individually. This project should constitute novel exploration beyond directly implementing concepts from lecture and should result in a report that roughly reads like an NLP/ML conference submission in terms of presentation and scope.

Mini 1: Classification for Person Name Detection [code and dataset download]

Project 1: CRF tagging for NER [code and dataset download] [great project writeup] [good project writeup]

Mini 2: Neural Networks for Sentiment Analysis [code and dataset download]

Project 2: Encoder-Decoder Models for Question Answering [code and dataset download]

Final Project [tips for academic writing]

Readings: Readings are purely optional and intended to supplement lecture and give you another view of the material. Two main sources will be used:

Readings for future lectures are tentative and subject to change.

Date Topics Readings Assignments
Aug 29 Introduction [4pp] Mini1 out
Sept 3 Binary classification [4pp] Eisenstein 2.0-2.5, 4.2-4.4.1, JM 4, JM 5.0-5.5
Sept 5 Multiclass classification [4pp] Eisenstein 4.2, JM 5.6, Structured SVM secs 1-2
Sept 10 Sequence Models 1: HMMs (Guest Lecture: Ray Mooney) [4pp] Eisenstein 7.0-7.4, 8.1, JM 8, Manning POS, Viterbi algorithm lecture note Mini1 due / Proj1 out
Sept 12 Sequence Models 2: CRFs [4pp] Eisenstein 7.5, 8.3, Sutton CRFs 2.3, 2.6.1, Wallach CRFs tutorial, Illinois NER
Sept 17 NN1: Feedforward [4pp] Eisenstein 3.0-3.3, Goldberg 1-4, 6, NLP with FFNNs, DANs, ffnn_example.py
Sept 19 NN2: Word embeddings [4pp] Eisenstein 3.3.4, 14.5-14.6, JM 6, Goldberg 5, word2vec, Levy, GloVe, fastText, Bolukbasi, Gonen Debiasing
Sept 24 NN3: RNNs [4pp] JM 9.1-9.4, Goldberg 10-11, Karpathy Proj1 due
Sept 26 NN4: Language Modeling and Pretraining [4pp] Eisenstein 6, JM 9.2.1, ELMo, Frozen vs fine-tuned Mini2 out
Oct 1 NN5: Interpretability, Neural CRFs [4pp] LIME, Simonyan Visualizing, Nguyen Evaluating Explanations, Collobert and Weston, Neural NER
Oct 3 Trees 1: Constituency, PCFGs [4pp] Eisenstein 10.0-10.5, JM 12.1-12.6, 12.8, Structural, Lexicalized, Stern, Kitaev
Oct 8 Trees 2: Dependency 1 [4pp] Eisenstein 11.1-11.2, JM 13.1-13.3, 13.5, Dozat, HPSG Mini2 due / FP out
Oct 10 Trees 3: Dependency Parsers [4pp] Eisenstein 11.3, JM 13.4, Parsey, Huang 2
Oct 15 Semantics 1 [4pp] Eisenstein 12, Zettlemoyer, Berant FP proposal due
Oct 17 Semantics 2 / Seq2seq 1 [4pp] Seq2seq, Jia Proj2 out
Oct 22 Seq2seq 2: Attention and Pointers [4pp] Attention, Luong Attention, Transformer
Oct 24 Machine Translation 1 [4pp] Eisenstein 18.0-18.2, HMM alignment, Pharaoh
Oct 29 Machine Translation 2 [4pp] Eisenstein 18.3-18.4, Google NMT, Attention is all you need, ByteNet
Oct 31 Pretrained Transformers / BERT [4pp] GPT, BERT, GPT2, RoBERTa, T5, BART, What does BERT look at?
Nov 5 Information Extraction / SRL [4pp] Eisenstein 13, 17, SRL, AMR parsing, Distant supervision, TextRunner, ReVerb Proj2 due
Nov 7 Question Answering 1 [4pp] SQuAD, BiDAF
Nov 12 Question Answering 2: Multi-hop etc. [4pp] Adversarial, QA on Wikipedia, Latent Retrieval, Multi-hop reasoning, NaturalQuestions
Nov 14 Dialogue [4pp] RNN chatbots, Diversity, PersonaChat, Wizard of Wikipedia, Goal-oriented, Latent Intention, QuAC, ShARC
Nov 19 Summarization [4pp] Eisenstein 19, MMR, Gillick, Sentence compression, SummaRuNNER, Pointer, Pre-trained transformer, Critical Evaluation
Nov 21 Multilinguality and morphology [4pp] Xlingual POS, Xlingual parsing, Xlingual embeddings, Multilingual word embs, Multilingual sent embs, How multilingual is mBERT?
Nov 26 Wrapup + Ethics [4pp]
Nov 28 No class (Thanksgiving)
Dec 3 Project presentations 1
Dec 5 Project presentations 2
Dec 13 FP due