CS388: Natural Language Processing (Fall 2019)

NOTE: This page is for an old semester of this class

Instructor: Greg Durrett, gdurrett@cs.utexas.edu
Lecture: Tuesday and Thursday 12:30pm - 2:00pm, GDC 4.302
Instructor Office Hours: Wednesday 4pm-5pm, Thursday 2pm-3pm, GDC 3.812
TA: Uday Kusupati, kusupatiuday@gmail.com
TA Office Hours: Monday 12pm-1pm, Tuesday 11am-12pm, GDC 1.302

Piazza

Description

This class is a graduate-level introduction to Natural Language Processing (NLP), the study of computing systems that can process, understand, or communicate in human language. The course covers fundamental approaches, largely machine learning and deep learning, used across the field of NLP as well as a comprehensive set of NLP tasks both historical and contemporary. Techniques studied include Bayes nets, conditional random fields, structured max-margin, tree-structured models, recurrent and convolutional neural networks, and attention mechanisms. Problems range from syntax (part-of-speech tagging, parsing) to semantics (lexical semantics, question answering, grounding) and include various applications such as summarization, machine translation, information extraction, and dialogue systems. Programming assignments throughout the semester involve building scalable machine learning systems for various NLP tasks.

Requirements

• 391L - Machine Learning, 343 - Artificial Intelligence, or equivalent AI/ML course experience
• Familiarity with Python (for programming assignments)
• Additional prior exposure to discrete math, probability, linear algebra, optimization, linguistics, and NLP useful but not required

Syllabus

Detailed syllabus with course policies

Assignments: There are four programming assignments that require implementing models discussed in class. Framework code in Python and datasets will be provided. In addition, there is an open-ended final project to be done in teams of 2 (preferred) or individually. This project should constitute novel exploration beyond directly implementing concepts from lecture and should result in a report that roughly reads like an NLP/ML conference submission in terms of presentation and scope.

Mini 1: Classification for Person Name Detection [code and dataset download]

Project 1: CRF tagging for NER [code and dataset download] [great project writeup] [good project writeup]

Mini 2: Neural Networks for Sentiment Analysis [code and dataset download]

Project 2: Encoder-Decoder Models for Question Answering [code and dataset download]

Final Project [tips for academic writing]

Readings: Readings are purely optional and intended to supplement lecture and give you another view of the material. Two main sources will be used:

• Eisenstein: Natural Language Processing. I recommend this book highly: I think the presentation is overall a bit closer to mine than the presentation in Jurafsky and Martin. However, I will suggest readings from both books and you can decide for yourself which one you prefer.
• Jurafsky and Martin: Speech and Language Processing (3rd ed. draft). This is a draft textbook so chapters are available online in PDF. This is the "classic" NLP book.
• Goldberg: A Primer on Neural Network Models for Natural Language Processing

Readings for future lectures are tentative and subject to change.

Date	Topics	Readings	Assignments
Aug 29	Introduction [4pp]		Mini1 out
Sept 3	Binary classification [4pp]	Eisenstein 2.0-2.5, 4.2-4.4.1, JM 4, JM 5.0-5.5
Sept 5	Multiclass classification [4pp]	Eisenstein 4.2, JM 5.6, Structured SVM secs 1-2
Sept 10	Sequence Models 1: HMMs (Guest Lecture: Ray Mooney) [4pp]	Eisenstein 7.0-7.4, 8.1, JM 8, Manning POS, Viterbi algorithm lecture note	Mini1 due / Proj1 out
Sept 12	Sequence Models 2: CRFs [4pp]	Eisenstein 7.5, 8.3, Sutton CRFs 2.3, 2.6.1, Wallach CRFs tutorial, Illinois NER
Sept 17	NN1: Feedforward [4pp]	Eisenstein 3.0-3.3, Goldberg 1-4, 6, NLP with FFNNs, DANs, ffnn_example.py
Sept 19	NN2: Word embeddings [4pp]	Eisenstein 3.3.4, 14.5-14.6, JM 6, Goldberg 5, word2vec, Levy, GloVe, fastText, Bolukbasi, Gonen Debiasing
Sept 24	NN3: RNNs [4pp]	JM 9.1-9.4, Goldberg 10-11, Karpathy	Proj1 due
Sept 26	NN4: Language Modeling and Pretraining [4pp]	Eisenstein 6, JM 9.2.1, ELMo, Frozen vs fine-tuned	Mini2 out
Oct 1	NN5: Interpretability, Neural CRFs [4pp]	LIME, Simonyan Visualizing, Nguyen Evaluating Explanations, Collobert and Weston, Neural NER
Oct 3	Trees 1: Constituency, PCFGs [4pp]	Eisenstein 10.0-10.5, JM 12.1-12.6, 12.8, Structural, Lexicalized, Stern, Kitaev
Oct 8	Trees 2: Dependency 1 [4pp]	Eisenstein 11.1-11.2, JM 13.1-13.3, 13.5, Dozat, HPSG	Mini2 due / FP out
Oct 10	Trees 3: Dependency Parsers [4pp]	Eisenstein 11.3, JM 13.4, Parsey, Huang 2
Oct 15	Semantics 1 [4pp]	Eisenstein 12, Zettlemoyer, Berant	FP proposal due
Oct 17	Semantics 2 / Seq2seq 1 [4pp]	Seq2seq, Jia	Proj2 out
Oct 22	Seq2seq 2: Attention and Pointers [4pp]	Attention, Luong Attention, Transformer
Oct 24	Machine Translation 1 [4pp]	Eisenstein 18.0-18.2, HMM alignment, Pharaoh
Oct 29	Machine Translation 2 [4pp]	Eisenstein 18.3-18.4, Google NMT, Attention is all you need, ByteNet
Oct 31	Pretrained Transformers / BERT [4pp]	GPT, BERT, GPT2, RoBERTa, T5, BART, What does BERT look at?
Nov 5	Information Extraction / SRL [4pp]	Eisenstein 13, 17, SRL, AMR parsing, Distant supervision, TextRunner, ReVerb	Proj2 due
Nov 7	Question Answering 1 [4pp]	SQuAD, BiDAF
Nov 12	Question Answering 2: Multi-hop etc. [4pp]	Adversarial, QA on Wikipedia, Latent Retrieval, Multi-hop reasoning, NaturalQuestions
Nov 14	Dialogue [4pp]	RNN chatbots, Diversity, PersonaChat, Wizard of Wikipedia, Goal-oriented, Latent Intention, QuAC, ShARC
Nov 19	Summarization [4pp]	Eisenstein 19, MMR, Gillick, Sentence compression, SummaRuNNER, Pointer, Pre-trained transformer, Critical Evaluation
Nov 21	Multilinguality and morphology [4pp]	Xlingual POS, Xlingual parsing, Xlingual embeddings, Multilingual word embs, Multilingual sent embs, How multilingual is mBERT?
Nov 26	Wrapup + Ethics [4pp]
Nov 28	No class (Thanksgiving)
Dec 3	Project presentations 1
Dec 5	Project presentations 2
Dec 13			FP due