CS378: Natural Language Processing (Fall 2021)

Instructor: Greg Durrett, gdurrett@cs.utexas.edu
Lecture: Tuesday and Thursday 9:30am - 11:00am, GDC 1.304 (streamed on Zoom)
Instructor Office Hours: Tuesdays 4pm-5pm, Wednesday 11am-12pm (held on Zoom)
TA: Kaj Bostrom (kaj@cs.utexas.edu), Abhilash Potluri (acpotluri@utexas.edu)
TA Office Hours:

edstem discussion board

Course Format

This course is offered in a hybrid format. Lectures will be delivered simultaneously in-person as well as over Zoom (via a Zoom call hosted on the instructor's device). We will work to provide a seamless experience for students attending in either modality. The midterm will be take-home; it will be possible to complete the course without attending in-person meetings. See the syllabus for more information.

Description

This course provides an introduction to modern natural language processing using machine learning and deep learning approaches. Content includes linguistics fundamentals (syntax, semantics, distributional properties of language), machine learning models (classifiers, sequence taggers, deep learning models), key algorithms for inference, and applications to a range of problems. Students will get hands-on experience building systems to do tasks including text classification, syntactic analysis, language modeling, and language generation.

Requirements

Syllabus

Detailed syllabus with course policies

Assignments:

Assignment 0: Warmup (ungraded) [nyt dataset] [tokenizer.py]

Assignment 1: Sentiment Classification (due September 14, 11:59pm) [Code and data]

Assignment 2: Feedforward Neural Networks and Optimization (due September 28, 11:59pm) [Code and data]

Assignment 3: Sequence Modeling and Parsing (due October 12, 11:59pm) [Code and data on Canvas]

Take-home Midterm (topics) [midterm on Canvas; out Weds October 13, 5pm; due Sat October 16, 5pm] [fall 2020 (longer than this one will be) midterm / solutions, spring 2020 midterm / solutions, spring 2019 midterm / solutions]

Assignment 4: Character Language Modeling with RNNs (due November 2, 11:59pm) [Code and data]

Assignment 5: Machine Translation (due November 9, 11:59pm) [Code and data]

Final Project

Readings: Textbook readings are assigned to complement the material discussed in lecture. You may find it useful to do these readings before lecture as preparation or after lecture to review, but you are not expected to know everything discussed in the textbook if it isn't covered in lecture. Paper readings are intended to supplement the course material if you are interested in diving deeper on particular topics. Bold readings and videos are most central to the course content; it's recommended that you look at these.

The chief text in this course is Eisenstein: Natural Language Processing, available as a free PDF online. (Another generally useful NLP book is Jurafsky and Martin: Speech and Language Processing (3rd ed. draft), with many draft chapters available for free online; however, we will not be using it much for this course.)

Schedule

Date Topics Readings Assignments
Aug 26 Introduction [4pp] A0 out (ungraded)
Aug 31 Classification 1: Features, Perceptron Classification lecture note
Eisenstein 2.0, 2.1, 2.3.1, 4.1, 4.3
A1 out
Sept 2 Classification 2: Logistic Regression, Optimization (slides: [1pp], [4pp]) Classification lecture note
Perceptron Loss (VIDEO)
perc_lecture_plot.py
Jurafsky and Martin 5.0-5.3 Pang+02
Wang+Manning12
Socher+13 Sentiment
Sept 7 Classification 3: Multiclass Multiclass lecture note
Eisenstein 2.4.1, 2.5, 2.6, 4.2
Sept 9 Classification 4: Examples, Fairness [4pp] / Neural 1 Goldberg 4
Schwartz+13 Authorship
Hutchinson and Mitchell18 Fairness
Eisenstein 3.0-3.3
Sept 14 Neural 2: Implementation, Word embeddings intro [4pp] Neural Net Optimization (VIDEO)
ffnn_example.py
Eisenstein 3.3
Goldberg 3, 6
Iyyer+15 DANs
Init and backprop
A1 due/A2 out
Sept 16 Neural 3: Word embeddings Other embedding methods (VIDEO)
Eisenstein 14.5-14.6
Goldberg 5
Mikolov+13 word2vec
Pennington+14 GloVe
Fairness response due
Sept 21 Neural 4: Bias in embeddings, multilingual embeddings [4pp] Bolukbasi+16 Gender
Gonen+19 Debiasing
Ammar+16 Xlingual embeddings
Mikolov+13 Word translation
Sept 23 Sequence 1: Tagging, POS, HMMs Eisenstein 7.1-7.4, 8.1
Sept 28 Sequence 2: HMMs, Viterbi Viterbi lecture note
Eisenstein 7.1-7.4
A2 due, Bias in embeddings response due, A3 out
Sept 30 Sequence 3: Viterbi, Beam Search, POS, CRFs/NER (slides: [1pp] [4pp]) Viterbi lecture note
Eisenstein 7.5-7.6
Manning POS
Oct 5 Trees 1: PCFGs, CKY (slides: [1pp] [4pp]) Eisenstein 10.1-3, 10.4.1
Oct 7 Trees 2: Dependency (slides: [1pp] [4pp]) Eisenstein 11.3-4
KleinManning03 Unlexicalized
Oct 12 Trees 3: Shift-reduce / Midterm review State-of-the-art Parsers (VIDEO)
Chen and Manning14
Andor+16
A3 due
Oct 14 Midterm (Take-home; NO LECTURE)
Oct 19 LM 1: N-grams, RNNs Eisenstein 6.1-6.2 A4 out
Oct 21 LM 2: Implementation (slides: [1pp], [4pp]) lstm_lecture.py
Eisenstein 6.3-6.5
Olah Understanding LSTMs
LSTM PyTorch documentation
RNNs with PyTorch
Karpathy Visualizing RNNs
Oct 26 MT 1: Alignment, Phrase-based MT Eisenstein 18.1-18.2, 18.4
Michael Collins IBM Models 1+2
JHU slides
History of MT
Custom FP proposals due
Oct 28 MT 2: Seq2seq, attention Eisenstein 18.2-18.3
Luong+15 Attention
Nov 2 MT 3: Systems, Transformers [4pp] (notes) Luong+15 Attention
Wu+16 Google
Chen+18 Google
SennrichZhang19 Low-resource
Vaswani+17 Transformers
Alammar Illustrated Transformer
A4 due / A5 out
Nov 4 Pre-training: ELMo, BERT [4pp] Peters+18 ELMo
Devlin+19 BERT
Alammar Illustrated BERT
Nov 9 Understanding NNs 1: Dataset Bias [4pp] Gururangan+18 Artifacts
McCoy+19 Right
Gardner+20 Contrast
Swayamdipta+20 Cartography
Utama+20 Debiasing
A5 due / FP out
Nov 11 Understanding NNs 2: Interpretability [4pp] Lipton+16 Mythos
Ribeiro+16 LIME
Simonyan+13 Visualizing
Sundararajan+17 Int Grad
Nguyen18 Evaluating Explanations
Interpretation Tutorial
Nov 16 Question Answering [4pp] Eisenstein 12
Chen+17 DrQA
Lee+19 Latent Retrieval
Guu+20 REALM
Kwiatkowski+19 NQ
Nov 18 Generation 1: BART/T5, GPT-3, Prompting [4pp] Raffel+19 T5
Lewis+19 BART
Radford+19 GPT2
Brown+20 GPT3
Sanh+21 T0
Liu+21 Prompting
Nov 23 Generation 2: Sampling, Dialogue, Ethics [4pp] Holtzman+19 Nucleus Sampling
Akoury+20 Storium
Adiwardana+20 Google Meena
Roller+20 Facebook Blender
Gehman+20 Toxicity
FP check-ins due
Nov 25 NO CLASS
Nov 30 Multilingual/Cross-lingual [4pp]
Dec 2 Wrapup + Ethics [4pp] FP due Dec 9