CS378: Natural Language Processing (Fall 2021)
NOTE: This page is for an old semester of this class
Instructor: Greg Durrett, gdurrett@cs.utexas.edu
Lecture: Tuesday and Thursday 9:30am - 11:00am, GDC 1.304 (streamed on Zoom)
Instructor Office Hours: Tuesdays 4pm-5pm, Wednesday 11am-12pm (held on Zoom)
TA: Kaj Bostrom (kaj@cs.utexas.edu), Abhilash Potluri (acpotluri@utexas.edu)
TA Office Hours:
- Monday 11am-12pm (Kaj, TA desk 5 in GDC basement)
- Monday 5pm-6pm (Abhilash, Zoom)
- Thursday 5pm-6pm (Kaj, Zoom)
- Friday 10am-11am (Abhilash, TA desk area)
edstem discussion board
Course Format
This course is offered in a hybrid format. Lectures will be delivered simultaneously in-person as well as over Zoom (via a Zoom call hosted on the instructor's device).
We will work to provide a seamless experience for students attending in either modality.
The midterm will be take-home; it will be possible to complete the course without attending
in-person meetings. See the syllabus for more information.
Description
This course provides an introduction to modern natural language processing
using machine learning and deep learning approaches. Content includes
linguistics fundamentals (syntax, semantics, distributional properties of
language), machine learning models (classifiers, sequence taggers, deep
learning models), key algorithms for inference, and applications to a range of
problems. Students will get hands-on experience building systems to do tasks
including text classification, syntactic analysis, language modeling, and language generation.
Requirements
- CS 429
- Recommended: CS 331, familiarity with probability and linear algebra, programming experience in Python
- Helpful: Exposure to AI and machine learning (e.g., CS 342/343/363)
Detailed syllabus with course policies
Assignments:
Assignment 0: Warmup (ungraded) [nyt dataset] [tokenizer.py]
Assignment 1: Sentiment Classification (due September 14, 11:59pm) [Code and data]
Assignment 2: Feedforward Neural Networks and Optimization (due September 28, 11:59pm) [Code and data]
Assignment 3: Sequence Modeling and Parsing (due October 12, 11:59pm) [Code and data on Canvas]
Take-home Midterm (topics) [midterm on Canvas; out Weds October 13, 5pm; due Sat October 16, 5pm] [fall 2020 (longer than this one will be) midterm / solutions, spring 2020 midterm / solutions, spring 2019 midterm / solutions]
Assignment 4: Character Language Modeling with RNNs (due November 2, 11:59pm) [Code and data]
Assignment 5: Machine Translation (due November 9, 11:59pm) [Code and data]
Final Project
Readings: Textbook readings are assigned to complement the material discussed in lecture. You may find it useful
to do these readings before lecture as preparation or after lecture to review, but you are not expected to know everything discussed
in the textbook if it isn't covered in lecture.
Paper readings are intended to supplement the course material if you are interested in diving deeper on particular topics.
Bold readings and videos are most central to the course content; it's recommended that you look at these.
The chief text in this course is Eisenstein: Natural Language Processing,
available as a free PDF online.
(Another generally useful NLP book is Jurafsky and Martin: Speech and Language Processing (3rd ed. draft), with many draft chapters available for free online; however,
we will not be using it much for this course.)
Schedule
| Date |
Topics |
Readings |
Assignments |
| Aug 26 |
Introduction [4pp] |
|
A0 out (ungraded) |
| Aug 31 |
Classification 1: Features, Perceptron
|
Classification lecture note
Eisenstein 2.0, 2.1, 2.3.1, 4.1, 4.3
|
A1 out |
| Sept 2 |
Classification 2: Logistic Regression, Optimization (slides: [1pp], [4pp]) |
Classification lecture note
Perceptron Loss (VIDEO)
perc_lecture_plot.py
Jurafsky and Martin 5.0-5.3
Pang+02
Wang+Manning12
Socher+13 Sentiment
|
|
| Sept 7 |
Classification 3: Multiclass |
Multiclass lecture note
Eisenstein 2.4.1, 2.5, 2.6, 4.2
|
|
| Sept 9 |
Classification 4: Examples, Fairness [4pp] / Neural 1 |
Goldberg 4
Schwartz+13 Authorship
Hutchinson and Mitchell18 Fairness
Eisenstein 3.0-3.3
|
|
| Sept 14 |
Neural 2: Implementation, Word embeddings intro [4pp] |
Neural Net Optimization (VIDEO)
ffnn_example.py
Eisenstein 3.3
Goldberg 3, 6
Iyyer+15 DANs
Init and backprop
|
A1 due/A2 out |
| Sept 16 |
Neural 3: Word embeddings |
Other embedding methods (VIDEO)
Eisenstein 14.5-14.6
Goldberg 5
Mikolov+13 word2vec
Pennington+14 GloVe
|
Fairness response due |
| Sept 21 |
Neural 4: Bias in embeddings, multilingual embeddings [4pp] |
Bolukbasi+16 Gender
Gonen+19 Debiasing
Ammar+16 Xlingual embeddings
Mikolov+13 Word translation
|
|
| Sept 23 |
Sequence 1: Tagging, POS, HMMs |
Eisenstein 7.1-7.4, 8.1
|
|
| Sept 28 |
Sequence 2: HMMs, Viterbi |
Viterbi lecture note
Eisenstein 7.1-7.4
|
A2 due, Bias in embeddings response due, A3 out |
| Sept 30 |
Sequence 3: Viterbi, Beam Search, POS, CRFs/NER (slides: [1pp] [4pp]) |
Viterbi lecture note
Eisenstein 7.5-7.6
Manning POS
|
|
| Oct 5 |
Trees 1: PCFGs, CKY (slides: [1pp] [4pp]) |
Eisenstein 10.1-3, 10.4.1
|
|
| Oct 7 |
Trees 2: Dependency (slides: [1pp] [4pp]) |
Eisenstein 11.3-4
KleinManning03 Unlexicalized
|
|
| Oct 12 |
Trees 3: Shift-reduce / Midterm review |
State-of-the-art Parsers (VIDEO)
Chen and Manning14
Andor+16
|
A3 due |
| Oct 14 |
Midterm (Take-home; NO LECTURE) |
|
|
| Oct 19 |
LM 1: N-grams, RNNs |
Eisenstein 6.1-6.2
|
A4 out |
| Oct 21 |
LM 2: Implementation (slides: [1pp], [4pp]) |
lstm_lecture.py
Eisenstein 6.3-6.5
Olah Understanding LSTMs
LSTM PyTorch documentation
RNNs with PyTorch
Karpathy Visualizing RNNs
|
|
| Oct 26 |
MT 1: Alignment, Phrase-based MT |
Eisenstein 18.1-18.2, 18.4
Michael Collins IBM Models 1+2
JHU slides
History of MT
|
Custom FP proposals due |
| Oct 28 |
MT 2: Seq2seq, attention |
Eisenstein 18.2-18.3
Luong+15 Attention
|
|
| Nov 2 |
MT 3: Systems, Transformers [4pp] (notes) |
Luong+15 Attention
Wu+16 Google
Chen+18 Google
SennrichZhang19 Low-resource
Vaswani+17 Transformers
Alammar Illustrated Transformer
|
A4 due / A5 out |
| Nov 4 |
Pre-training: ELMo, BERT [4pp] |
Peters+18 ELMo
Devlin+19 BERT
Alammar Illustrated BERT
|
|
| Nov 9 |
Understanding NNs 1: Dataset Bias [4pp] |
Gururangan+18 Artifacts
McCoy+19 Right
Gardner+20 Contrast
Swayamdipta+20 Cartography
Utama+20 Debiasing
|
A5 due / FP out |
| Nov 11 |
Understanding NNs 2: Interpretability [4pp] |
Lipton+16 Mythos
Ribeiro+16 LIME
Simonyan+13 Visualizing
Sundararajan+17 Int Grad
Nguyen18 Evaluating Explanations
Interpretation Tutorial
|
|
| Nov 16 |
Question Answering [4pp] |
Eisenstein 12
Chen+17 DrQA
Lee+19 Latent Retrieval
Guu+20 REALM
Kwiatkowski+19 NQ
|
|
| Nov 18 |
Generation 1: BART/T5, GPT-3, Prompting [4pp] |
Raffel+19 T5
Lewis+19 BART
Radford+19 GPT2
Brown+20 GPT3
Sanh+21 T0
Liu+21 Prompting
|
|
| Nov 23 |
Generation 2: Sampling, Dialogue, Ethics [4pp] |
Holtzman+19 Nucleus Sampling
Akoury+20 Storium
Adiwardana+20 Google Meena
Roller+20 Facebook Blender
Gehman+20 Toxicity
|
FP check-ins due |
| Nov 25 |
NO CLASS |
|
|
| Nov 30 |
Multilingual/Cross-lingual [4pp] |
Das+11 Xlingual POS
McDonald+11 Xlingual parsing
Ammar+16 Xlingual embeddings
Conneau+19 XLM-R
Pires+19 How multilingual is mBERT?
|
|
| Dec 2 |
Wrapup + Ethics [4pp] |
HovySpruit2016 Social Impact of NLP
Zhao+17 Bias Amplification
Rudinger+18 Gender Bias in Coref
BenderGebru+21 Stochastic Parrots
Gebru+18 Datasheets for Datasets
Raji+20 Auditing
|
FP due Dec 9 |