CS378: Natural Language Processing (Fall 2020)

Instructor: Greg Durrett, gdurrett@cs.utexas.edu
Lecture: Tuesday and Thursday 9:30am - 11:00am, on Zoom (see Canvas for Zoom links)
Instructor Office Hours: Tuesday 11am-12pm, Wednesday 4pm-5pm (see Canvas for Zoom links)
TA: Tanya Goyal; Proctor: Shivang Singh
TA Office Hours (see Canvas for Zoom links):

Piazza Piazza

Description

This course provides an introduction to modern natural language processing using machine learning and deep learning approaches. Content includes linguistics fundamentals (syntax, semantics, distributional properties of language), machine learning models (classifiers, sequence taggers, deep learning models), key algorithms for inference, and applications to a range of problems. Students will get hands-on experience building systems to do tasks including text classification, syntactic analysis, language modeling, and language generation.

Requirements

Syllabus

Detailed syllabus with course policies

Assignments: The assignments are largely similar to those from last year, but they are subject to changes in both specific questions as well as framework code.

Assignment 0: Warmup [nyt dataset] [tokenizer.py]

Assignment 1: Sentiment Classification [code and dataset download]

Assignment 2: Feedforward Neural Networks and Optimization [code and dataset download]

Assignment 3: Sequence Modeling and Parsing [code and dataset on Canvas]

Take-home Midterm (topics) [midterm on Canvas] [spring 2020 midterm / solutions, spring 2019 midterm / solutions]

Assignment 4: Character Language Modeling with RNNs [code and dataset download]

Assignment 5: Machine Translation [code and dataset download]

Final Project: Independent Project (propose by Oct 27) or Question Answering [Github repo]

Readings: Textbook readings are assigned to complement the material discussed in lecture. You may find it useful to do these readings before lecture as preparation or after lecture to review, but you are not expected to know everything discussed in the textbook if it isn't covered in lecture. Paper readings are intended to supplement the course material if you are interested in diving deeper on particular topics. Bold readings and videos are most central to the course content; it's recommended that you look at these.

The chief text in this course is Eisenstein: Natural Language Processing, available as a free PDF online. For deep learning techniques, this text will be supplemented with selections from Goldberg: A Primer on Neural Network Models for Natural Language Processing. (Another generally useful NLP book is Jurafsky and Martin: Speech and Language Processing (3rd ed. draft), with many draft chapters available for free online; however, we will not be using it much for this course.)

Readings for future lectures are tentative and subject to change.

Date Topics Readings Assignments
Aug 27 Introduction [4pp] A0 out (ungraded)
Sept 1 Classification 1: Features, Perceptron Classification lecture note
Perceptron Loss (VIDEO)
perc_lecture_plot.py
Eisenstein 2.0, 2.1, 2.3.1, 4.1, 4.3
Sept 3 Classification 2: Logistic Regression Classification lecture note
Optimization (VIDEO)
Jurafsky and Martin 5.0-5.3
Sept 8 Classification 3: Sentiment, Multiclass (slides: [1pp] [4pp]) Multiclass lecture note
Fairness (VIDEO)
Eisenstein 2.4.1, 2.5, 2.6, 4.2
Pang+02
Wang+Manning12
Socher+13 Sentiment
Sept 10 Classification 4: Multiclass Log. Reg. / Neural 1: Feedforward, Backpropagation (slides: [1pp], [4pp]) Multiclass lecture note
Eisenstein 3.0-3.3
Goldberg 4
Schwartz+13 Authorship
Sept 15 Neural 2: FFNNs, Implementation, DANs [4pp] Neural Net Optimization (VIDEO)
ffnn_example.py
Eisenstein 3.3
Goldberg 3, 6
Iyyer+15 DANs
Init and backprop
A1 due/A2 out
Sept 17 Neural 3: Word embeddings Eisenstein 14.5-14.6
Goldberg 5
Mikolov+13 word2vec
Pennington+14 GloVe
Sept 22 Neural 4: Bias in embeddings (slides: [1pp], [4pp]) / Sequence 1: Tagging, POS Other embedding methods (VIDEO)
Bolukbasi+16 Bias
Eisenstein 8.1
Sept 24 Sequence 2: Tagging as Classification, HMMs Viterbi lecture note
Eisenstein 7.1-7.4
Sept 29 Sequence 3: Viterbi, Beam Search Viterbi lecture note
HMMs for POS Tagging (VIDEO)
Eisenstein 7.4.2
A2 due/A3 out
Oct 1 Sequence 4: CRFs / Trees 1: Constituency [4pp] Eisenstein 7.5-7.6
Manning POS
Oct 6 Trees 2: PCFGs, CKY Eisenstein 10.1-3, 10.4.1
KleinManning03 Unlexicalized
Oct 8 Trees 3 (notes): Finish constituency, Dependency (slides: [1pp], [4pp]) Refining grammars (VIDEO)
Eisenstein 11.3-4
Chen and Manning14
Andor+16
Oct 13 Trees 4: Shift-Reduce; Midterm review State-of-the-art Parsers (VIDEO) A3 due
Oct 15 Take-home Midterm (NO CLASS) Midterm out Oct 14 9am, due Oct 16 5pm
Oct 20 LM 1: N-grams, RNNs Eisenstein 6.1-6.2
Oct 22 LM 2: LSTMs, Implementation (slides: [1pp] [4pp]) lstm_lecture.py
Eisenstein 6.3-6.5
Olah Understanding LSTMs
LSTM PyTorch documentation
RNNs with PyTorch
Karpathy Visualizing RNNs
A4 out
Oct 27 Finish LSTMs (slides from last time) / MT 1: Alignment, Phrase-based MT Eisenstein 18.1-18.2, 18.4
Michael Collins IBM Models 1+2
JHU slides
History of MT
Custom FP proposals due (NEW DUE DATE)
Oct 29 MT 2: Phrase-based, seq2seq (slides: [1pp], [4pp]) Eisenstein 18.2
Nov 3 Generation: Dialogue, Summarization, Ethics [OPTIONAL] [4pp] Yu+19 Gunrock
Radford+19 GPT-2
Adiwardana+20 Google Meena
Roller+20 Facebook Blender
Gehman+20 Toxicity
Nov 5 MT 3: Attention, Neural MT (slides: [1pp], [4pp]) Eisenstein 18.3
Luong+15 Attention
Wu+16 Google
SennrichZhang19 Low-resource
A4 due / A5 out
Nov 10 QA 1: Semantic representations, semantic parsing (slides: [1pp], [4pp]) Eisenstein 12
ZettlemoyerCollins05
JiaLiang16
Nov 12 QA 2: Reading comprehension Eisenstein 17.5
Stanford Attentive Reader
SQuAD
DrQA
QA span visualization
A5 due / FP out
Nov 17 QA 3 / Pre-training 1: ELMo (slides: [1pp], [4pp]) ELMo
Nov 19 Pre-training 2: Transformers / BERT (slides: [1pp], [4pp]) Transformers
Illustrated Transformer
BERT
Nov 24 NLP in other languages, cross-lingual models [4pp] Xlingual POS
Xlingual parsing
Xlingual embeddings
Multilingual sent embs
XLM-R
How multilingual is mBERT?
FP check-in due November 24
Nov 26 NO CLASS
Dec 1 Interpreting Neural Models [4pp] Ribeiro+16 LIME
Simonyan+13 Visualizing
Nguyen18 Evaluating Explanations
Rudin+18 Stop Explaining
Bansal+20 Whole Exceed Its Parts
EMNLP 2020 Interpretability tutorial
Dec 3 Wrapup + Ethics [4pp] HovySpruit2016 Social Impact of NLP
Zhao+17 Bias Amplification
Rudinger+18 Gender Bias in Coref
Gebru+18 Datasheets for Datasets
Raji+20 Auditing
FP due Dec 9