CS371N: Natural Language Processing (Fall 2023)

Instructor: Greg Durrett, gdurrett@cs.utexas.edu
Lecture: Tuesday and Thursday 9:30am - 10:45am, JGB 2.218
Instructor Office Hours: Tuesdays 5pm, Thursdays 3pm, GDC 3.812 and on Zoom (hybrid; see Canvas for link)
TAs: Manya Wadhwa, Jennifer Mickel
TA Office Hours:

See Canvas for a link to the discussion board (Ed Discussion)

[Link to syllabus with detailed course policies]


This course provides an introduction to modern natural language processing using machine learning and deep learning approaches. Content includes linguistics fundamentals (syntax, semantics, distributional properties of language), machine learning models (classifiers, sequence taggers, deep learning, language models), key algorithms for inference, and applications to a range of problems. Students will get hands-on experience building systems to do tasks including text classification, language modeling, and textual entailment.


Course Details

The course lectures will be delivered in a traditional, in-person format. Recordings will be made available via the LecturesOnline service for students to browse after class. All course materials will be posted on this website. Note that additional pre-recorded video content overlapping with the concepts in this course is available on the CS388 website.

The exam will be given in-person. Contact the instructor ASAP if this poses a problem for you.

Syllabus [Clickable link with important information about the course policies. The current page you are on is NOT the syllabus]


Assignment 0: Warmup (ungraded) [nyt dataset] [tokenizer.py] [solutions]

Assignment 1: Sentiment Classification (due September 7, 11:59pm) [Code and data]

Assignment 2: Feedforward Neural Networks and Optimization (due September 21, 11:59pm) [Code and data]

Assignment 3: Transformer Language Modeling (due October 5, 11:59pm) [Code and data]

Assignment 4: Sequence Modeling and Parsing (due October 17, 11:59pm)

Midterm (topics) [midterm in-class on Thursday, October 19] [fall 2022 midterm / solutions, fall 2021 (longer than this one will be) midterm / solutions, fall 2020 (longer than this one will be) midterm / solutions]

Assignment 5: Factuality of ChatGPT (due November 7, 11:59pm) [Code and data]

Final Project: Dataset Artifacts [instructions], [Code], [instructions for independent projects] (custom proposals due October 27)

Readings: Textbook readings are assigned to complement the material discussed in lecture. You may find it useful to do these readings before lecture as preparation or after lecture to review, but you are not expected to know everything discussed in the textbook if it isn't covered in lecture. Paper readings are intended to supplement the course material if you are interested in diving deeper on particular topics. Bold readings and videos are most central to the course content; it's recommended that you look at these.

The chief text in this course is Eisenstein: Natural Language Processing, available as a free PDF online. (Another generally useful NLP book is Jurafsky and Martin: Speech and Language Processing (3rd ed. draft), with many draft chapters available for free online; however, we will not be using it much for this course.)

Schedule (subject to change through the first day of classes)

Date Topics Readings Assignments
Aug 22 Introduction [4pp] A0 out (ungraded)
Aug 24 Classification 1: Features, Perceptron Classification lecture note
Perceptron Loss (VIDEO)
Eisenstein 2.0, 2.1, 2.3.1, 4.1, 4.3
A1 out
Aug 29 Classification 2: Logistic Regression Classification lecture note
Optimization (VIDEO)
Jurafsky and Martin 5.0-5.3
Aug 31 Classification 3: Multiclass, Sentiment, Authorship (slides: [1pp] [4pp]) Multiclass lecture note
Eisenstein 2.4.1, 2.5, 2.6, 4.2
Socher+13 Sentiment
Schwartz+13 Authorship
Sept 5 Fairness / Neural 1: Feedforward, Backpropagation [4pp] (handwritten notes: pdf) Fairness (VIDEO)
HutchinsonMitchell18 Fairness
Eisenstein 3.0-3.3
Goldberg 4
Sept 7 Neural 2: Implementation, Word embeddings intro [4pp] Neural Net Optimization (VIDEO)
Eisenstein 3.3
Goldberg 3, 6
Iyyer+15 DANs
Init and backprop
A1 due/A2 out
Sept 12 Neural 3: Word embeddings [4pp] (notes: [pdf]) Eisenstein 14.5-14.6
Goldberg 5
Mikolov+13 word2vec
Pennington+14 GloVe
Sept 14 Neural 4: Bias, multilingual [4pp] Bolukbasi+16 Gender
Gonen+19 Debiasing
Ammar+16 Xlingual embeddings
Mikolov+13 Word translation
Sept 19 LM 1: N-grams, RNNs Eisenstein 6.1-6.2
Sept 21 LM 2: Attention, Self-attention Luong+15 Attention
Vaswani+17 Transformers
Alammar Illustrated Transformer
A2 due / A3 out
Sept 26 LM 3: Transformers, Implementation [handwritten notes] (slides: [1pp] [4pp]) Vaswani+17 Transformers
Alammar Illustrated Transformer
Sept 28 LM 4: Transformer Language Models / Pre-training 1: Encoders (BERT) [4pp] Kaplan+20 Scaling Laws
Peters+18 ELMo
Devlin+19 BERT
Alammar Illustrated BERT
Oct 3 Pre-training 2: Decoders (GPT), Tokenization, Decoding Methods [4pp] BostromDurrett20 Tokenizers
Radford+19 GPT2
Brown+20 GPT3
Holtzman+19 Nucleus Sampling
Oct 5 Sequence 1: Tagging, POS, HMMs Eisenstein 7.1-7.4, 8.1 A3 due/A4 out
Oct 10 Sequence 2: HMMs, Viterbi Viterbi lecture note
Eisenstein 7.1-7.4
Oct 12 Trees 1: PCFGs, CKY (slides: [1pp] [4pp]) Eisenstein 10.1-3, 10.4.1
Oct 17 Trees 2: Dependency / Midterm review [4pp] Eisenstein 11.3-4
KleinManning03 Unlexicalized
A4 due
Oct 19 Midterm (TBD)
Oct 24 Understanding GPT3 1: Prompting GPT-3, Factuality [4pp] Zhao+21 Calibrate Before Use
Min+22 Rethinking Demonstrations
Gonen+22 Demystifying Prompts
Min+23 FActScore
Gao+22 RARR
Olson+22 Induction Heads
A5 out
Oct 26 Understanding GPT3 2: Rationales, Chain-of-thought [4pp] Camburu+18 e-SNLI
Wei+22 CoT
YeDurrett22 Unreliability
Kojima+22 Step-by-step
Gao+22 Program-aided
Ye+22 Complementary
Custom FP proposals due Oct 27
Oct 31 Understanding GPT3 3: Instruction tuning, RL in NLP [4pp] Sanh+21 T0
Liu+21 Prompting
Chung+22 Flan-PaLM
Ouyang+22 Human Feedback
Ramamurthy+22 RL for NLP
Roller+20 Facebook Blender
Gehman+20 Toxicity
Thoppilan+22 LaMDA
Nov 2 Understanding NNs 1: Dataset Bias [4pp] Gururangan+18 Artifacts
McCoy+19 Right
Gardner+20 Contrast
Swayamdipta+20 Cartography
Utama+20 Debiasing
Nov 7 Understanding NNs 2: Interpretability [4pp] Lipton+16 Mythos
Ribeiro+16 LIME
Simonyan+13 Visualizing
Sundararajan+17 Int Grad
Bansal+20 Whole Exceed Parts
Interpretation Tutorial
A5 due / FP out
Nov 9 Guest Lecture: Eunsol Choi, Question Answering [4pp]
Nov 14 Machine Translation, Multilinguality [4pp]
Nov 16 Language Grounding [4pp]
Nov 28 No class: project clinic
Nov 30 Wrapup + Ethics [4pp] HovySpruit16 Social Impact of NLP
Zhao+17 Bias Amplification
Rudinger+18 Gender Bias in Coref
BenderGebru+21 Stochastic Parrots
Gebru+18 Datasheets for Datasets
Raji+20 Auditing
FP due Dec 8