CS395T · Fall 2025

Lecture Schedule

Below is the tentative schedule for the course. Note that dates and topics may change as the semester progresses.

The schedule can be found in Excel format here for paper presentation and review sign-ups.

Lecture	Date	Topic	Materials/Readings	Assignments & Deadlines
1	8/26 (T)	Introduction (Slides)	AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery (Novikov et al., 2025) DeepSeek-V3 Technical Report (2024) AlphaGo
2	8/28 (Th)	Abstract NN & gradient computation (Slides)	Haykin Textbook - Introduction; Chapter 1: Perceptron; Chapter 4: Multi-layer Perceptrons (MLPs) Who invented backpropagation? (Schmidhuber 2014) Tutorial on Autograd	Introduction post/survey due
3	9/2 (T)	DNNs, CNNs, RNNs, practical issues (Slides)	Deep Residual Learning for Image Recognition (ResNet) (Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015) Haykin Textbook - Chapter 15: RNNs Neural Machine Translation by Jointly Learning to Align & Translate (Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014)	Presentation sign-up due (one paper presentation, one paper review)
4	9/4 (Th)	Attention, Transformers, LLMs	Attention Is All You Need (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017) Understanding and Coding Self-Attention (Sebastian Raschka)	Assignment 1 handed out
5	9/9 (T)	Presentations: Optimizing Attention	FlashAttention (Ma et al., 2022) FlashAttention-3 (Shah et al., 2024) PagedAttention (Kwon et al., 2023)
6	9/11 (Th)	Monte Carlo methods & variance reduction	Barto & Sutton – Ch 5 Monte Carlo Methods
7	9/16 (T)	Presentations: DeepSeek	DeepSeek-V3 Technical Report (2024)
8	9/18 (Th)	Markov Decision Processes (MDPs)	Barto & Sutton – Ch 3 MDPs	Assignment 1 due
9	9/23 (T)	Sampling (TD(0), TD(n), MC, Q-learning, MC)	Barto & Sutton – Ch 6 Temporal Difference Learning
10	9/25 (Th)	Presentations: DQN, Double DQN, Experience Replay	Human-level Control through Deep Reinforcement Learning (Mnih et al., 2015) Deep Reinforcement Learning with Double Q-learning (van Hasselt, Guez, Silver, 2016) Hindsight Experience Replay (Andrychowicz et al., 2017)
11	9/30 (T)	Policy gradients (I): REINFORCE	Barto & Sutton – Ch 13: Policy Gradients OpenAI DRL – Intro to Policy Optimization Definitive Guide to Policy Gradients §1–2 (Matthias Lehmann, 2024) Deep RL: Pong from Pixels (Andrej Karpathy) Karpathy - Overview of Policy Gradients Video
12	10/2 (Th)	Presentations: RL Environments	ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning (Marcin Miłkowski, 2016) Gymnasium MuJoCo Playground
13	10/7 (T)	Policy gradients (II): Baseline methods	Tutorial TBD	Assignment 2 handed out
14	10/9 (Th)	Presentations: Actor-Critics & DDPG	Asynchronous Methods for Deep Reinforcement Learning (Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016) Continuous Control with Deep Reinforcement Learning (DDPG) (Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, 2015)	Project Proposal due
15	10/14 (T)	Policy gradients (III): Trust-region methods	Definitive Guide to Policy Gradients §3-4 (Matthias Lehmann, 2024) Trust Region Policy Optimization (TRPO) (John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz, 2015) Proximal Policy Optimization (PPO) (John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017)	Project ideas due (meeting required)
16	10/16 (Th)	Presentations: Policy Optimization Methods	DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
17	10/21 (T)	Reinforcement Learning from Human Feedback (RLHF) and Imitation Learning	Reinforcement Learning from Human Feedback: Progress and Challenges (John Schulman, 2023) Training language models to follow instructions with human feedback (Long Ouyang et al., 2022) Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (Jan Brauner et al., 2023) Tutorial on Imitation Learning	Assignment 2 due
18	10/23 (Th)	Presentations: RLHF	Deep Reinforcement Learning from Human Preferences (Paul F. Christiano et al., 2017) RLTF: Reinforcement Learning from Unit Test Feedback (Liu et al., 2023)
19	10/28 (T)	Presentations: Imitation Learning	Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (Rajeswaran, Kumar, Gupta, Vezzani, Schulman, Todorov, Levine, 2017) Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (Finn et al. 2016)
20	10/30 (Th)	Evolutionary Computation	Neuroevolution (Risto Miikkulainen, 2022) Evolving neural networks through augmenting topologies (Kenneth O. Stanley, Risto Miikkulainen, 2002) Evolutionary Policy Optimization (Lain Mustafaoglu, Keshav Pingali, Risto Miikkulainen, 2025) Tutorial TBD	Assignment 3 handed out
21	11/4 (T)	Presentations: Applications of Evolutionary Computation	Evolution Strategies as a Scalable Alternative to Reinforcement Learning (Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever, 2017) Accelerating Evolution Through Gene Masking and Distributed Search (Hormoz Shahrzad, Risto Miikkulainen, 2023) AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery (Novikov et al., 2025)
21	11/4 (T)	Imitation Learning	Tutorial on Imitation Learning
22	11/6 (Th)	Presentations: ML for Systems	Meta Large Language Model Compiler: Foundation Models of Compiler Optimization (Cummins et al., 2024) PrefixRL: Optimization of Parallel Prefix Circuits using Deep Reinforcement Learning (Roy et al. 2022)
23	11/11 (T)	Presentations: ML for Systems (II)	The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition (Lange et al., 2025) KernelBench: Can LLMs Write Efficient GPU Kernels? (Ouyang et al., 2025)	Assignment 3 due
24	11/13 (Th)	Parallel/Distributed RL	Tutorial TBD	Project check-in (meeting required)
25	11/18 (T)	Presentations: Large-scale distributed RL	IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (Espeholt et al., 2018) SEED RL: Scalable & Efficient Deep-RL (Espeholt et al. 2019) Recurrent Experience Replay in Distributed Reinforcement Learning (ICLR 2019)
26	11/20 (Th)	Presentations: Other RL Topics	The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (Yu et al., 2022) Decision Transformer: Reinforcement Learning via Sequence Modeling (Chen et al., 2021)
THANKSGIVING BREAK
27	12/2 (T)	Project presentations
28	12/4 (Th)	Project presentations		Final project paper due