Fall 2025 - CS395T: Foundations of Machine Learning for Systems Researchers

Course Overview

Rapid advances in machine learning have enabled us to build systems with capabilities that were unimaginable just a few years ago, such as AlphaGo, which plays the board game Go better than any human player, DeepSeekCoder, which can generate programs in more than 300 computer languages, and AlphaEvolve, which can discover advanced algorithms. The goal of this course is to analyze the key breakthroughs that underlie these kinds of systems, and to understand how they can be used to build systems for solving other problems.

Lectures will cover three main machine learning technologies used in these systems: deep neural networks, reinforcement learning, and evolutionary computing. Unlike in machine learning courses, this material will be presented using PL/systems concepts such as dataflow analysis. Lectures will be complemented each week by student presentations of key papers in these areas from recent AI/ML conferences. Some of these papers go into greater depth in the core AI/ML technologies while others are case studies that analyze how they are deployed in systems for board games, multiplayer games, coding assistants, and algorithm discovery.

Prerequisites

Proficiency in Python
Familiarity with undergraduate-level algorithms and data structures
Familiarity with calculus, statistics, and linear algebra, and strong mathematical skills
Coursework or equivalent experience in AI and ML at the level of CS342 is strongly recommended. Practical experience with training machine learning models will be useful.

Coursework

Class presentations 25%
Class participation 10%
3-4 programming assignments 25%
A substantial term project 40%

Academic Honesty

You may discuss concepts with classmates, but all written work and programming assignments must be your own or your project team's work when teamwork is permitted. You may not search online for existing implementations of algorithms related to the programming assignments, even as a reference. Students caught cheating will automatically fail the course and will be reported to the university. If in doubt about the ethics of any particular action, talk to the instructor or the TA.

Course Staff

Dr. Keshav Pingali

pingali@cs.utexas.edu
Office: POB 4.126
OH: Tue 2–3 PM
Website

Lain Mustafaoglu

zsm@utexas.edu
OH: Wed 2–3 PM
Website

Course Outcomes

By the end of this course, students will be able to:

Build a solid foundation in the core technologies that underly modern AI/ML systems including deep neural networks, reinforcement learning, and evolutionary computing, while understanding their limitations.
Learn how these technologies are deployed in modern AI/ML systems.
Be able to evaluate and use advanced AI/ML technologies to build new systems.

Lecture Schedule

Below is the tentative schedule for the course. Note that dates and topics may change as the semester progresses.

Lecture	Date	Topic	Materials/Readings	Assignments & Deadlines
1	8/26 (T)	Introduction	AlphaEvolve: A coding agent for scientific and algorithmic discovery DeepSeekCoder/DeepSeek-R1 AlphaGo
2	8/28 (Th)	Abstract NN & gradient computation	NN Textbook – Introduction Chapter 1 – Perceptron Chapter 4 – MLPs Tutorial on Autograd	Introduction post/survey due
3	9/2	DNNs, CNNs, RNNs, practical issues	CNNs Deep Residual Learning for Image Recognition (ResNet) (Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015) RNNs NN Textbook Ch 15 – RNNs Neural Machine Translation by Jointly Learning to Align & Translate (Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014)
4	9/4	Sparsity in DNNs	Dynamic Sparsity TBD
5	9/9	Attention, Transformers, LLMs	Attention Is All You Need (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017) (Additional readings TBA)
6	9/11	Presentations Training LLMs	QLoRA: Efficient Finetuning of Quantized LLMs (Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, 2023) KernelBench: Can LLMs Write Efficient GPU Kernels? (Long Ouyang, et al., 2025)
7	9/16	Monte Carlo methods & variance reduction	Barto & Sutton – Ch 5 Monte Carlo Methods	Assignment 1 due Train a model with a PEFT technique
8	9/18	MDPs	Barto & Sutton – Ch 3 MDPs
9	9/23	Sampling (TD(0), TD(n), MC, Q-learning)	Barto & Sutton – Ch 6 Temporal Difference Learning
10	9/25	Presentations Planning (MCTS/AlphaGo)	Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm (Silver et al., 2017) Monte Carlo Tree Search: A Review of Recent Modifications and Applications (Maciej Świechowski, Konrad Godlewski, Bartosz Sawicki, Jacek Mańdziuk, 2022)
11	9/30	Policy gradients (I): REINFORCE	Barto & Sutton – Ch 13: Policy Gradients OpenAI DRL – Intro to Policy Optimization Definitive Guide to Policy Gradients §1–2 (Matthias Lehmann, 2024) Deep RL: Pong from Pixels (Andrej Karpathy) Video ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning (Marcin Miłkowski, 2016)
12	10/2	Presentations Double DQN, DQN, HER	Deep Reinforcement Learning with Double Q-learning (van Hasselt, Guez, Silver, 2016) Human-level Control through Deep Reinforcement Learning (Mnih et al., 2015) Hindsight Experience Replay (Andrychowicz et al., 2017)
13	10/7	Policy gradients (II): Baseline methods	TBD	Assignment 2 due Grid-world (≥ 2 methods) or Pong (MLP + CNN)
14	10/9	Presentations A2C/A3C & DDPG	Asynchronous Methods for Deep Reinforcement Learning (Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016) Continuous Control with Deep Reinforcement Learning (DDPG) (Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, 2015)
15	10/14	Policy gradients (III): Trust-region methods	Definitive Guide to Policy Gradients §3-4 (Matthias Lehmann, 2024) Trust Region Policy Optimization (TRPO) (John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz, 2015) Proximal Policy Optimization (PPO) (John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017)	Project ideas due (meeting required)
16	10/16	Presentations Applications of Advanced Policy Gradient methods (DeepSeek/GRPO)	DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Rulebook: bringing co-routines to reinforcement learning environments (Massimo Fioravanti, Samuele Pasini, Giovanni Agosta, 2025)
17	10/21	Reinforcement Learning from Human Feedback (RLHF)	Reinforcement Learning from Human Feedback: Progress and Challenges (John Schulman, 2023) A Survey of Reinforcement Learning from Human Feedback (Kauffmann, Weng, Bengs, Hullermeier, 2023)
18	10/23	Presentations RLHF	Training language models to follow instructions with human feedback (Long Ouyang et al., 2022) Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (Jan Brauner et al., 2023) Deep Reinforcement Learning from Human Preferences (Paul F. Christiano et al., 2017) RLTF: Reinforcement Learning from Unit Test Feedback (Yujia Qin et al., 2023)
19	10/28	Evolutionary Computation	Neuroevolution (Risto Miikkulainen, 2022) Evolving neural networks through augmenting topologies (Kenneth O. Stanley, Risto Miikkulainen, 2002)	Assignment 3 due Implement an actor-critic method of choice
20	10/30	Presentations Applications of Evolutionary Computation (AlphaEvolve, etc.)	Evolution Strategies as a Scalable Alternative to Reinforcement Learning (Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever, 2017) Accelerating Evolution Through Gene Masking and Distributed Search (Hormoz Shahrzad, Risto Miikkulainen, 2023) AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery (Lain Mustafaoglu, Risto Miikkulainen, 2024) Evolutionary Policy Optimization (Lain Mustafaoglu, Keshav Pingali, Risto Miikkulainen, 2025)
21	11/4	Imitation Learning	Tutorial on Imitation Learning	Project check-in #1 (meeting required)
22	11/6	Presentations Imitation Learning	Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (Rajeswaran, Kumar, Gupta, Vezzani, Schulman, Todorov, Levine, 2017) TBD
23	11/11	Parallel/Distributed RL	Tutorial	Assignment 4 due Evolve a neural network or implement a genetic algorithm
24	11/13	Presentations Large-scale distributed RL	SEED RL: Scalable & Efficient Deep-RL Recurrent Experience Replay in Distributed RL (R2D2)
25	11/18	Project check-ins
26	11/20	Project check-ins
THANKSGIVING BREAK
27	12/2	Project presentations
28	12/4	Project presentations		Final project paper due

Assignments

Programming Assignment 1 (release TBA)
Programming Assignment 2 (release TBA)
Programming Assignment 3 (release TBA)
Final Project (milestones TBA)

Resources

Canvas
Ed Discussion
Reading List
Reinforcement Learning: An Introduction (2nd edition) by Sutton & Barto
Neural Networks and Learning Machines, Third Edition by Simon Haykin