Foundations of Machine Learning for Systems Researchers
High-performance Kernel Generation
Course Overview
Rapid advances in machine learning have enabled us to build
systems with capabilities that were unimaginable just a few
years ago, such as AlphaGo, DeepSeekCoder, and AlphaEvolve.
The goal of this course is to analyze the key breakthroughs
that underlie these kinds of systems, and to understand how
they can be used to build systems for solving other problems.
This semester, the course will focus on the generation of
high-performance kernels using AI/ML techniques. This is a
more challenging problem than automatically generating code
for websites, for example, since there is not much training
data available, particularly for the many new and innovative
processors that have appeared recently.
Lectures will cover three main machine learning
technologies: deep neural networks, reinforcement learning,
and evolutionary computing. Unlike standard machine learning
courses, this material is presented using PL/systems concepts
such as dataflow analysis. Lectures will also discuss how
high-performance kernels are implemented manually on CPUs and
GPUs, using important kernels like FlashAttention as examples.
Students will present recent papers in conferences like
NeurIPS, ICML, ICLR for automating kernel generation. Term
projects will focus on using these ideas to generate
high-performance kernels and evaluate their performance.
Prerequisites
- Proficiency in Python
- Familiarity with parallel programming at the level of CS
377P
- Familiarity with calculus, statistics, linear algebra, and strong mathematical skills
- Coursework or equivalent experience in AI/ML at the level of CS 342 is strongly recommended
Coursework
- Paper presentations — 30%
- Paper reviews — 10%
- A substantial term project — 50%
- Participation and reading quizzes — 10%
