Programming Languages Lunch - Emery Berger, University of Massachusetts Amherst, "Stabilizer: Statistically Sound Performance Evaluation"

Contact Name: 
John Thywissen
GDC 6.302
Feb 18, 2014 12:00pm - 1:00pm

Talk Audience: UTCS Faculty, Grads, Undergrads, Other Interested Parties

Host:  Keshav K Pingali

Talk Abstract: Researchers and software developers require effective performance evaluation. Researchers must evaluate optimizations or measure overhead. Software developers use automatic performance regression tests to discover when changes improve or degrade performance. The standard methodology is to compare execution times before and after applying changes.

Unfortunately, modern architectural features make this approach unsound. Statistically sound evaluation requires multiple samples to test whether one can or cannot (with high confidence) reject the null hypothesis that results are the same before and after. However, caches and branch predictors make performance dependent on machine-specific parameters and the exact layout of code, stack frames, and heap objects. A single binary constitutes just one sample from the space of program layouts, regardless of the number of runs. Since compiler optimizations and code changes also alter layout, it is impossible to distinguish the impact of an optimization from that of its layout effects.

In this talk, I will present Stabilizer, a system that enables the use of the powerful statistical techniques required for sound performance evaluation on modern architectures. I will explain how Stabilizer works, and demonstrate Stabilizer’s efficiency (< 7% median overhead) and its effectiveness by evaluating the impact of LLVM's optimizations on the SPEC CPU2006 benchmark suite. While -O2 has a significant impact relative to -O1, the performance impact of -O3 over -O2 optimizations turns out to be indistinguishable from random noise.

Speaker Bio: Emery Berger is an Associate Professor in the Department of Computer Science at the University of Massachusetts Amherst, the flagship campus of the UMass system. He graduated with a Ph.D. in Computer Science from the University of Texas at Austin in 2002. Professor Berger has been a Visiting Scientist at Microsoft Research and at the Universitat Politècnica de Catalunya (UPC) / Barcelona Supercomputing Center (BSC).

Professor Berger's research spans programming languages, runtime systems, and operating systems, with a particular focus on systems that transparently improve reliability, security, and performance. He is the creator of Hoard, a widely-used scalable memory manager on which the Mac OS X memory manager is based, and invented DieHard/DieHarder, which influenced security and reliability features in Windows 7 and 8.

His honors include a Microsoft Research Fellowship, an NSF CAREER Award, a Lilly Teaching Fellowship, a Most Influential Paper Award at OOPSLA 2012, a Google Research Award, and a Microsoft SEIF Award he was named an ACM Senior Member in 2010.