Notes on Chairing Program Committees:

Notes on Chairing Program Committees
April 2013

Kathryn S McKinley, Microsoft Research and The University of Texas at` Austin


I have served as program committee chair for CGO 2013, ISMM 2012, PLDI 2007, PACT 2005, and ASPLOS 2004, and I on over 50 program committees of these and other conferences. This document contains reflections on my experiences and advice for program chairs.

Conferences and program committee (PC) meetings serve critical social and research functions in the computer science community. Ensuring a fair process that encourages, guides, and rewards high quality and forward looking research is the responsibility of the reviewing process.

Poor review quality and poor process is very damaging to our community. It alienates all community members and is especially damaging because it disproportionately discourages new community members (i.e., graduate students and young researchers), since they have no or many fewer good previous experiences to counter a bad experience. PC members and chairs should take their job very seriously.

PC organization

To improve review expertise, reviewer accountability, handle PC submissions, and reduce PC member workload, I recommend a moderately sized Program Committee (PC) with an in person PC meeting and a named External Review Committee (XRC), as pioneered in the SIGPLAN and SIGARCH communities by Steve Blackburn at ISMM 2008. ASPLOS, ISCA, ISMM, PLDI, POPL, and other conferences now regularly use this structure.

Another key to high quality reviews and moderating the workload (which are related!), is two round reviewing, in which each paper gets two PC reviews and one XRC review in a first round. With this information in hand, the committee rejects at least half the submissions. The second round adds one or two reviews, such that every paper that will be considered further will have at least 3 PC reviews.

Your committee should be large and diverse enough to provide expert reviewers on a paper with one conflict in the topic area. Large committees decrease the work load, but sometimes increase the probability one or more members wont do their work or do it on time. Make sure everyone agrees to attend the PC meeting. PC attendance and applying no limit on paper acceptance are key to a diverse and forward looking program.

Selecting PC members

As the program chair, you want to create a committee that covers all the technical areas for the conference and that write useful reviews in a timely fashion (timely is key to reducing your stress). To get high quality reviews, you need responsible and knowledgeable reviewers. Your personal experience and those of prior PC chairs are a great source of this information. Ask them who does their work? Who wrote the best reviews? Who was productive in the PC meeting, e.g., made thoughtful comments and moved papers to their conclusion in a principled fashion?

Secondary, but very important should be diversity of institution, rank, and experience. I always use a spreadsheet that includes any previous service on this conference's PC, gender, institution, career stage (early, middle, late), and topic coverage. The conference should be tracking all PC members in the past in a spreadsheet. To create the topic list and a source of program committee members beyond my personal network, I use and recommend extracting topics and researchers from the past 5 years of the conference. I also look at related conferences for emerging topics and researchers.

Asking early or late? For ASPLOS, I asked very early (more than 1 year out from the work), but then several people had personal/time conflicts at the last minute. For PACT, ISMM and CGO, we asked late (4 months out), which made it more difficult to obtain senior members but yielded a committee that with less turnover and personal/time conflicts.

Software

I recommend a change to the current system, where everyone turns over every year and thus the PC chair is left to choose the software submission and reviewing system, and then populate the entire process. I recommend a three year term for a web submission czar for each conference to configure and manage the submission software: one year in training, one year running it by herself or himself, and one year training the next person. Besides the obvious benefit that the PC chair has someone to help them who has a lot of experience with the software, the key benefit is that if the PC wants to change process, topics, conflicts, review form, etc., they still can, but at least the change is intentional instead of random.

Paper assignation

Conflicts. I prefer that the authors specify their PC conflicts and institution conflicts with a check box, and others in a list which greatly eases paper assignation. However, this mechanism gives authors the ability to veto reviewers, without conflicts, and several PC chairs have discovered this problem. Reversing the process to avoid this problem reveals authors to PC members, slightly unblinding the process. Reminding authors to only submit conflicts based on your policy should be sufficient.

Matching. I recommend bidding for the papers by the PC and XRC, and then doing paper assignments with this information and topic expertise in hand. I used this system for all the PCs I chaired except ASPLOS. For CGO, Lieven Eeckhout and I together spent about 10 to 15 hours refining the automated system to make sure each paper had good expertise. For ASPLOS, I made all the paper assignments by matching paper topic areas to PC member topic areas. I found that I did not do a great job (with respect to PC interest and expertise without their input) and for 169 papers and 18 PC members, and highly recommend bidding, which I used for all subsequent PCs I ran.

I used two round reviewing at PLDI, ASPLOS, and CGO. I first used two round reviewing at ASPLOS by necessity rather than design due to an unexpected quantity of submissions. In the first round, I assigned each paper two committee member reviewers (about 20-25 hours of work for me), and each of them specified one or more reviewers from the PC. For the 112 papers with a significant number of positive reviews or conflicting reviews, I then assigned an additional committee member reviewer (another 10 hours of work) and in a few cases, external reviews. Thus, each paper had at least 4 reviews and all papers discussed at the committee meeting had 5 reviews, with 3 from committee members. Most of these reviews came in before the rebuttal period. In addition, each PC member had a higher density of papers in the top group. I recommend this two tier strategy for better focus with a large number of submissions, but it is more work for the PC chair.

For all the other PCs, I instead gave the PC members a list of all their non-conflict papers, and let them indicate their preferences. With this information, I achieved a 100% match of submissions to one or more PC and XRC members that wanted to review the submission, and it took less time. As a result, I believe we had better and more qualified reviewing than we would have otherwise.

Blind Submission, Rebuttal, & Timely Reviewing

I highly recommend blind submission to reduce bias, and the rebuttal process to encourage constructive and accurate reviewing.

I have seen the rebuttal process change several, but not numerous numbers of reject and accept decisions. These decisions are however important and it has the following additional benefits. Reviewers submit their reviews before the rebuttal period, instead of doing them on the plane on the way to the PC meeting. Reviewers are more likely to do a careful job for that reason, and because the authors can correct them and their peer reviewers are watching. Furthermore, rebuttal helps build community, when reviewers and authors communicate through rebuttal, especially when the reviewers add a final statement about the accept/reject result in light of the committee discussions and rebuttal comments.

Running the Meeting

I have a few points of possible interest about the purpose of a PC meeting and how to run the meeting itself smoothly.

More Advice

I also recommend the following. In particular, I followed all the guidelines in Patterson's document on ethics, PC selection, PC submission, double-blind submission, conflicts in reviewing and in the PC meeting, etc.


I hope you found my experience and advice useful.