As the program chair, you want to create a committee that covers all the technical areas for the conference and that writes useful reviews in a timely fashion (timely is key to reducing your stress). Secondary, but still important should be diversity of institution, rank, and experience. Your committee should be large and diverse enough to provide expert reviewers on a paper with one conflict in the topic area. Large committees decrease the work load, but can increase the probability one or more members wont do their work. Make sure everyone agrees to attend the PC meeting. PC attendance is key to a good program.
Asking early or late? For ASPLOS, I asked very early (more than 1 year out from the work), but then several people had personal/time conflicts at the last minute. For PACT, I asked late (4 months out), which made it more difficult to obtain senior members but yielded a committee that with less turnover and personal/time conflicts.
I used the CRP software which I needed an assistant with database experience to use well, and still I encountered challenges. Having a new person run the software at every conference instantiation is probably a bad choice. It probably would be a good investment to use one of the conference submission services for SIGARCH and IEEE conferences. (SIGPLAN is already using these services successfully.) We borrowed or added several features to the software to make the following changes.
Conflicts. The authors specified their PC conflicts and institution conflicts with a check box, and others in a list which greatly eased paper assignation.
Matching. For ASPLOS, I made all the paper assignments by matching paper topic areas to PC member topic areas. I found that I did not do a great job (wrt PC interest and expertise) and for 169 papers and 18 PC members. Because of the unexpected quantity of submissions, I applied two rounds of reviewing. In the first round, I assigned each paper two committee member reviewers (about 20-25 hours of work for me), and each of them requested one or more reviews. For the 112 papers with a significant number of positive reviews or conflicting reviews, I then assigned an additional committee member reviewer (another 10 hours of work) and in a few cases, external reviews. Thus, each paper had at least 4 reviews and all papers discussed at the committee meeting had 5 reviews, with 3 from committee members. Most of these reviews came in before the rebuttal period. In addition, each PC member had a higher density of papers in the top group. I recommend this two tier strategy for better focus with a large number of submissions, but it is more work for the PC chair.
For PACT, I instead gave the PC members a list of all their non-conflict papers, and let them indicate their preferences. With this information, I achieved a 100% match of submissions to one or more PC members that wanted to review the submission, but it only took me about 10 hours. I assigned each submission 3 PC members and they choose an additional 3 outside reviewers. Since the ratio between the PC members and number of submissions was low, I did not do two tiers of reviewing. On the PC side, I assigned PC members from 70% to 100% of the submissions in which they had high interest. As a result, I believe we had better and more qualified reviewing than we would have otherwise.
I highly recommend blind submission to reduce bias, and the rebuttal process to encourage constructive and accurate reviewing.
I have seen the rebuttal process change only a few reject and accept decisions, but these decisions are important and it has the following additional benefits. Reviewers submit their reviews before the rebuttal period, instead of doing them on the plane on the way to the PC meeting. They are more likely to do a careful job for that reason, and because the authors can correct them.
I have a few points of possible interest about how to run the meeting itself smoothly.