We're making some changes to the typical review process for AISTATS. The main changes are: (1) a two-stage reviewing process with more time for the authors to revise their papers, which will hopefully improve the quality of the final versions; and (2) awards for an increased number of notable papers, along with a process for giving these papers increased visibility and discussion, hopefully motivating authors to submit their best work.
In addition, we will conduct reviews in two phases: an initial review by two reviewers, and a second phase with zero or more additional reviewers. This feature is motivated, first, to be compatible with the expanded author response phase described above, and second, to try to reduce the total effort spent by our volunteer reviewers without reducing the quality of the review process, by focusing reviewer effort on papers that are likely to be accepted. We expect that the quality of reviews will even be increased in some cases: first, by not overloading the reviewers, and second, by having the benefit of the initial reviews when selecting additional reviewers. (Note: some recent conferences have tried to reduce reviewer effort by having initial “quick reviews” to determine which papers get full reviews. Our impression is that such experiments have typically not worked as well as hoped: it requires some time to fully understand a paper and its strengths and weaknesses, so quick reviews are even noisier than regular ones. So, our thought is that the best way to shortcut the process is by having a smaller number of full reviews.)
Finally, for all accepted papers, we will make drafts available well ahead of the conference, and facilitate online discussion.
In addition, we will invite a “discussant” for each notable paper. The discussant will write a short summary discussion of the paper's most significant benefits, caveats, and relationships to the broader literature. We will publish these summaries as invited papers in the journal special issue, and make initial drafts of the summaries available online before the conference. Each discussant will have access to the reviewer discussion and to the later online discussion of the paper, and will be expected to work without breaking the anonymity of the original reviewers. The goal is to keep the community as a whole from losing the benefit of the effort that reviewers have put in to understand the paper and place it within the larger context of related work.