Kathryn S McKinley, The University of Texas at Austin
I think, and research has shown (see below in my letter to the SIGPLAN executive committee), that double-blind submission and reviewing reduces bias and improves the quality of conference and journal publications. Many ACM conferences, such as those sponsored by SIGARCH, SIGMETRICS, SIGMICRO, and SIGMOD, and by other scientific venues successfully use double-blind submission.
I recommend the following implementation of double-blind submission. The authors omit their names from the submission, replacing them with "Submitted for Blind Review." In the submission, the authors' should refer to their own prior and all prior work in the third person, and include all relevant citations. The authors should exclude citations to their own work which is not fundamental to understanding the paper, including any references to prior versions of the same paper, and include these ciations in a cover letter.
The submission software should provide a mechanism by which the authors can identify all PC members with whom they have a conflict (current or in the past 5 year coauthors, advisor/advisee relationship, family, close friends, and same institution). As Program Chairs do now, no PC or other reviewer should have a conflict with the authors.
Double-blind reviewing requires more work on the part of the program committee chair. The submission software and the chair must track conflicts. The chair may make a few mistakes, and when the PC members go out for additional reviews, they may make a few as well. Some mistakes will reveal themselves immediately and occasionally compromise double-blind reviewing. Other mistakes will reveal themselves when the authors are revealed which is why it should be in the PC meeting, before the final decisions.
The reviewers should be made aware that knowing the authors names and institutions before reading a submission can introduce positive and negative bias based on the reviewers current opinions and experiences (or lack thereof) with work from the same. The reviewers should not endeavor to discover the authors.
After the review is submitted, or after the rebuttal period or at the PC meeting, the author(s) and the reviewers should be revealed. No person with a conflict with the submission should see reviews (or reviewer names) or stay in the room with a conflict submission, including the PC member on their own submission obviously. This element is key to single-blind as well as double-blind reviewing and ensures the privacy of the reviewer to be frank without fear of reprisals.
The ranking of the paper is thus double-blind, but the community can use other factors to decide the final outcome. This structure adds the benefit of revealing any potentially undiscovered conflicts. In addition, the corporate decision (rather than an individual) can take into account building on an established research program, starting a new area, or taking a risk on a wacky idea from a top group, or whatever.
Here are some links and references to other thoughts and implementations of blind reviewing from the SIGMOD community.
SIGMOD Submission Guidelines, by Jennifer Widom.
Snodgrass, R. T., "Single- Versus Double-Blind Reviewing: An Analysis of the Literature," ACM SIGMOD Record, 35(3):8--21, September 2006.
Tung, A. K. H., Impact of double blind reviewing on SIGMOD publication: a more detailed analysis, ACM SIGMOD Record, v.35 n.3, p.6-7, September 2006.
I think SIGPLAN would greatly benefit from perceived and actual improved fairness by adopting this model. Would this system be perfect? No, but it would be better.
Kathryn S McKinley
Hi,
I am writing in your roles as SIGPLAN executive committee members,
SIGPLAN conference steering committee chairs, and TOPLAS Editor. In
this email, I advocate that your conference and all SIGPLAN
conferences and journals move to double-blind reviewing within the
next year to increase the appearance of fairness and to reduce actual
bias to which we, as human reviewers, are subject. I believe that we
are doing the SIGPLAN community a disservice by continuing in
non-blind reviewing, and in particular, women researchers suffer
disproportionately from the bias introduced by non-double-blind
reviewing. To back up my request, I include some of the research on
this subject that has shown without double-blind reviewing, reviewers
are biased.
A number of people object to double-blind reviewing because they believe that reviewers can identify many authors based on the submission itself. Research bears out that authors cite themselves more than other authors, and thus established, prolific researchers can often be identified through their citation list.
However, even though the reviewer can sometimes guess the authors, research on the bias introduced by double-blind reviewing shows that articles published in journals/conferences that use blind reviewing have higher impact as measured by subsequent citations than non-double-blind reviewing journals/conferences. Thus double-blind reviewing helps reviewers make more objective decisions, even though they may guess the author.
In addition, this bias seems to work more against women than men, as evidenced by the stricter evaluation criteria often applied to the work of women. A gender bias analysis from recent competition in Europe shows significant bias continues: "Mysterious Disappearance of Female Investigators" Nature 436, 174 (14 July 2005) .
"In a now-famous study of the Swedish Medical Research Council's procedures for hiring postdocs that was published in "Nepotism and sexism in peer-review", Nature 387, 341 - 343 (22 May 1997), Christine Wenneras and Agnes Wold discovered that women had to be two-and-a-half times as productive as men to obtain the same scores from evaluators. Only 8 percent of all women received positions, compared with 25 percent of men. Had the research council awarded postdocs to women equitably, the quality of the new hires could have increased, Alice M. Agogino pointed out."
I request you move as quickly as possible to address this problem in all the SIGPLAN conference and journal publications, and furthermore that you advocate to ACM to adapt this policy for all the SIGs.
best regards,
Kathryn S McKinley
Professor, Department of Computer Sciences
The University of Texas at Austin
512 232 7467
http://www.cs.utexas.edu/users/mckinley/
PS. For those of you who do not know me, I publish mostly in SIGPLAN and SIGARCH sponsored conferences, I have served on over 25 program committees in the past ten years, and have recently served as Program Chair for PACT (2005), ASPLOS (2004), and PLDI (2007), all of which used double-blind review. Many other SIGs already use double-blind reviewing, for example, SIGARCH, SIGMETRICS, SIGMICRO, SIGMOD, etc. My personal experience as a program chair and reviewer is that double-blind review is better than non-blind. With non-blind review, I find that my personal preferences for or against the work of some author or institution predisposes my evaluation. With double-blind review, I am often surprised to find I guess the authors wrong and I like work from authors that I have never heard of previously.