Before I get too far in, let me just say that overall, CHI has treated me very well. I love the projects that I see at the conferences, I think the people in it are bright and intelligent, and it has helped to give me a career in doing stuff that I really enjoy (research in HCI). I was also very lucky to attend the recent CHI program committee meeting in Seoul, and thoroughly enjoyed the experience (as I described to a friend, it was kind of like going to a really great reading group). I also personally fared reasonably well with several papers (conditionally) accepted. So, overall: good.
This “rant” is about the paper reviewing/selection process that CHI uses, and a structural flaw in its use of external vs. and secondary reviewers. I’ll begin by describing how the process works (right now). Then I’ll get into my rant. Unfortunately, I don’t have any great new solutions… I don’t think I have enough experience to figure out what great new ones would be. Instead, I propose only a few “well-known” solutions.
Once an author submits his/her paper to a review subcommittee, the reviewing machine begins. The CHI review process has five separate phases, and three different types of reviewers. The five phases are as follows: first, the subcommittee chairs assign the paper to a primary reviewer (a person that is on the program committee — usually, this is R4); second, the primary reviewer assigns three external reviewers (external to the program committee) to review the paper; third, the authors get to see the first four reviews (from R1, R2, R3 and R4) and write a rebuttal/response; fourth, the submission, if it is “good enough”, is assigned a secondary reviewer (another member of the program committee — usually, R5) to read in time for the program committee meeting; finally, at the program committee meeting, additional committee members (R6, and so on) are assigned to offer an opinion, where necessary.
There are three different kinds of reviewers here:
There is a rationale for the process right now. First, the rebuttal allows for the authors to clarify things that the original set of reviewers (R1-R4) “got wrong.” A secondary (R5) is assigned only to submissions that stand a chance of getting in — this saves resources on the program committee side of things, as submissions that don’t really stand a chance don’t need to get assigned a secondary. Finally, in the PC meeting itself, sometimes the primary and secondary disagree about the fate of a submission (or need additional expertise), and it is here that additional secondary reviewers are assigned (R6+). These additional secondary reviewers give the paper a quick read and offer an opinion.
The structural problem in the CHI reviewing process is that reviewers that come online later in the process have essentially veto power over any reviewers earlier (and usually, these reviewers have had even less time with the paper). This happens in at least two ways:
Thus, the structural problem results in two effects: first, subsequent reviewers essentially have veto power over anything that happened before, and second, the authors do not get a chance to rebut anything that happens after R4’s review — this is problematic, particularly if subsequent reviewers actually misread something (or interpreted something incorrectly).
I think fundamentally, this is crazy because the philosophically, the role of the three external reviewers is to offer an expert opinion. Cynically: if members of the PC (who, admittedly, have been entrusted with the direction of the research community) can overturn the opinion of the three experts, then why even bother the experts to begin with?
I also think it is crazy because the rebuttal is almost in the wrong place — if the purpose of the rebuttal is to clarify misinterpretations in the reviewers’ reading of the submission, then why is it that an author only gets to rebut a small portion of the reviewers? Just as an extreme variation on the rebuttal idea to make the point — imagine you wrote the rebuttal after seeing only the first two reviews that came in. Just as external reviewers can sometimes get it wrong (i.e. misread the paper), committee members can also misread something (and frankly, are more likely to, as they are responsible for reading and offering an opinion on submissions with a heavy time pressure).
I saw a few papers (and experienced one myself) where the reviews going into the PC meeting (i.e. from R1-R4) were on average, quite good (i.e. >3.5). This gives the initial impression that, “Things will be fine,” and thoughts of, “Hey! This is a slam dunk! Woohoo!” And then, we (and others) were probably quite surprised to find that the paper had been kaiboshed at the PC meeting. This feels profoundly unfair… (In our case, it feels extremely unfair because the secondaries actually misread something in a factually incorrect way — again, something a rebuttal could have cleared up.)
Tenure and promotion decisions are made on the basis of this stuff. We should try to get the process right, otherwise, we are unfairly messing with people’s careers due to a (slightly) random process. In the worst case, we’ve messed up someone’s career. In the second worst case, people stop going (and submitting) to CHI.
I’m a young guy — still wet behind the ears. I haven’t thought long about this, nor do I have the experience to have seen how else it could have been done. Nevertheless, here are some solutions (half-baked as they may be) to the structural problem I outlined earlier.
In spite of this, I have a lot of faith in the CHI community. When I see the kinds of things that Jofish is talking about on his tumblr blog, it makes me feel better — other people are seeing these problems too. It is a good thing smarter people are working on figuring out good solutions. Crossing fingers.
TLDR; We got screwed. I ranted. Smart people are on the case.