Via Daniel Lemire comes a link to a page by John Wu where he rakes a VLDB 2006 paper over the coals and doing so, mounts a serious criticism of the review process that led to the paper being accepted. I don't have a dog in this fight, and know next to nothing about the area, but the criticisms are prima facie plausible, even if mitigated by the fact that he's the "injured party" in this argument. One idea that he proposes, and that Daniel endorses, is this:
Our first recommendation is for the program committee to assign one member to be responsible for a paper. To encourage committee members to take this responsibility seriously, the accepted paper should be printed with the name of the responsible person. Many journal articles are printed with a footnote to the effect of “Accepted on the recommendation of So-and-So.” Similar footnote could be printed on the final copy of the conference proceedings.It's an interesting idea. I've heard this being proposed before, and apart from the fact that papers can often slip in without strong champions, I don't see too much that's wrong with it.
Quite a few papers are in a subarea where there might not be any "authoritative" program committee member (off-hand and unverified examples: certain topics in logic, distributed or parallel algorithms in STOC/FOCS, perhaps crypto at SODA etc.).
ReplyDeleteThis suggestion might lead to rejecting such papers much more easily, thereby narrowing the conference's scope (by people being selfish).
Perhaps at VLDB this is less of a problem because they have very large PCs. On the other hand, a typical PC member at VLDB may review only 10-15 papers (isn't it?) and typically do not use external reviewers, effectively disclosing more information about the actual reviewers of each submission.
I was a bit surprised at the tone of the initial article (sounds pretty irresponsible in itself... but I digress..)
ReplyDeleteI was surprised that there are fields where the PC only partially reads a paper. I certainly read each of my assigned papers completely and I can safely vouch for most of my fellow PC members (architecture conferences).
For papers that require better presentation, the PC typically assigns a shepherd (although, the shepherd is not officially listed on the paper). I see no problem in extending this concept to every paper and giving the shepherd credit. But it looks like this field has a much bigger problem: a conference where PC members only partially read a paper is NOT thoroughly peer-reviewed!! PC members are better off spending a few more hours reading each paper rather than spending many hours shepherding their assigned paper.
well I don't think there's an official mandate to read a paper partially. Reviewers are of course expected to read papers thoroughly: it's the practice that occasionally differs from the principle.
ReplyDeleteI also would hesitate to tar an entire conference with the (alleged) mistakes of one reviewer.
I am referring to the practice. I don't have much direct contact with the VLDB community, but if this claim of John Wu (possibly regarding some conferences) is true: "There is a common belief that the conference program committees never read more than a few pages of a conference paper when they make a decision on whether to accept or reject a paper", then such a conference should not be deemed "thoroughly peer-reviewed". A better path for the conference is to insist (and exert peer-pressure) that everyone read every assigned paper. Such a solution is more time-efficient than assigning a shepherd to every paper and less likely to allow a poor paper to slip through the cracks.
ReplyDeleteNote that the two statements do not contradict: ie.
ReplyDelete"Every reviewer must read every paper in its entirety" AND
"Reviewers tend to make judgements about papers after reading a few pages"
do not contradict each other. The first refers to the letter and spirit of the reviewing process, and the second references the underlying psychological dynamic by which people tend to process arguments.
Which is why the main structural features of a reviewing process must address and counteract people's inherent biases; multiple reviewers, a PC discussion, even public shaming if appropriate :)
Yes, we form an opinion after the initial pages, but I (and many others) don't pass Accept/Reject judgements until we read the entire paper. We especially don't make "Accept" judgements until the very end of the paper (fortunately or unfortunately, reviewers are combing the text for reasons to reject a paper).
ReplyDeleteBut we digress again :-). I don't believe the system is broken for conferences in my area. Each paper receives 5-6 thorough reviews (based on a reading of the entire paper) and it is extremely rare for a paper with obvious flaws to get accepted. If this is already being done for a conference, it is not clear that the filter is made much stronger by assigning a formal shepherd to every paper (at least not enough to warrant all the work).
By the way, I can't see a button to post a comment for your post a few down on quantum computing, but the answer is no. Measurements are truly random and can't be repeated. The ability to do so would likely have interesting consequences---much faster algorithms for a few problems.
ReplyDeleteWe should first differentiate perception from reality. Is VLDB fair or not? I do not know, and I do not care much for the matter. Can you improve the process? I think we can.
ReplyDeleteThe title of this blog entry is a little sensational. Guess that is how one gets readers.
ReplyDeleteSpeaking as someone who knows a little about VLDB, it is a pretty good conference in database field. Like Daniel said, the peer review process can be improved and it should. SIGMOD, another conference in database field, just instituted Experimental Repeatability Requirements. It won't have caught the particular problem, but at least it is doing something.
There is an updated version of the VLDB 2006 paper in question
ReplyDeletehttp://www.cse.ohio-state.edu/~hakan/publications/VLDB06.pdf
Here is another update with additional references to Bloom filters.
ReplyDelete