So, how's STOC going?

I suggested that I'd blog about my experiences as STOC PC chair, so I thought I'd give an update.

We're nearing the point where all the "first round" reviews are supposed to be in. All papers had 3 PC members assigned to them, and my goal was to have 3 reviews for each paper a few weeks before the PC meeting. So far, it seems to be going well; most PC members have been putting in reviews. A few who seem to be working with an offline scorecard haven't uploaded what they've completed -- which makes things a bit more difficult for me, naturally :( -- and I expect some papers won't have 3 reviews by the deadline for various reasons, the most likely being that a subreviewer hasn't finished. Subreviewers, please get things in! (And thank you.)

After that point, I hope to reject a slew of papers outright. Recall that I'm trying a new scoring scale. 1 = bottom 50%, 5 = top 10 %, other scores in between. Anything with 3 1's seems to be an easy reject. Arguably, if there's no score of 3 or above in the 3 reviews, a paper should be rejected. To be clear I won't make those decisions unilaterally, but will set up votes for the PC.

The other work at this point is finding more reviewers for papers with widely disparate scores (where the problem does not seem to be "this paper has a bug" that one reviewer noticed but others didn't, but rather a wide disagreement on the quality/importance of the result) or for papers that seem like they'll be on the borderline. Here I'm looking for people both on and off the PC -- so don't be too surprised if you get an e-mail asking for a quick review -- with the emphasis on trying to find relevant experts in the area of the paper.

How many papers will have both a 1 and a 5 score? (Or 2/5s, or 1/4s?) Seems like a statistic for the business meeting roundup. But so far, the answer seems to be more than you might think...