I'm in the middle of chairing the Programme Committee (PC) of CP 2009, the major conference for Constraint Programming. In fact I've just sent out all acceptances and rejections.
One author, who queried something about their rejection, said after my reply the nicest thing I can imagine:
"In future I will endeavour to be rejected by conferences that you are chairing"
I really can't imagine higher praise.
Meanwhile, I have had some thoughts on the process, which I wanted to jot down while I remember.
Each paper was (by default) reviewed by three reviewers.
1) I am happy to have genuinely made every decision for every paper on a case by case basis. Also, genuinely nobody was automatically outvoted. A number of papers had one vote for and two against but were accepted and vice versa. I would strongly recommend that for the future, (i.e. that two reviewers don't get to outvote one just by weight of numbers).
2) It's true I messed up the applications track process, but I am reasonably happy with the papers we have. I messed up because a senior colleague repeatedly advised me to have a special applications track committee, but I refused. Then I think the apps papers received less focus than they should have done.
3) I was pleasantly surprised that only 2 reviews (out of about 450) came in too late to be sent to the authors on the deadline, and none before it was too late for authors to reply to them. I strongly recommend having an author rebuttal phase.
4) To help not get any complaints from PC it helps (I suspect) to previews how you are going ot run things and policies etc, so people can complain about the general policy rather than its application.
5) I think the not accepting shorter versions worked well this year. What I mean is, no papers were submitted as long papers and accepted as short papers only. It concentrates the mind of PC members and chair. Again I would recommend this for future years. There are long papers which I think most years would have been accepted as short papers but which I think are much better to be presented at the length submitted. Some papers were borderline from review scores so were obvious ones to ask to shorten. But they should really be in the conference in toto or not at all.
6) The acceptance rate is a smidge higher than I was aiming at, I expected just under 40%, not just over. But no complaints as of yet, but I was scared enough not to announce the rate until I had sent out decisions! However, papers were not accepted or rejected against a rate. And, I found that a lot of the papers that were really borderline will (I hope) punch above their weight in terms of value to the conference. Some of these papers were trying to do something completely different with or about constraints, which is great. Nobody can suggest they are just another CP paper.
7) I made no attempt to read every paper or even skim them. I did however try to read every review of about 400. I failed but I must have read easily 300+ and got a couple of people to read the others against any outrageous comments. I also tried to read all the discussions though not the technical stuff people might be arguing about. This was very helpful in having a feeling of knowing what was going on and catching big issues quickly. Again I would recommend doing this but all of this was on the edge of feasibility in the time available. If the number of submissions goes up significantly, which would be great, co-chairs or a small senior PC members might be a good idea.
8) I've found out my favourite kind of PC members. Those who either say in discussion: "ok accept this even though I said reject" or vice versa; or those who say "no this is crazy, do not accept it because ...." and then give solid reasons which we can think about.
9) Touch wood, I've been lucky with complaints from authors. Not counting random queries etc, I think I only had one author email me to say "these reviews are a disgrace" and another case of multiple submission where the paper got withdrawn. Of course something else might come up later.