More on reviewing

Dominique Sportiche sent me this interesting break down on the NIPS experiment mentioned by Alex C in the comments section (here). It also provides a possible model for the results that presents in a more formal idiom one hypothesis that I floated for the results of the MRC results, namely that once one takes account of the clear winners and the clear losers the messy middle gets in in a tossup. At any rate, the details are interesting for the results as interpreted in the link come very close to assuming that acceptance is indeed a crapshoot. Let me repeat, again, that this does not make the decision unfair.  Nobody has  right to have their paper presented or their work funded. Arbitrary processes are fair if everyone is subject to the same capricious decision procedures.

That raises an interesting question, IMO. What makes for a clear winner or looser?  Here I believe that taste plays a very big role, and though I am a big fan of disputing taste, in fact I think that it is about the only thing worth real discussion (de gustibus dispudandum est), there is no doubt that it can move around quite a bit and not be easy to defend.

Still, what else to do? Not review? Randomly accept? This would likely delegitimize the reported work. So, I have no great ideas here. It seems that our judgments are not all that reliable when it comes to judging current work. Are you surprised? I'm not.

Addendum:

Let me add one more point. How do we decide what's good and what's not? Well on influence I suspect is what gets published/accepted/funded and what doesn't. If correct, then we can get a reinforcing process where the top and bottom ends of the acceptance/rejection process are themselves influenced by what was and wasn't done before. This implies that even where there is consensus, it might itself be based on some random earlier decisions. This is what can make it hard for novelty to break through, as we know that it is (see here for a famous modern case).

One of the most counterintuitive probability facts I know of is the following: If one takes two fair coins and starts flipping them and one coin gets, let's say, 10 heads "ahead" of the other how long will it take the second coin to "overtake" the first wrt heads? Well, a hell of a long time (is "never" long enough for you?). This indicates that random effects can be very long lasting.

One last comment: fortunately, I suspect that most science advances based on (at most) a dozen papers per field per year (and maybe fewer). Of course, it is hard to know ex ante which dozen these will be. But that suggests that over time, the noisy methods of selection, though they are of great personal moment, may make little difference to the advancement of knowledge. Humbling to consider, isn't it?