Sometimes words do speak louder than numbers it seems. Klea Grohmann sent me this link to a recent piece in Nature where the relationship between grant scoring (a number apparently) and grant getting were not that closely related to one another. The Medical Research Council (MRC), the funding agency, defends this by saying that the raw scores don't always line up with comments made and that the latter can be more revealing. This sounds plausible to me right now given that I am reading admissions files. At UMD, letters of recommendation forms ask for both a numeric rating of a candidate's attributes (how smart, original, motivated, mature etc) as well as a letter. To my surprise these often don't seem to match up that well. So, I can believe that this is also true at the MRC.
However, I suspect that what's closer to the truth is a comment made in the last paragraph by Ferric Fang. He notes that grant funding is something of a lottery at present and that with money becoming scarcer relative to number and quality of grants there will be more disappointed (and so disaffected) applicants. Moreover, the fact that there are so many good quality grants being submitted (and I believe that there are) only makes whatever distinctions utilized to make a decision seem arbitrary (largely because they are so). The hard decision arises between equally strong options. And hard decisions rely on more tendentious criteria precisely because the most obviously relevant ones don't decide. Isn't that what stairwells are for?
Last point: I believe that this is likely also true for linguistic grants. There is not enough funding for linguistic research, or more accurately, the funding has stayed about the same while the number of people shooting for it has greatly increased. I have the impression that landing a grant is much much harder now than it used to be, somewhere in the vicinity of 10% or less. I sincerely doubt that the criteria segmenting the top 10% from the next 10% are all that reliable, so that means that a lot of granting decisions are haphazard. Note that this does not mean that they are unfair, or dishonest, or done in some underhanded way, as Ian Eperon suggests in the link piece. Rather, it is that among roughly equal applications there is no reasonable way to make a decision, so the choices become noisy, influenced as they must be by rather minor differences among the applications. When all deserve funding but all cannot get it then even if there is no difference between winners and losers it does not mean that winners and losers are selected unfairly. What's unfair about the flip of a coin?
At any rate, someone with more knowledge about the ling funding scene or the psycho/neuro ling funding scene than I have please chime in. I would love to know how much our world resembles theirs. Thx Klea.