Every year grad departments select an incoming class,
funding agencies decide on who is going to get how much and journals decide who
is going to get published. Each of these decision involves a scare resource
issue: a (usually small) finite number of positions, a (usually much too small)
finite number of dollars and a (usually small) finite number of pages.[1]
The decisions amount to how to allocate these scare resources. The reason the
resources are scarce is that there are often more applicants/submissions than
there are places to allocate. But the real reason resources are scarce is that
quality alone, at least evident quality, does not suffice to perfectly fit the
applicants/submissions to the allocations available. In other words, there are
many ways to allocate the money to high quality applicants/submissions, at
least prima facie. And that’s the problem.
How do academics solve this problem? In general we do this
by applying every more refined (i.e. recondite?) markers of quality to winnow
out the really truly deserving from the merely truly deserving from the merely
deserving. It is well known that even blunt methods suffice to lop off the
clearly undeserving from the rest. In other words, academics embrace the
conceit that if we try hard enough we can
find the very very very…very best,
and can thus optimize our choices;
the decision procedure being a simple easily defensible one: select the most
deserving and allocate to them the scarce resources. So, every year departments
(including mine, of course) select the very
very…very best of the applicant pool, funding agencies fund the very very very…very best proposals and
journals publish the very very very…very best
papers. Those readers that find that the last sentence rings true, please
follow me as I have bridge for sale for you to look at.
Ok, everyone knows that the above story is more than a tad
tendentious. We all recognize that our powers of discernment, even if applied
with the utmost seriousness (which is seldom the case), are loaded with piles
of false positives and negatives. We know this, but as an institution we also
believe that overall looking for the
best and rewarding it is the superior strategy. But is it? Here
is a recent piece that questions this in interesting ways and suggests an
alternative.
Before getting into the nub of the proposal, here are some
points that the piece makes that (largely, though not perfectly) fit with my
experience (see some comments below). The author Shahar Avin (SA), only
discusses research funding, but I would add that the same holds for journal
pages, and grad slots.
1. Currently,
“the monetary cut-off point still tends
to be way above the quality cut-off point.”
2. “…expert reviewers spend a lot of time
allocating grant money by trying to identify the best work. But the truth is
that they’re not very good at it, and that the process is a huge waste of time.”
3. Peer
review, rather than adding information, often adds “another layer of
irrationality” to the decision process.
4. The
application process asks that you to be more sure of where you are going and
how you will get there than it is rational to expect someone who is aiming at
original research to be.
5. Our
capacities to identify those novel ideas most likely to succeed is much more
limited than we believe.
6. The
belief that we can make this kind of
rational decision “demand[s] excessive
amounts of information from applicants, and waste a colossal amount of their
time.”
7. For some areas, we can even quantify how time
consuming this is. SA cites a study in Nature
which calculates that “[i]n Australia, during a recent annual funding round for
medical research, scientists spent the equivalent of 400 years writing applications that were eventually rejected.” That’s a lot of
effort.
8. “…‘expert reviewers’ are not fungible commodities. One reviewer is not the
same as another, and their judgements tend to be highly personal. Of the nearly
3,000 medical research proposals submitted for public funding in Australia in 2009,
nearly half would have received the opposite decision if the review panel had
been different…”
9. “…the process isn’t just ineffective – it’s
systematically biased. There’s evidence that women and minorities have lower chances of securing grants than people who are male or white,
respectively.”
10. Unorthodox proposals are at a disadvantage in
the current funding structures.
As I said, most of these observations seem right to me.
Points 1/2 perfectly fits my little world. In my experience the quality of the
typical applicant pool, after the first triage, leaves many excellent
candidates to fill too few spots. I would also add that choosing the best among the crop of very good is not
something that I have found academics are good at. Certainly much of what comes to be recognized
as the best fails to find much
support much of the time. And this is especially so if the work is original or
the applicant has an unusual pedigree.
Let me expand on this for a moment in the context of grad
admissions. In assessing applicants we are going, reasonably enough, on track
record (what else is there). But, I have found that what makes you a good
undergrad is not necessarily what makes you a good grad student, nor is what
makes you a good grad student necessarily what makes you a fecund researcher.
Undergrads are, rightly, much more coddled. They are assessed by how well they
solve problems that have answers in the back of the book (or, now, online).
Grad students are largely assessed on how well they solve problems without
available answers, as are researchers. But, in addition, the latter are judged
by how well they find new problems
and generate novel research. The best
are those whose work create enlightenment that generates puzzlement and so
underwrites many years of further work.
In my experience, the qualities that allow success in any one of these
endeavors does not reliably signal success in any of the others. So, even if we
are conscientious in reviewing the available material, it is not clear what we
are looking for. And, I should add, an hour or two interview, does not, in my
experience, add much useful information. Luckily, how well we decide does not
really matter as in a large number of cases any selection will be ok. We don’t need
to find the best of the bunch because even the “worst” of the “best” is very
good.
Another reaction: I was surprised at how wasteful the
process can be! See point 7 above. 400 years is a long time, even if there are
collateral benefits of writing a grant that does not get funded or applying to
schools that don’t accept you or writing papers that never see the light of
day. And I would question how large these collateral benefits actually are. The
claims that there are benefits of doing the work even if it is unrewarded often
strike me as self-serving (often touted by those at the top of the food chain).
I can see the benefits of thinking through what one is working on as a valuable
exercise. What I am less convinced of is doing this in a grant application
format or a grad student admissions form is a good way of thinking things
through. Maybe it is. But it is not obvious to me that it is clearly a superior
way of doing this useful activity. And if it is not very useful, then it is questionable given that it clearly is very
time consuming.
I have made noises echoing point 4 on FoL many times. I
think that this is especially true for “theoretical” work where laying out the
timeline of research is a silly exercise. If a problem is really good, then how
you plan to solve it is not terribly apparent. The best you can do is motivate
a conjecture, and funding agencies (or at least the NSF) finds this very
insufficient. I suspect that this is so beyond theory. In my experience, grants
seldom fund the research proposed. Rather a grant submission often presents
finished research which if funded is ready to go “public” and the generated new
grant is used to fund novel research that is as yet indeterminate (and only
lightly described in the proposal). Ok, maybe I am being too cynical here. But
I don’t think I am being way too
cynical. At any rate, SA seems to agree.
Last point wrt 8/9 and then the “solution.” I also agree
that the current process is unstable in that choose your reviewers and things
will change dramatically. This
suggests that if there really are objective bases of quality that could be used
to rank alternatives then this factor is epistemologically elusive, at least
over a large domain. Moreover, this elusiveness allows for systematic bias to
creep in part because the process is believed to squeeze the bias out by
focusing on “quality” alone. Sadly, we
all know how this works. The point AH makes is that it might be working more effectively in a context in which
we are aiming to choose the best.[2]
Ok, the solution:
“Fortunately, there’s a simple solution to many of these problems. We
should tell the experts to stop trying to pick the best research.
Instead, they should focus on filtering out the worst ideas, and
admit the rest to a lottery. That way, we can make do with shorter proposals,
because the decision to accept or reject a ticket to a random draw requires
less information – and highly specific proposals are unrealistic anyway. So
instead of asking reviewers to make unreasonable predictions, they can turn
their minds to weeding out cranks and frauds. Bias will still occur in the
filtering stage, of course, but many more proposals will make it through to a
lottery, which is inherently unbiased.”
I confess to finding
this idea appealing. I believe that weeding out the clearly undeserving is a
lot easier than identifying the best. So, I believe, a lottery among the
triaged could save lots of time and effort. It might also, as AH notes,
eliminate some bias, against unconventional ways of thinking, biases of various
sorts and allow some things outside the prevailing fashion (though not too far
out, I suspect).
But I think that there
is a possible second very salutary effect of adopting a lottery system. It
might force academics to acquire a bit more modesty. Academics are natural
meritocrats. We reward the rewarded and denigrate the failures. We tend to
downplay how much luck plays in the process of “success.” One nice feature of a
lottery system is that it will make weaken this fantasy by making crystal clear
that luck plays a non-negligible role. It does this by institutionalizing luck as an overt feature of the process. This
might also have the beneficial side benefit of forcing administrators to
consider more refined ways of measuring academic quality than just counting
entries on a CV (weighted by venue quality). No more just listing publications
in “major” journals or “large” grants funded. Maybe some thought will need to
go into the process. Ok, I admit that there are problems with this too. But right
now the mechanization of the whole acadmic evaluation process has, I believe,
gotten out of hand and some push back is required. This could help by, as AH
says, making it official, that the whole process is (in large part) a lottery
anyhow. So not only might recognizing this and institutionalizing it, make “the
whole process… cheaper, fairer and more efficient.” It might also help to make it
more honest, with the attendant benefits that honesty often promotes.
[1]
In the case of journal space, web publishing offers a plausible solution to the
“too few pages” problem. However, one benefit of the page limits is that it
helps manage information overload for the reader. If page numbers expand then
the selection of “quality”/”relevance” will be off loaded to the reader. Right
now many rely on editors and journals to cull the good stuff. It is unclear
that it works well, but managing the tidle wave of material out there is an
important problem.
[2]
I don’t know if the review process in linguistics leads to obvious bias, but I
would not be surprised if it did. I just don’t know. Anyone with info to share
is invited to do so.
No comments:
Post a Comment