One of things that makes doing research (and by this I mean original research) hard is that there is
no guarantee of success. There really are no good rules for doing it. More often
than not, it’s unclear what the question being asked should be, let alone what
it actually is. It’s similarly unclear what kinds of answers would serve to
advance said partially formulated question, nor what kinds of data would offer
the best kinds of evidence. The research game is played while the rules are being made, remade and remade again and even if
done well there is no reason to think that the whole damn thing might not
collapse or spin out of control. There’s no insurance one can buy to protect
against this kind of failure. That’s just the way it is. Put more concisely,
there is no scientific method wherein you press a research button, work hard,
and reap the rewards.
But why not? Why can’t we automate the whole process? Why do
we fail so often? Well, interestingly, people have been thinking about this
question and a recent Sci Am blog
post discusses some recent research on the topic (see here).
It seems that one reason things are so hard is that a lot of the process is
driven by luck, and this, again interestingly, has important implications.
First, the claims: science likes to think of itself as a
meritocracy, perhaps the paradigmatic meritocracy.[1]
Scientists are judged by their work, the successful ones being so largely
because they are smarter, more disciplined, more careful, more creative than
others. Research success leads to increased resources to fund further success
and in the best of all worlds, resources flow to those with proven track
records.
Rewards enhance this perception and promote a
hero-conception of discovery in which a brave person battles against the forces
of scientific darkness and by originality, grit and brains to burn triumphs
over the forces of darkness and ignorance. Examples of such include the average
Nobel prize winner. In this regard, such prizes are considered as significantly
different from lotteries. Their rewards are deserved because earned, and not
the result of dumb luck. The Sci Am
piece asks whether Nobel prizes and lotteries are not more similar than
generally believed. And if they are, what does this do to the conception of
merit and rewards to the deserving?
There is one more point to make: this meritocracy is not
only taken to be right and proper but is also understood to be the best way to
advance the scientific search for truth. Merit gains rewards which advances the
common project. But is this actually the case?
Here’s what I mean. What if it turned out that luck plays a
VERY large role in success? What if
success in science (and in life) is not a matter of the best and brightest and
hardest working gaining as a result of their smarts, creativity and
perseverance, but but is more a product of being in the right place at the
right time? What, in other words, should happen if we came to appreciate that
luck plays an inordinately large role in success? Were this so, and the blog
post linked to cites work arguing for this conclusion, then the idea that
science should be run as a meritocracy would require some rethinking.
How good are the arguments proffered? Well, not bad, but not
dispositive either. They consist of two kinds of work: some modeling (largely
of the ‘toy’ variety) and some review of empirical work that argues the models
are pointing in the right direction. I leave it to you to evaluate the results.
FWIW, IMO, the argument is pretty good and, at the very least, goes some way
towards noting something that I have long considered obvious: that lots of
success, be it academic or otherwise, is due to luck. As my mother used to say:
“it’s better to be lucky than smart.” Apparently, the models bear her out.
What follows from this if it is correct? Well, the biggest
implications are for those activities where we reward people based on their
track records (e.g. promotion, tenure and funding). In what follows I want to
avoid discussing promotion and tenure and concentrate on funding for the papers
note some interesting features of funding mechanisms that avoid the
meritocratic route. In particular, it seems that the most efficient way to fund
research, the one that gets most bang for the buck, targets “diversity” rather
than “excellence” (the latter term is so abused nowadays that I assume that
shortly it will be a synonym for BS).
For example, one study of over 1200 Quebec researchers over
15 years concludes: “both in terms of quantity of papers produced and of their
scientific impact, the concentration of research funding in the hands of a
so-called ‘elite’ of researchers produces diminishing marginal returns” (5).
Indeed, the most efficient way to distribute funds according to the models (and
the empirical studies back this up) is equally and consistently over a lifetime
of research (6):
…the best funding
strategy of them all was one where an equal number of funding
was distributed to everyone. Distributing funds at a rate of 1 unit every
five years resulted in 60% of the most talented individuals having a greater
than average level of success, and distributing funds at a rate of 5 units
every five years resulted in 100% of the most talented individuals having an
impact! This suggests that if a funding agency or government has more money
available to distribute, they'd be wise to use that extra money to distribute
money to everyone, rather than to only a select few. As the researchers
conclude,
"[I]f the goal is to reward
the most talented person (thus increasing their final level of success), it is
much more convenient to distribute periodically (even small) equal amounts of
capital to all individuals rather than to give a greater capital only to a
small percentage of them, selected through their level of success - already
reached - at the moment of the distribution."
This is what one would expect if luck (“serendipity and
chance” (5)) is the dominant factor in scientific breakthrough.
So, does this mean it is only
luck? No, clearly, other things matter. But luck plays an outsized role. Where talent
etc. comes in is readying one to exploit the luck that comes one’s way (a
prepared mind is a lucky one?). The post ends with the following reasonable
observation based on the studies it reviews:
The results of this
elucidating simulation, which dovetail with a growing number of studies based
on real-world data, strongly suggest that luck and opportunity play an
underappreciated role in determining the final level of individual success. As
the researchers point out, since rewards and resources are usually given to
those who are already highly rewarded, this often causes a lack of
opportunities for those who are most talented (i.e., have the greatest
potential to actually benefit from the resources), and it doesn't take into
account the important role of luck, which can emerge spontaneously throughout
the creative process. The researchers argue that the following factors are
all important in giving people more chances of success: a stimulating
environment rich in opportunities, a good education, intensive training, and an
efficient strategy for the distribution of funds and resources. They argue that
at the macro-level of analysis, any policy that can influence these factors
will result in greater collective progress and innovation for society (not to
mention immense self-actualization of any particular individual).
So, there may be a good reason for why research feels so
precarious. It is. It requires luck to succeed, lots of luck, lots of
consistent luck. And if this is correct, it suggests that the winner take all
strategies that funding agents tend to favor is likely quite counterproductive
for it relies on picking winners which is very hard to do if the distribution
of winners is largely a matter of luck.
That said, I doubt that things will change very soon. First,
in an era of big science, big grants are needed and if there is a money
shortage, then the only way to have big grants is to eliminate little ones.
Second, there is an internal dynamic. Winners like to think that their success
is due to their own efforts. That’s the charm of meritocracy. And as winners
tend to make the rules don’t expect the promotion of “excellence” (and rewarding
it) to end anytime soon, even if it would make the scientific life a whole lot
better.
Last point: a while ago FoL discussed an interesting
interview with the biologist Sydney Brenner (here).
It generated a lively discussion in the comments that bears on the above. The
point that Brenner made is that science as practiced today would have stifled
the breakthrough research that was carried on in his youth. Some noted a
certain kind of nostalgia for a bygone era in Brenner’s remarks, a period with
its own substantial downsides. This is
likely correct. However, in light of the “luck is critical” thesis, Brenner’s
point might have been based on the fact that in his day funding was more widely
spread out among the relevant players and so it was possible for more people to
“get” lucky. The problem then with the current state of play is not merely the
insufficient level of funding, but the distribution of that funding across
potential recipients. In earlier days,
the money flowed to the bulk of the research community. Nowadays it does not.
And if luck matters, then the spread matters too. More pointedly, if luck
matters, then rewarding the successful is a bad
strategy.
[1]
Rewards enhance this perception and promote a hero-conception of discovery in
which a brave person battles against the forces of scientific darkness and by
originality, grit and brains to burn triumphs over the forces of darkness and
ignorance. Examples of such include the average Nobel prize winner. In this
regard, such prizes are considered as significantly different from lotteries. Their
rewards are deserved because earned, and not the result of dumb luck. The Sci Am piece asks whether Nobel prizes
and lotteries are not more similar than generally believed. And if they are,
what does this do to the conception of merit and rewards to the deserving?
No comments:
Post a Comment