Friday, March 16, 2018

Lotsa luck, please

One of things that makes doing research (and by this I mean original research) hard is that there is no guarantee of success. There really are no good rules for doing it. More often than not, it’s unclear what the question being asked should be, let alone what it actually is. It’s similarly unclear what kinds of answers would serve to advance said partially formulated question, nor what kinds of data would offer the best kinds of evidence. The research game is played while the rules are being made, remade and remade again and even if done well there is no reason to think that the whole damn thing might not collapse or spin out of control. There’s no insurance one can buy to protect against this kind of failure. That’s just the way it is. Put more concisely, there is no scientific method wherein you press a research button, work hard, and reap the rewards.

But why not? Why can’t we automate the whole process? Why do we fail so often? Well, interestingly, people have been thinking about this question and a recent Sci Am blog post discusses some recent research on the topic (see here). It seems that one reason things are so hard is that a lot of the process is driven by luck, and this, again interestingly, has important implications.

First, the claims: science likes to think of itself as a meritocracy, perhaps the paradigmatic meritocracy.[1] Scientists are judged by their work, the successful ones being so largely because they are smarter, more disciplined, more careful, more creative than others. Research success leads to increased resources to fund further success and in the best of all worlds, resources flow to those with proven track records.

Rewards enhance this perception and promote a hero-conception of discovery in which a brave person battles against the forces of scientific darkness and by originality, grit and brains to burn triumphs over the forces of darkness and ignorance. Examples of such include the average Nobel prize winner. In this regard, such prizes are considered as significantly different from lotteries. Their rewards are deserved because earned, and not the result of dumb luck. The Sci Am piece asks whether Nobel prizes and lotteries are not more similar than generally believed. And if they are, what does this do to the conception of merit and rewards to the deserving?

There is one more point to make: this meritocracy is not only taken to be right and proper but is also understood to be the best way to advance the scientific search for truth. Merit gains rewards which advances the common project. But is this actually the case?

Here’s what I mean. What if it turned out that luck plays a VERY large role in success?  What if success in science (and in life) is not a matter of the best and brightest and hardest working gaining as a result of their smarts, creativity and perseverance, but but is more a product of being in the right place at the right time? What, in other words, should happen if we came to appreciate that luck plays an inordinately large role in success? Were this so, and the blog post linked to cites work arguing for this conclusion, then the idea that science should be run as a meritocracy would require some rethinking.

How good are the arguments proffered? Well, not bad, but not dispositive either. They consist of two kinds of work: some modeling (largely of the ‘toy’ variety) and some review of empirical work that argues the models are pointing in the right direction. I leave it to you to evaluate the results. FWIW, IMO, the argument is pretty good and, at the very least, goes some way towards noting something that I have long considered obvious: that lots of success, be it academic or otherwise, is due to luck. As my mother used to say: “it’s better to be lucky than smart.” Apparently, the models bear her out.

What follows from this if it is correct? Well, the biggest implications are for those activities where we reward people based on their track records (e.g. promotion, tenure and funding). In what follows I want to avoid discussing promotion and tenure and concentrate on funding for the papers note some interesting features of funding mechanisms that avoid the meritocratic route. In particular, it seems that the most efficient way to fund research, the one that gets most bang for the buck, targets “diversity” rather than “excellence” (the latter term is so abused nowadays that I assume that shortly it will be a synonym for BS).

For example, one study of over 1200 Quebec researchers over 15 years concludes: “both in terms of quantity of papers produced and of their scientific impact, the concentration of research funding in the hands of a so-called ‘elite’ of researchers produces diminishing marginal returns” (5). Indeed, the most efficient way to distribute funds according to the models (and the empirical studies back this up) is equally and consistently over a lifetime of research (6):

…the best funding strategy of them all was one where an equal number of funding was distributed to everyone. Distributing funds at a rate of 1 unit every five years resulted in 60% of the most talented individuals having a greater than average level of success, and distributing funds at a rate of 5 units every five years resulted in 100% of the most talented individuals having an impact! This suggests that if a funding agency or government has more money available to distribute, they'd be wise to use that extra money to distribute money to everyone, rather than to only a select few. As the researchers conclude,

"[I]f the goal is to reward the most talented person (thus increasing their final level of success), it is much more convenient to distribute periodically (even small) equal amounts of capital to all individuals rather than to give a greater capital only to a small percentage of them, selected through their level of success - already reached - at the moment of the distribution."

This is what one would expect if luck (“serendipity and chance” (5)) is the dominant factor in scientific breakthrough.

So, does this mean it is only luck? No, clearly, other things matter. But luck plays an outsized role. Where talent etc. comes in is readying one to exploit the luck that comes one’s way (a prepared mind is a lucky one?). The post ends with the following reasonable observation based on the studies it reviews:

The results of this elucidating simulation, which dovetail with a growing number of studies based on real-world data, strongly suggest that luck and opportunity play an underappreciated role in determining the final level of individual success. As the researchers point out, since rewards and resources are usually given to those who are already highly rewarded, this often causes a lack of opportunities for those who are most talented (i.e., have the greatest potential to actually benefit from the resources), and it doesn't take into account the important role of luck, which can emerge spontaneously throughout the creative process. The researchers argue that the following factors are all important in giving people more chances of success: a stimulating environment rich in opportunities, a good education, intensive training, and an efficient strategy for the distribution of funds and resources. They argue that at the macro-level of analysis, any policy that can influence these factors will result in greater collective progress and innovation for society (not to mention immense self-actualization of any particular individual).

So, there may be a good reason for why research feels so precarious. It is. It requires luck to succeed, lots of luck, lots of consistent luck. And if this is correct, it suggests that the winner take all strategies that funding agents tend to favor is likely quite counterproductive for it relies on picking winners which is very hard to do if the distribution of winners is largely a matter of luck.

That said, I doubt that things will change very soon. First, in an era of big science, big grants are needed and if there is a money shortage, then the only way to have big grants is to eliminate little ones. Second, there is an internal dynamic. Winners like to think that their success is due to their own efforts. That’s the charm of meritocracy. And as winners tend to make the rules don’t expect the promotion of “excellence” (and rewarding it) to end anytime soon, even if it would make the scientific life a whole lot better.

Last point: a while ago FoL discussed an interesting interview with the biologist Sydney Brenner (here). It generated a lively discussion in the comments that bears on the above. The point that Brenner made is that science as practiced today would have stifled the breakthrough research that was carried on in his youth. Some noted a certain kind of nostalgia for a bygone era in Brenner’s remarks, a period with its own substantial downsides.  This is likely correct. However, in light of the “luck is critical” thesis, Brenner’s point might have been based on the fact that in his day funding was more widely spread out among the relevant players and so it was possible for more people to “get” lucky. The problem then with the current state of play is not merely the insufficient level of funding, but the distribution of that funding across potential recipients.  In earlier days, the money flowed to the bulk of the research community. Nowadays it does not. And if luck matters, then the spread matters too. More pointedly, if luck matters, then rewarding the successful is a bad strategy.



[1] Rewards enhance this perception and promote a hero-conception of discovery in which a brave person battles against the forces of scientific darkness and by originality, grit and brains to burn triumphs over the forces of darkness and ignorance. Examples of such include the average Nobel prize winner. In this regard, such prizes are considered as significantly different from lotteries. Their rewards are deserved because earned, and not the result of dumb luck. The Sci Am piece asks whether Nobel prizes and lotteries are not more similar than generally believed. And if they are, what does this do to the conception of merit and rewards to the deserving?

No comments:

Post a Comment