Wednesday, June 29, 2016

The end of science yet again

While incommunicado, I read another article (here) declaring the end of science as an institution, or at least declaring that it is very corrupt, maybe irreparably so and that it is chock full of perverse incentives that it would be amazing if anything at all got done right. I confess that I am getting tired of these doom declarations and I suspect that they are largely BS, though reflecting an importantly wrong view of inquiry (see the end). Or, more accurately, I have no doubt that bad work gets done and that scientists share the same character traits as other mortals and so are influenced by the possibility of fame and fortune and that this influences how they carry on their work, but, what I don’t see is any indication that we now live in a fallen state and that once there was a golden age during which truth and beauty alone motivated scientific inquiry. In other words, I have no reason to think that the quality of work done today is any different than that done before. There is just a lot more of it. Oddly, I think that the above could be considered an idiosyncratic perspective.

The author of the linked to piece above, Jerome Ravetz, thinks otherwise. He sees corruption and malpractice everywhere. He identifies the problem as the rise of industrial science where “Gesellschaft” has replaced “Gemeischaft” (love those German words; just what you need to add some suggestion of depth) and quality control has disappeared. Couple this with a very real job squeeze for scientists and “pathologies inevitably ensue,” most particularly chasing “impact” displaces the “self-sacrificing quest for scientific rigour.” This leads to “shoddy” and “sleazy” science where standards have so “slipped” and basic skills so “atrohp[ied]” that most practitioners don't know that their work is “sub-standard.” Wow! The end of the scientific days! Sciencemageddon! What crap!!

Let me say why I can’t take this junk seriously anymore. Three reasons:

First, the empirical basis of these analyses is that lots of work in many domains don’t replicate. This is what John Ionnidis (see here) showed a while back. Much work in biomedicine, social psych, and neuroscience fails to replicate. Often the numbers cited are over 50% of this work. Ok, say that this is so. What I have never heard is whether this is a big failure rate, a low one or just par for the course. In other words, what should I expect the non-replication rate to be a priori? Should we expect science in general to run at an efficiency of better than 50%? Maybe a success rate of 10% is unbelievably good (especially in domains where we really don’t know much). I have heard that Ray Bradbury is reputed to have said that 90% of everything is junk. If so, a 50% replication rate would be amazing. So, absent specifying an expected base rate, these numbers really don’t mean much. They should serve to warn the uninitiated from taking every reported experiment at face value, but for any working scientist, this should not be news.

Second, I see no evidence that things have gotten worse in this regard. When I was a tot, I hazarded to read some of the earliest proceedings of the Royal Academy of Science. This was fun stuff. Many reports of odd creatures and other funny findings. In fact, aside from its entertainment value, most of this is, from our point of view, junk. Entertaining junk I grant you, but really not of current scientific value. My suspicion is that the ration of important science and, ahem, questionable stuff is always roughly as it was then, at least in many domains. The stuff that survives and we covet is the smallest good bit sitting on a huge pile of detritus. Maybe it takes lots of junk to make the good stuff (who knows). But, I have seen nothing credibly arguing that today we produce more garbage than yesteryear. But if the problem with science today is that it is science done today viz. big Gesselchaft science where impact rather than truth is the lure then we should expect that all that ails us once did not. Color me very very skeptical.

Third, I am pretty sure that lots of the shoddiness is localized in particular areas. The logic of the modern decline should extend to the successful sciences and not be limited to the aspirational domains. So, do we have evidence that current practice in physics is shoddier? Do physicists rush to publish shoddy work more now than they did? Does the regular experimental submission to the Physical Review only replicate at a rate of 50%? How about cell biology or molecular genetics or chemistry? These are surely subject to the very same pressures as any other domain of scientific inquiry and, moreover, we have ways of comparing what is done now with what they did before because these fields have a before that we can compare our decline with. I, at least, have not heard of these domains suffering a replicability crisis. But if they are not, then it is very unlikely that the generic incentives that the doomsayers like Ravetz like to point to are really behind the problems in domains like social psych, neuro-science or bio-medicine. Not that there isn’t a clear difference between the “real” sciences and the ones most often criticized. There is, and here it is: we know something in physics and chemistry and genetics and cell biology and we know much less in the “crises” riven areas. We have real theories in physics and genetics and cell biology, theories that tell us something about the basic underlying mechanisms. This is not true in much of what we call “science” today. The problem with the poorer performing areas is not that the practitioners are shoddy or corrupt or venal or less skilled than their predecessors. The problem is that we are still largely ignorant of the basic lay of the causal land in these domains, and this is because we lack ideas not methods.[1] All of this brings me to my familiar refrain.

What really bugs me about this “end of science” doomology is the presupposition that failure to gain insight must be due to failure to apply the methods of inquiry correctly. The assumption is that there is such a method and that the only reason that we are failing (if we are) is that we are not applying it correctly. Given that the method is presupposed to be clear and simple (as well as domain general) then the only reason that it is not being applied right must be due to the personal failings of the investigators. They whore after the wrong gods: fame, fortune, impact. But this presupposition is poddle poop. There is no method and no mechanical recipe for inquiry that guarantees success if only conscientiously applied. The idea that there could be one is one of the most baleful legacies of Empiricism.

Empiricism is dedicated to the idea that what there is largely reflects what you can see. It’s all there in front of view if only you look carefully. There are no hidden forces or mechanisms. Empiricism is built on a faith in shallow explanation reflecting a surfacy metaphysics.[2] In such a world, the idea that there exists a Scientific Method makes sense, as does the idea that failure to apply it must reflect some kind of pathology. If you think that science is hard because it involves trying to locate hidden structure that is only imperfectly reflected in what you can see (experimentally or otherwise) then a high failure rate is to be expected until you discover ways of thinking (theory) that tracks this underlying reality. Only then might you be able to avoid being misled by what you see.

Let me finish with one more observation. Part of the doomsaying has been prompted by scientific over reach (aka scientism). Scientists love to preen that they are just motivated by the facts and not swayed by vulgar conceptions the way regular folk are. Scientists are hard headed and deserve respect, kudos, reward and deference because science has a method to check itself and so when science speaks it’s not just opinion. In other words, when it suits us, we scientists often trumpet the Scientific Method to claim deference. It’s what makes us “experts” and expertise trumps mere opinion. But if there is no Scientific Method then there is no expertise in virtue of having been produced by such a method. There is expertise, but it must be harder won and it is very limited. If this is right, then there is a lot less science out there than is advertised, and a lot less expertise than generally claimed. And that suits me just fine. That’s a doomsday I can both live with and rejoice in. That, however, has nothing to do with the status of actual scientific inquiry. It’s doing no worse than before, muddling along and saddled with all the problems it has always had.



[1] Though, this said, does anyone think that bio-medicine has made no progress? Would you rather give up modern methods for those from 1990, or 1950, or 1920? I am pretty sure that I wouldn’t. This would suggest that despite all the problems, at least in bio-medicine, we have learned something useful, even if many basic mechanisms are opaque.
[2] I stole “shallow explanation” from Chomsky who used this in another (related) context.

3 comments:

  1. “Gesellschaft” has replaced “Gemeischaft” (love those German words; just what you need to add some suggestion of depth)

    They translate mostly straightforwardly as "society" and "community"... the latter should have been Gemeinschaft, though. :-)

    Should we expect science in general to run at an efficiency of better than 50%?

    Well, we should expect people to run their own experiments often enough before they publish them that other people won't need to do it. One of the first things you learn as a chemistry undergrad (or indeed in highschool) is to do everything in triplicate just as the default baseline before you even think about it.

    I have heard that Ray Bradbury is reputed to have said that 90% of everything is junk.

    Sturgeon's Law: "90 % of everything is crud."

    does anyone think that bio-medicine has made no progress?

    Obligatory video with transcript under "show more". :-)

    ==========

    I agree that science is nowhere near doomed, and it's also evident to me that of course there's a single scientific method that can be applied to physics and sociology alike – it's just very simple: falsification and parsimony. However, various problems are real.

    1) Publish or perish. Most scientists in most countries and most disciplines* are forced to publish in journals with an impact factor, specifically the highest impact factor they can get; otherwise they can forget about jobs and grants. This has a bunch of consequences. One is the rise of the Least Publishable Unit: publish more frequently by dividing your coherent project into a series of minimalist papers, each of which says almost nothing and was made with the minimum of effort the reviewers required. Of course, most journals won't even let you publish a monograph, because they have space restrictions, and most of those that publish monographs lack an impact factor. (Fortunately, there are now a few exceptions among online-only journals.) Another consequence is that authors have to make their manuscripts look as mind-blowingly groundbreaking as possible, or the high-IF journals will reject them without even sending them out for review. – On the darker end, this is where peer-review rings and similar methods of fraud come from.

    2) Money. Disciplines with money in them, like everything vaguely related to medicine, offer an incentive for fraud.

    3) Of course there have always been and probably always will be a few personalities who want to be famous or similar things at all costs. One colleague has, I'm told, flat-out said he wants to have more publications than that famous 19th-century colleague who basically cranked out Least Publishable Units daily (most of them almost useless descriptions of new species, commonly the same species several times under different names); he has even resorted to plagiarism. However, this is not some kind of general crisis of science.

    4) "The closer you get to humans, the worse the science gets". That's a proverb among biologists and applies within biology as well as in a larger context. The closer you get to humans, the harder it is to isolate one factor and test its effects under laboratory conditions; the closer you get to humans, the more unexamined ideological baggage you probably carry around, and so do the reviewers and editors of your manuscript. To be blunt, my impression is that psychology can be done as a science, but usually isn't.

    * Most linguists seem to begin their career by publishing their PhD thesis as a book. In biology that would be suicidal. My thesis consisted of a series of papers, most of which were published before I defended.

    ReplyDelete
    Replies
    1. Thx for these very thoughtful comments. Let me add a word or two as comment to your comments.

      Of course you should do your experiments carefully. But what carefully means actually changes over time. We do get better and standards should always be vetted and improved. However, I am not sure that this would have the effect many think it would. Like I noted, in some domains of the "embarrassed" sciences (e.g. psych) there are domains where there is no crisis of replication (e.g. visual perception, and, IMO, linguistic judgments). Here the experiments seem to work fine. I would still like to know what the expected baseline is before I get all hot and bothered about the scary numbers being produced. A good baseline is how this compares to days of yore or in domains where there are the same social pressures but more mature sciences. This said, yes, of course, do things well.

      Scientific Method: falsification and parsimony. Well, yes. There is also the one that I like from Bridgeman: "use your noodle and no holds barred." So find data, use it and think. If that be method, we are all in favor. But when not much of a method is it? Sort of like "buy low sell high." Good advice, operationalizing it is tougher.

      1: publish or perish: Yes this is unfortunate and there are pressures. I have mentioned these in the past but I am not sure what to do about these. Are they worse now? Probably as there were more jobs in the "golden age" (i.e. during the cold war). So long as their are fewer prestige jobs (or jobs at all) than there are people who want them we will have to judge. The best way will be on the basis of the work. This means publications (why? nobody wants a return to the old boy network, right?). We would all love it were careful judgement of the work as a whole were the operating standard. But this was never so, and won't be now. So, until we can all do what we want without fighting for it, this problem, I believe, will remain in one way or another. I also lament its existence.

      Yes: money matters. Actually, I suspect that it is not money per se but not enough of it so that fraud (academic rather than just stealing the stuff) pays. If there was enough to go around then the incentive would drop.

      I agree completely with 3.

      And 4 is perfect. I will steal this adage in the future.

      Delete
  2. I wrote:

    commonly the same species several times under different names

    Not fraudulently, mind you; just carelessly, based on fragmentary material that he could have recognized as the same thing if he had taken more time.

    ReplyDelete