While incommunicado, I read another article (here)
declaring the end of science as an institution, or at least declaring that it is
very corrupt, maybe irreparably so and that it is chock full of perverse
incentives that it would be amazing if anything at all got done right. I
confess that I am getting tired of these doom declarations and I suspect that
they are largely BS, though reflecting an importantly wrong view of inquiry
(see the end). Or, more accurately, I have no doubt that bad work gets done and
that scientists share the same character traits as other mortals and so are
influenced by the possibility of fame and fortune and that this influences how
they carry on their work, but, what I don’t see is any indication that we now
live in a fallen state and that once there was a golden age during which truth
and beauty alone motivated scientific inquiry. In other words, I have no reason
to think that the quality of work done today is any different than that done
before. There is just a lot more of it. Oddly, I think that the above could be
considered an idiosyncratic perspective.
The author of the linked to piece above, Jerome Ravetz,
thinks otherwise. He sees corruption and malpractice everywhere. He identifies
the problem as the rise of industrial science where “Gesellschaft” has replaced
“Gemeischaft” (love those German words; just what you need to add some
suggestion of depth) and quality control has disappeared. Couple this with a
very real job squeeze for scientists and “pathologies inevitably ensue,” most
particularly chasing “impact” displaces the “self-sacrificing quest for
scientific rigour.” This leads to “shoddy” and “sleazy” science where standards
have so “slipped” and basic skills so “atrohp[ied]” that most practitioners
don't know that their work is “sub-standard.” Wow! The end of the scientific
days! Sciencemageddon! What crap!!
Let me say why I can’t take this junk seriously anymore. Three
reasons:
First, the empirical basis of these analyses is that lots of
work in many domains don’t replicate. This is what John Ionnidis (see here)
showed a while back. Much work in biomedicine, social psych, and neuroscience
fails to replicate. Often the numbers cited are over 50% of this work. Ok, say
that this is so. What I have never heard is whether this is a big failure rate,
a low one or just par for the course. In other words, what should I expect the
non-replication rate to be a priori?
Should we expect science in general to run at an efficiency of better than 50%?
Maybe a success rate of 10% is unbelievably good (especially in domains where
we really don’t know much). I have heard that Ray Bradbury is reputed to have
said that 90% of everything is junk. If so, a 50% replication rate would be
amazing. So, absent specifying an expected base rate, these numbers really
don’t mean much. They should serve to
warn the uninitiated from taking every reported experiment at face value, but
for any working scientist, this should not be news.
Second, I see no evidence that things have gotten worse in
this regard. When I was a tot, I hazarded to read some of the earliest
proceedings of the Royal Academy of Science. This was fun stuff. Many reports
of odd creatures and other funny findings. In fact, aside from its
entertainment value, most of this is, from our point of view, junk.
Entertaining junk I grant you, but really not of current scientific value. My
suspicion is that the ration of important science and, ahem, questionable stuff
is always roughly as it was then, at least in many domains. The stuff that
survives and we covet is the smallest good bit sitting on a huge pile of
detritus. Maybe it takes lots of junk to make the good stuff (who knows). But,
I have seen nothing credibly arguing that today we produce more garbage than
yesteryear. But if the problem with science today is that it is science done today viz. big Gesselchaft science where
impact rather than truth is the lure then we should expect that all that ails
us once did not. Color me very very skeptical.
Third, I am pretty sure that lots of the shoddiness is
localized in particular areas. The logic of the modern decline should extend to
the successful sciences and not be limited to the aspirational domains. So, do
we have evidence that current practice in physics is shoddier? Do physicists
rush to publish shoddy work more now than they did? Does the regular experimental
submission to the Physical Review only
replicate at a rate of 50%? How about cell biology or molecular genetics or
chemistry? These are surely subject to the very same pressures as any other
domain of scientific inquiry and, moreover, we have ways of comparing what is
done now with what they did before because these fields have a before that we
can compare our decline with. I, at least, have not heard of these domains
suffering a replicability crisis. But if they are not, then it is very unlikely
that the generic incentives that the doomsayers like Ravetz like to point to
are really behind the problems in domains like social psych, neuro-science or
bio-medicine. Not that there isn’t a clear difference between the “real”
sciences and the ones most often criticized. There is, and here it is: we know
something in physics and chemistry and genetics and cell biology and we know
much less in the “crises” riven areas. We have real theories in physics and
genetics and cell biology, theories that tell us something about the basic
underlying mechanisms. This is not true in much of what we call “science”
today. The problem with the poorer performing areas is not that the
practitioners are shoddy or corrupt or venal or less skilled than their
predecessors. The problem is that we are still largely ignorant of the basic
lay of the causal land in these domains, and this is because we lack ideas not
methods.[1]
All of this brings me to my familiar refrain.
What really bugs me about this “end of science” doomology is
the presupposition that failure to gain insight must be due to failure to apply
the methods of inquiry correctly. The assumption is that there is such a method
and that the only reason that we are failing (if we are) is that we are not
applying it correctly. Given that the method is presupposed to be clear and
simple (as well as domain general) then the only reason that it is not being
applied right must be due to the personal failings of the investigators. They
whore after the wrong gods: fame, fortune, impact. But this presupposition is
poddle poop. There is no method and no mechanical recipe for inquiry that
guarantees success if only conscientiously applied. The idea that there could be one is one of the most baleful
legacies of Empiricism.
Empiricism is dedicated to the idea that what there is
largely reflects what you can see. It’s all there in front of view if only you
look carefully. There are no hidden forces or mechanisms. Empiricism is built
on a faith in shallow explanation reflecting a surfacy metaphysics.[2]
In such a world, the idea that there exists a Scientific Method makes sense, as
does the idea that failure to apply it must reflect some kind of pathology. If
you think that science is hard because it involves trying to locate hidden
structure that is only imperfectly reflected in what you can see
(experimentally or otherwise) then a high failure rate is to be expected until
you discover ways of thinking (theory) that tracks this underlying reality.
Only then might you be able to avoid being misled by what you see.
Let me finish with one more observation. Part of the
doomsaying has been prompted by scientific over reach (aka scientism).
Scientists love to preen that they are just motivated by the facts and not
swayed by vulgar conceptions the way regular folk are. Scientists are hard
headed and deserve respect, kudos, reward and deference because science has a
method to check itself and so when science speaks it’s not just opinion. In
other words, when it suits us, we scientists often trumpet the Scientific
Method to claim deference. It’s what makes us “experts” and expertise trumps
mere opinion. But if there is no Scientific Method then there is no expertise
in virtue of having been produced by such a method. There is expertise, but it
must be harder won and it is very limited. If this is right, then there is a
lot less science out there than is advertised, and a lot less expertise than generally
claimed. And that suits me just fine. That’s a doomsday I can both live with
and rejoice in. That, however, has nothing to do with the status of actual
scientific inquiry. It’s doing no worse than before, muddling along and saddled
with all the problems it has always had.
[1]
Though, this said, does anyone think that bio-medicine has made no progress?
Would you rather give up modern methods for those from 1990, or 1950, or 1920?
I am pretty sure that I wouldn’t. This would suggest that despite all the
problems, at least in bio-medicine, we have learned something useful, even if
many basic mechanisms are opaque.
[2]
I stole “shallow explanation” from Chomsky who used this in another (related)
context.
“Gesellschaft” has replaced “Gemeischaft” (love those German words; just what you need to add some suggestion of depth)
ReplyDeleteThey translate mostly straightforwardly as "society" and "community"... the latter should have been Gemeinschaft, though. :-)
Should we expect science in general to run at an efficiency of better than 50%?
Well, we should expect people to run their own experiments often enough before they publish them that other people won't need to do it. One of the first things you learn as a chemistry undergrad (or indeed in highschool) is to do everything in triplicate just as the default baseline before you even think about it.
I have heard that Ray Bradbury is reputed to have said that 90% of everything is junk.
Sturgeon's Law: "90 % of everything is crud."
does anyone think that bio-medicine has made no progress?
Obligatory video with transcript under "show more". :-)
==========
I agree that science is nowhere near doomed, and it's also evident to me that of course there's a single scientific method that can be applied to physics and sociology alike – it's just very simple: falsification and parsimony. However, various problems are real.
1) Publish or perish. Most scientists in most countries and most disciplines* are forced to publish in journals with an impact factor, specifically the highest impact factor they can get; otherwise they can forget about jobs and grants. This has a bunch of consequences. One is the rise of the Least Publishable Unit: publish more frequently by dividing your coherent project into a series of minimalist papers, each of which says almost nothing and was made with the minimum of effort the reviewers required. Of course, most journals won't even let you publish a monograph, because they have space restrictions, and most of those that publish monographs lack an impact factor. (Fortunately, there are now a few exceptions among online-only journals.) Another consequence is that authors have to make their manuscripts look as mind-blowingly groundbreaking as possible, or the high-IF journals will reject them without even sending them out for review. – On the darker end, this is where peer-review rings and similar methods of fraud come from.
2) Money. Disciplines with money in them, like everything vaguely related to medicine, offer an incentive for fraud.
3) Of course there have always been and probably always will be a few personalities who want to be famous or similar things at all costs. One colleague has, I'm told, flat-out said he wants to have more publications than that famous 19th-century colleague who basically cranked out Least Publishable Units daily (most of them almost useless descriptions of new species, commonly the same species several times under different names); he has even resorted to plagiarism. However, this is not some kind of general crisis of science.
4) "The closer you get to humans, the worse the science gets". That's a proverb among biologists and applies within biology as well as in a larger context. The closer you get to humans, the harder it is to isolate one factor and test its effects under laboratory conditions; the closer you get to humans, the more unexamined ideological baggage you probably carry around, and so do the reviewers and editors of your manuscript. To be blunt, my impression is that psychology can be done as a science, but usually isn't.
* Most linguists seem to begin their career by publishing their PhD thesis as a book. In biology that would be suicidal. My thesis consisted of a series of papers, most of which were published before I defended.
Thx for these very thoughtful comments. Let me add a word or two as comment to your comments.
DeleteOf course you should do your experiments carefully. But what carefully means actually changes over time. We do get better and standards should always be vetted and improved. However, I am not sure that this would have the effect many think it would. Like I noted, in some domains of the "embarrassed" sciences (e.g. psych) there are domains where there is no crisis of replication (e.g. visual perception, and, IMO, linguistic judgments). Here the experiments seem to work fine. I would still like to know what the expected baseline is before I get all hot and bothered about the scary numbers being produced. A good baseline is how this compares to days of yore or in domains where there are the same social pressures but more mature sciences. This said, yes, of course, do things well.
Scientific Method: falsification and parsimony. Well, yes. There is also the one that I like from Bridgeman: "use your noodle and no holds barred." So find data, use it and think. If that be method, we are all in favor. But when not much of a method is it? Sort of like "buy low sell high." Good advice, operationalizing it is tougher.
1: publish or perish: Yes this is unfortunate and there are pressures. I have mentioned these in the past but I am not sure what to do about these. Are they worse now? Probably as there were more jobs in the "golden age" (i.e. during the cold war). So long as their are fewer prestige jobs (or jobs at all) than there are people who want them we will have to judge. The best way will be on the basis of the work. This means publications (why? nobody wants a return to the old boy network, right?). We would all love it were careful judgement of the work as a whole were the operating standard. But this was never so, and won't be now. So, until we can all do what we want without fighting for it, this problem, I believe, will remain in one way or another. I also lament its existence.
Yes: money matters. Actually, I suspect that it is not money per se but not enough of it so that fraud (academic rather than just stealing the stuff) pays. If there was enough to go around then the incentive would drop.
I agree completely with 3.
And 4 is perfect. I will steal this adage in the future.
I wrote:
ReplyDeletecommonly the same species several times under different names
Not fraudulently, mind you; just carelessly, based on fragmentary material that he could have recognized as the same thing if he had taken more time.