Experimental work is really hard. Lila once said loud enough for me to hear that you need to be nuts to do an experiment if you really don't have to. The reason is that they are time consuming, hard to get right and even harder to evaluate. We have recently being finding out how true this is, with paper after paper coming out arguing that much (most?) of what's in our journals is of dubious empirical standing. And this is NOT because of fraud. But because of how experiments get done, reported, evaluated and assessed with respect to professional advancement. Add to this the wonders of statistical methodology (courses in stats seem designed as how to manuals for gaming the system) and what we end up with is, it appears, junk science with impressive looking symbols. Paul Pietroski sent me this link to a book that reviews some of this in social psychology. But don't snigger, the problems cited go beyond this field, as the piece indicates.
I said that statistical methods are partly to blame for this. This should not be taken to imply that such methods when well used are not vital to empirical investigation. Of course they are! The problem is first, that they are readily abusable, and second the industry has often left the impression that facts that are statistically scrutinized are indubitable. In other words, the statistical industry has left the impression that facts are solid while theories are just airy fairy confabulation, if not downright horse manure. And you get this from the greats, i.e. how theory is fine but in the end the test of a true theory come from real world experiments yada yada yada. It is often overlooked how misleading real world experiments can be and how it often takes a lot of theory to validate them.
I think that there is a take home message here. Science is hard. Thinking is hard. There is no magic formula for avoiding error or making progress. What makes these things hard is that they involve judgment and this cannot be automated or rendered algorithmically safe. Science/thinking/judgment is not long division! But many think that it really is. That speculation is fine so long as made to meet the factual tribunal on a regular basis. On this view, the facts are solid, the theories in need of justification. Need I say that this view has a home in a rather particular philosophical view? Need I say that this view has a name (psst, it starts with 'Emp…). Need I say that this view has, ahem, problems? Need I say that the methodological dicta this view favors are misleading at best? We like to think that there are clear markers of the truth. That there is a method which if we follow it will get us to knowledge if only we persevere. There isn't. Here's a truism: We need both solid facts and good theories and which justify which is very much a local contextual matter. Facts need theoretical speculation as match as theoretical speculations need facts. It's one big intertwined mess, and we forget this, we are setting ourselves up for tsuris (a technical term my mother taught me).