Here are a pair of posts by Andrew Gelman in issues that you might find interesting.
The first (here) is on peer review. It notes the principle virtues and some of the drawbacks of the process. The main virtue, as he sees it, is that it serves as a check on coherence and egregious data mishandlings. The main vice is that (i) peer review mainly enforces the accepted wisdom and (ii) real data problems are too hard for reviewers to find as serious procedural and data vetting is way beyond what it is reasonable to ask a reviewer to do. Thus, a publication in a peer reviewed journal serves largely as a stamp that the paper reflects the views that the journal thinks are reasonable. Thus, the paper is a good example of the kinds of papers that the journal publishes. Of course, should the journal’s judgment of what is worthy be off the mark then the fact that something is published therein carries the obvious consequences. The radical conclusion is that peer review is mildly useful but hardly a guarantee that what is being published is likely to be true, roughly on the mark, or even interesting. It reflects the consensus world-view of the journal. That’s it.
I think that Gelman makes an important point here. I have ranted endlessly on how strictly theoretical work in GG is effectively considered journal inadmissible. It is not strictly speaking impossible to get something like this in (though with the exception of Chomsky pieces I don’t think I’ve ever seen such a paper in our leading journals), but for all practical purposes it is. A publishable paper has to have a certain structure, and theoretical investigations rooted in exploring the basic concepts and the ties between them (rather than the empirical consequences of some set of concepts (and this down to actual language specific numbered examples of data in a specific language)) are submitta non grata. Don’t bother. No chance.
Indeed, I would go quite a bit further. In my experience exploring ideas empirically that fall on the wrong side of what is considered the right view even when couched in the same idiom as the standards are treated with great skepticism. I know more than a few examples of papers exploring, for example, the virtues of the Movement Theory of Control that are harshly treated simply because they are exploring the virtues of this theory. In other words, the unstated assumption is that theories that are considered wrong cannot have interesting properties therefore they should not be investigated and cannot have interesting empirical or conceptual consequences. I believe that this encapsulates the essence of the anti-theoretical worldview. And, from my experience, it is pervasive in our little corner of the sciences.
At any rate, take a look at Gelman’s piece for some (IMO, salutary) push back against the idea that what makes science so reliable and insightful is the peer review process.
The second post (here) is equally interesting. It notes that there are many reasons to collect data, only one of which is to test a theory. The discussion is couched in a distinction between testing and generating theories. It serves as a useful antidote to the idea that looking for generalizations in the data is somehow not being scientific (i.e. chasing significance is evil). It’s not that looking for patterns is bad, rather confusing generating hypotheses (i.e. looking for patterns) is not the same as testing hypotheses. Correct. They are not the same. But as Gelman notes, generating hypotheses (especially non-trivial ones) is not easy and needs nurturing. Need I add that this is what theoretical speculation is also for? But, I already ranted about this above, so no more here.
At any rate, two interesting posts which you might enjoy.
 Thx to Bill Idsardi for first pointing me to Gelman’s blog.
 Part of my physics envy is rooted in my admiration for the tolerance physicists have for thought-experiments and the investigation of models that though they know them to be empirically inadequate. See here for discussion of just one example of this kind of theoretical investigation. The money sentence for current purposes is the following: “The theories involved are generally not viable models of the real world (my emphasis NH), but they have certain features, such as their particle content or high degree of symmetry, which make them useful for solving problems in quantum field theory and quantum gravity, nonetheless contain interesting features that they hope to incorporate in more adequate accounts down the road.”
Post a Comment