Here are a pair of posts by Andrew Gelman in issues that you
might find interesting.[1]
The first (here)
is on peer review. It notes the principle virtues and some of the drawbacks of
the process. The main virtue, as he sees it, is that it serves as a check on
coherence and egregious data mishandlings. The main vice is that (i) peer
review mainly enforces the accepted wisdom and (ii) real data problems are too
hard for reviewers to find as serious procedural and data vetting is way beyond
what it is reasonable to ask a reviewer to do. Thus, a publication in a peer
reviewed journal serves largely as a stamp that the paper reflects the views
that the journal thinks are reasonable. Thus, the paper is a good example of
the kinds of papers that the journal publishes. Of course, should the journal’s
judgment of what is worthy be off the mark then the fact that something is
published therein carries the obvious consequences. The radical conclusion is
that peer review is mildly useful but hardly a guarantee that what is being
published is likely to be true, roughly on the mark, or even interesting. It
reflects the consensus world-view of the journal. That’s it.
I think that Gelman makes an important point here. I have
ranted endlessly on how strictly theoretical work in GG is effectively
considered journal inadmissible. It is not strictly speaking impossible to get
something like this in (though with the exception of Chomsky pieces I don’t
think I’ve ever seen such a paper in our leading journals), but for all
practical purposes it is. A publishable paper has to have a certain structure,
and theoretical investigations rooted in exploring the basic concepts and the
ties between them (rather than the empirical consequences of some set of
concepts (and this down to actual language specific numbered examples of data
in a specific language)) are submitta non
grata. Don’t bother. No chance.
Indeed, I would go quite a bit further. In my experience
exploring ideas empirically that fall on the wrong side of what is considered
the right view even when couched in the same idiom as the standards are treated
with great skepticism. I know more than a few examples of papers exploring, for
example, the virtues of the Movement Theory of Control that are harshly treated
simply because they are exploring the virtues of this theory. In other words, the
unstated assumption is that theories that are considered wrong cannot have interesting properties
therefore they should not be investigated and cannot have interesting empirical
or conceptual consequences. I believe that this encapsulates the essence of the
anti-theoretical worldview. And, from my experience, it is pervasive in our
little corner of the sciences.[2]
At any rate, take a look at Gelman’s piece for some (IMO, salutary)
push back against the idea that what makes science so reliable and insightful
is the peer review process.
The second post (here)
is equally interesting. It notes that there are many reasons to collect data,
only one of which is to test a
theory. The discussion is couched in a distinction between testing and
generating theories. It serves as a useful antidote to the idea that looking
for generalizations in the data is somehow not being scientific (i.e. chasing
significance is evil). It’s not that looking for patterns is bad, rather confusing
generating hypotheses (i.e. looking for patterns) is not the same as testing
hypotheses. Correct. They are not the same. But as Gelman notes, generating
hypotheses (especially non-trivial ones) is not easy and needs nurturing. Need
I add that this is what theoretical speculation is also for? But, I already
ranted about this above, so no more here.
At any rate, two interesting posts which you might enjoy.
[1]
Thx to Bill Idsardi for first pointing me to Gelman’s blog.
[2]
Part of my physics envy is rooted in my admiration for the tolerance physicists
have for thought-experiments and the investigation of models that though they
know them to be empirically inadequate. See here for discussion
of just one example of this kind of theoretical investigation. The money
sentence for current purposes is the following: “The
theories involved are generally not
viable models of the real world (my emphasis NH), but they have certain
features, such as their particle content or high degree of symmetry, which make
them useful for solving problems in quantum field theory and quantum gravity,
nonetheless contain interesting features that they hope to incorporate in more
adequate accounts down the road.”
No comments:
Post a Comment