Monday, November 12, 2012

Publication Blues


Get a bunch of syntacticians in a room and it’s not long before they begin to regale each other (and themselves) with titillating tales of publishing porn:

Do you know that it took me two and half years to get that paper into LI? You should see these absurd reviews that I got from NLLT, two say publish and one guy says reject and the editor doesn’t see that his comments are full of S*&$. Language why would anyone publish there?  They hate theoretical linguistics!! Cognition got the people I was criticizing to review the paper and they sank it! I got a review from Syntax longer than the paper I submitted, and most of it just missed the point.

You can add your own favorite vignettes, I am sure.  What surprised me this weekend is to discover that publishing porn is not restricted to syntax or even linguistics but extends quite broadly across academia (see below). I had always thought that the disgruntlement arose because of the dearth of journal outlets for publishing in syntax. The big three -LI, NLLT and Syntax- have relatively few pages between them. Language doesn’t (won’t?) publish hard-core theoretical syntax for love or money (btw, this is quite odd as the professional society journals in economics, philosophy, physics etc. are where the newest stuff is showcased. Language, in contrast, is the last place to look for new cutting edge syntax (or, from the little that I can tell, semantics or phonology)). Limited pages probably matter.

Page limits also make editors lives more troublesome. To live with the restrictions the editorial decision is not ‘take the good ones, reject the bad’ but ‘take these good papers and not those.’ Couple this with the fact that IMHO there is a lot of good syntax being done and the result is that there have to be arbitrary ways of managing the flow.  One way of doing this is restricting submissions (e.g. the one-paper-under-review-at-a-time policy at LI). Another is queu management (e.g. the long lag time between submission-review-resubmission-re-review-…-acceptance-publication). A third is content management (e.g. the dearth of debate in the pages of the journals, a policy that frees up space). 

Editors also face the “herding cats” problem.  I don’t know about you, but reviewing papers is not my favorite pastime. I know that this is important and a contribution to the field and important for people’s careers yada, yada, yada. But…you can fill in the rest, I suspect.  Part of this results from the expectations people have of reviews.  They are “supposed” to be long and very detailed and very thorough. Even typos! Rather than expecting a judgment about the relevance or importance of the paper with a review of the central argument (something that would be about 1-2 pages long), the reviewer is expected to effectively write a reply, moreover a reply that won’t be published.  There is, I am sure, a lot of interesting discussion being carried out furtively in the review process that deals with interesting (because though contentious) issues that really should be public but never will be.  

To add to the burden, papers are typically very long. In the past natural experiments occasionally emerged that permitted one to measure paper-bloat (this is no longer possible as most journals will not publish previously aired work, even conference proceedings).  Sometimes a paper that first appeared as a conference proceeding was subsequently reissued in more elaborated form as a journal article.  To my hazy recollection, the more detailed paper often failed to convey the basic idea as clearly as the 15 page conference paper did.  There is a kind of “defensive” linguistics that lards many papers, no doubt the result of the thorough vetting process, which serves to obscure the main idea. It sometimes seems like a paper cannot simply leave a problem knowingly unsolved and still expect the paper to make the printed page.  This results in “explanation creep” that often muddies a relatively clear main theme, to the detriment of the insight the paper offers.

There is one final problem, though I cannot tell how endemic it is.  I have occasionally gotten the impression that reviewers like to make sure that papers that gore their favorite oxen don’t see the light of publishing day.  I cannot tell how widespread it is, but I doubt that it is negligible.  Let me relate an anecdote: I was once asked to review a MS for a pretty prestigious press. I read the MS and found myself unconvinced. However, I also concluded that the direction being taken, though not one that I found congenial, represented a common perspective was well done given this perspective and deserved an airing. My review said as much, as well as including some more detailed critical points mainly for the author’s interest. I was contacted by the editors and asked if I thought the MS deserved publication. As my first sentence was something to the effect that this MS should be published as it represented an important perspective on timely matters I was a bit surprised. I was told that it was clear that I disagreed with the ideas and remained unconvinced by the arguments so why would I recommend publication?  I found this odd. Is the standard of publication whether the MS persuades the reviewer that it is true?  I hope not. That is a very high bar. Interesting, ok, provocative, sure, makes you think, check.  But true? Really true?  Nope, too much to ask.

So given all of this what would gruntle me?  I really don’t know, which is partly why I am writing about this here; you know, generate some chatter about the problem, discover that my mood is largely dyspepsia and that I should stop with the fried chicken and anchovy pizza.  Here are some minor proposals:
·      Limit all paper submissions to NELS length 15 2-space pages
·      Limit all reviews to 2 pages
·      Penalize late reviewers (say, deny publication in the journal for some period)
·      Consider ways of migrating all journals to the web where page numbers won’t matter and do so in a PLOS like open format
·      Have a more active editorial board, one that solicits and recommends MS for publication rather than just reviewing adventitious submissions
These are relatively conservative changes that could be implemented relatively quickly. 

I would like to end with a few words on a more radical proposal that caught my eye.  It’s called “A Rant on Refereeing.” It suggests that we dump reviewing altogether as it has become a way of stifling the circulation of ideas rather than promoting them. There are several linked papers (here, here, here) that are also worth reading, some in favor some not. The main idea is that web based circulation of papers should replace the refereed journal format and that there are ways of managing the downsides (how to assure quality (answer: no magic bullet even in journals), how to finesse the promotion and tenure role of reviewed publications (answer: letters will be more important), how readers will find what’s “important” (answer: not easy but aggregate sites will naturally arise to organize the paper flow) etc.  It is very provocative and if you are interested in these issues I suggest you take a look.

Ok, enough: I am curious about what others think about these things, so if you are inclined, let me know.

3 comments:

  1. As I have spent the past year largely prepping non-linguistic papers I now see that ling is VERY different from the chemistry, chemical engineering, meteorology, and microbiology worlds I've had the chance to be exposed to. First, as we all probably knew already, the manuscripts are on the order of 1-10 journal pages and therefore bite off far smaller chunks of problem. All the detail gets dumped in the Supporting Information, which goes straight to the web. Second, manuscripts are not double-blind reviewed (the authors are known to the reviewers but not vice-versa). This means that for better or for worse, one's reputation goes with the manuscript: if somebody with a well-known lab writes a paper with surprising results, it's far more likely to get published than if it's a relative outsider, who is likely to receive extreme skepticism. Third, reviewers only get about 2-3 weeks of turnaround time (except for Nature and Science, it seems) and the authors are similarly limited in the time they have to re-submit (a month or two). I don't think all of these practices necessarily promote responsible science or responsible reviewing, but some might.

    ReplyDelete
  2. The journal Lingua has no page limit, and we strive to do what Norbert proposes, i.e. to make a journal into a forum for circulating ideas, even those we do not agree with, rather than into a stamp-of-approval machine publishing your article 3 years after submission. We may not be the big three, but we are open to what they tend to publish. We publish about 2500 pages a year. Our ranking in ERIH is INT1, in SCImago Q1. Our figures show that authors are quite happy with the way their article was being treated. When authors complain about their reviews, I give them a second chance. The problem is that most authors prefer to tell tall stories of the publishing porn variety rather than to engage constructively with the editor. As an editor, I tend not to take reviews that are basically full fledged replies very seriously.

    In my (14 years) experience as a Lingua editor, I have found that it is quite hard to make reviewers write a 2 page review. Most are shorter, on average. I sometimes have to beg reviewers to write more than half a page, or ask for an additional review because that half page does not contain anything useful.

    Editorial boards are difficult to get into active mode. Believe me, I have tried, both at Lingua and at the Linguistic Variation Yearbook. Most icebergs are more nimble. I have also mooted the idea of penalizing late reviewers with fellow editors at other journals, and the reaction was generalized horror: we depend on reviewers, so we cannot chide them in any way. We are now thinking instead of an award for the most active reviewer, funded by Elsevier, the publisher of Lingua. Our goal a Lingua is to reduce turnaround times between submission and decision. The record suggests that we are successful. However, this takes daily vigilance on behalf of the editors, who are not exactly rewarded for any of it.

    I like the idea of shorter papers. Maybe they could be promoted by promising a shorter decision route for them, with quick, editor-only yes or no decisions. Glad to have your thoughts on this

    ReplyDelete
  3. Glad to hear that there is some attempt at rational management. My beef with lingua is the cost. As you no doubt know there is currently a little revolt in academia against Elsevier. I confess to being sympathetic. I believe that the future belongs to open source publication, with a lot of the real "reviewing" taking place by online comment. However, the future is not yet on us and I think that shorter papers with fast turn around times is a very good idea. One obvious way of cutting down size is cutting down on example sentences. All but a couple can be posted online on a dedicated site. However, the real saving would come from realizing that a good paper makes one or two points and we should expect even good ones to have some unresolved issues. Ad hoc verbiage to finesse these does not help and should be actively discouraged.

    ReplyDelete