tag:blogger.com,1999:blog-5275657281509261156.post1370567029196611693..comments2024-03-28T04:04:55.806-07:00Comments on Faculty of Language: More on reviewingNorberthttp://www.blogger.com/profile/15701059232144474269noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-5275657281509261156.post-66144418544584309752015-01-23T17:49:46.140-08:002015-01-23T17:49:46.140-08:00Perhaps a bit tangential, but I don't think ar...Perhaps a bit tangential, but I don't think arbitrary processes are necessarily fair if everyone is subject to the same capricious decision procedures. Arbitrary processes are <i>only</i> fair if everyone is subject to the same capricious decision procedures <i>and</i> the starting conditions for everyone are the same (or at least equal/equitable in some relevant sense).<br /><br />Perhaps this doesn't matter when viewed solely from the context of conference acceptances, but I can imagine it mattering when viewed from a larger context. It's hard for me to say much on this since I'm at a point in my career still very far removed from something like applying for a job and thus don't really have any familiarity with the process. However, I can at least imagine that it might matter when looking for a job if you have other things working against you—such as graduating from an institution that doesn't have 'prestige' associated with it, (structural) sexism, (structural) racism, etc.—and your competition doesn't have those things working against them. In other words, if one thinks about the randomness just in the context of whether a particular paper is accepted to a conference, then sure, it might be fair. But, in the larger picture, such randomness probably isn't fair to someone who has other things working against them from the start.Anonymoushttps://www.blogger.com/profile/14613154152690142744noreply@blogger.comtag:blogger.com,1999:blog-5275657281509261156.post-43712907677646294632015-01-23T16:53:11.033-08:002015-01-23T16:53:11.033-08:00I completely agree, that is a great feature of Eas...I completely agree, that is a great feature of EasyChair (one of the many reasons why it is preferable to EasyAbs). Unfortunately it seems that many reviewers aren't actually interested in discussions and ignore these comments --- at least that's been my experience so far, but I'll also admit that the sample size is fairly small since most conferences I've reviewed for had the feature deactivated. Of course there's also the issue that reviews sometimes aren't uploaded until one or two days before the deadline, which might limit a reviewer's willingness to rethink their evaluation.<br /><br />I would actually like to see an even more open system where everybody with an EasyChair account can read the abstract and comment on reviews (or at least give them a point rating). That has at least three advantages: 1) additional incentive to write thoughtful reviews rather than one-liners or rants, 2) more feedback from the community and all the positives that entails, 3) students get some insight on the reviewing process and how to write good abstracts.<br /><br />One could even enforce that reviewers for an abstract may only participate in the discussion until the reviewing deadline while comments may be made later on, too. Then it is in the reviewer's interest to upload their review asap so that they can defend their evaluation as long as possible.<br /><br />Finally, usernames can be hidden if people wish to remain anonymous, but there should be an option for non-anomyous reviews and comments, which in turn should count more than anonymous ones.<br /><br />tl;dr crowdsource the reviewing processAnonymoushttps://www.blogger.com/profile/07629445838597321588noreply@blogger.comtag:blogger.com,1999:blog-5275657281509261156.post-31264821165981108772015-01-23T15:52:12.785-08:002015-01-23T15:52:12.785-08:00Dominique Sportiche sent me this comment which I s...Dominique Sportiche sent me this comment which I share with you with his permission. It is a positive suggestion, rather than just more whining. Comments appreciated.<br />*****<br /><br />Hi Norbert, I did not have grants in mind so much as conferences and was trying to think of ways to improve the selection process which I see pretty much as a lottery (I wonder if people agree with me), a thought that is consistent with this NIPS story, perhaps even more so in linguistics where standards can be fuzzy.<br />This is bad for the field, and also for younger researchers whose acceptance at conferences is a big deal. <br /><br />All this to say that I prefer easychair to easyabs (linguist list) in particular because it has a feature - which must be truend on - that I really like (even though I may be in a minority??)<br /><br />Once a review of an abstract is entered, the reviewer can (anonymously) see all other reviews, and comment on them, and it is possible <br />to change one's grade or one's comments on the basis of comments by other reviewers (the grade history remains however). <br />For conferences I review which have this feature turned on, I always read the other reviews. I have changed my grade on the basis of remarks by other reviewers. I have commented on other people's reviews.<br /><br />It seems to me that this is very valuable: it keeps the reviewers more <br />honest (they know someone could look very carefully at what they <br />write), allows for discussion among reviewers, provides further input <br />for the selection committee and makes the whole process more transparent.<br />I see no downside, are there some?<br />Norberthttps://www.blogger.com/profile/15701059232144474269noreply@blogger.com