Monday, September 30, 2013

Evilicious


As I have noted in the past, Andrew Gelman, a statistician at Columbia (and someone whose work I both follow and have learned from) has a bee in his bonnet about Marc Hauser (and, it seems, Chomsky) (see here).  In many of his posts he has asserted that Hauser fraudulently published material that he knew to be false, and this is why he takes such a negative view of Hauser (and those like Chomsky who have been circumspect in their criticisms).  Well, Hauser has a new book out Evilicious (see here for a short review). Interestingly, the book has a blurb from Nicholas Wade, the NYTs science writer that covered the original case. More interestingly the post provides links to Wade’s coverage for the NYT of the original “case” and because I have nothing better to do with my time I decided to go back and read some of that coverage.  It makes for very interesting reading.  Here are several points that come out pretty clearly:

1.     The case against Hauser was always quite tenuous (see here and here).  Of the papers for which he as accused of fabrication, two were replicated very quickly to the satisfaction of Science’s referees. The problem for these was not fabrication, but not having the original tapes readily available.  Sloppy? Perhaps. Fraud? No evidence here.
2.     Of the eight charges of misconduct, five involved unpublished material.  This is a very high standard. I would be curious to know how many other scientists would like to be held responsible for what they did not publish.  A Dr. Galef from McMaster University (here) notes incredulously in the NYT (rightly in my view): “How in the world can they get in trouble for data they didn’t publish?”
3.     L’Affaire Hauser then devolves down to one experiment published in Cognition that Galef notes “was very deeply flawed.” However, as I noted (here) the results have since been replicated. That, like the Science replications, suggests that the original paper was not as flawed as supposed. Sloppy? Yes. Flawed? Perhaps (the replication suggests that what was screwed up did not matter much). Fraud? Possible, but the evidence is largely speculative.
4.     The NYT’s pieces provide a pretty good case that (at least one) outside scientists, who reviewed the case against Hauser, thought that “the accusations were unfounded.” Maybe Galef is a dupe, but his creds look ok to me. At any rate, someone with expertise in Hauser’s field reviewed the evidence against him and concluded that there was not enough evidence to conclude fraud, or anything close to it.
5.     Galef further noted that the Harvard investigating committee did not include people familiar with “the culture of an animal behavior laboratory,” which has “a different approach to research and data-keeping” than what one finds in other domains, especially the physical sciences from which the members of the Harvard investigating committee appeared to come from. I’m pretty sure that few behavioral or Social scientists would like to be subject to the standards of experimental hygiene characteristic of the work in the physical sciences. Is it possible that in the investigation that set the tone for what followed, Hauser was not judged by scientists with the right background knowledge?
6.     The final ORI report concentrates on the retracted (subsequently replicated) Cognition article (here). The claim is that “half the data in the graph were fabricated.” Maybe. It would be nice to know what the ORI based this judgment on. All involved admit that the experimental controls were screwed up and that some of the graphs, as reported, did not reflect the experiments conducted. I have no trouble believing that there was considerable sloppiness (though to repeat, it seems not to have been fatal given the subsequent replication), but this would not support ORI’s assertion of fabrication, a term that carries the connotation of intentional deceit.  I suspect that the ORI judgment rests on the prior Harvard findings. This leaves me thinking: why did this particular set of data get through the otherwise pretty reliable vetting process in Hauser’s lab, one that nixed earlier questionable data?  Recall, the data from five of the other investigated papers were vetted before being sent out and as a result the reported data were changed. What happened in this case? Why did this one slip through? I can understand ORI’s conclusion if tons of published data was fabricated. But this is precisely what there is no evidence for. Why then this one case?  Is it so unlikely that some goof up casued the slip? As Altmann, one of Hauser’s early accusers, notes in the NYT: “It is conceivable that the data were not fabricated, but rather that the experiment was set up wrong, and that nobody realized this until after it was published.” In the end, does the whole case against Hauser really devolve to what happened in this one case? With effectively one set of controls? Really?

I suspect that the real judgment against Hauser rests on the fact that he resigned from Harvard and that the Harvard committee initially set up to investigate him (but whose report, so far as I know, has never been publically made available) decided he was guilty. Again, as Altmann notes in the NYT: his earlier accusation was “heavily dependent on the knowledge that Harvard found Professor Hauser guilty of misconduct.” This, coupled with the thought that Hauser would not have quit were he innocent. But, it’s not a stretch to think of many reasons why a non-guilty person might quit, being fed up with one’s treatment perhaps topping the list. I am not saying that this is why Hauser resigned. I don’t know. But to conclude that he must be guilty because he left Harvard (who but a criminal would leave this august place, right? Though on second thought…) is hardly an apodictic conclusion. In fact, given all the evidence to date, it strikes me that the charges of fraud have very little evidential support, and that there are decent alternative explanations for what took place absent the intention to deceive. In fact, I would suggest that given the gravity of the charge, we should set a pretty high evidential bar before confidently declaring fraud. As far as I can tell from reading what’s in the NYT, this bar has not been remotely approached, let alone cleared.

I think that this will be the last post on this topic for me.  I have harped on it because, frankly, I think Hauser got a raw deal, from what I can tell, and that the condemnations he drew down on himself struck me as a tad too smug, uninformed, and self satisfied. As I’ve said before: fraud does not seem to me to be the biggest source of data pollution, nor the hardest one to root out.  However, such charges are serious and should not be leveled carelessly. Careers are at stake. And from what I can tell, in this particular case, Hauser was not given the benefit of the doubt, as he should have been.   

5 comments:

  1. This post reveals a very pleasant side of Norbert. His willingness to extend the benefit of the doubt as extensively as he does is touching to say the least. I fully agree that one ought to rule out carefully any and all alternatives before accusing someone of fraud. Whether 'sloppiness' resulting in publication of flawed articles sufficiently supports the charge of 'academic misconduct' is an independent question that really does not need to concern us when we are approaching the higher moral grounds Norbert wants us to move towards.

    Now given all what Norbert wrote I am looking very forward to a future post applying the same standards requested for our treatment of Hauser to the treatment of Everett. In my books, accusing someone of being a charlatan requires probably more [but certainly not less] evidence than accusing someone of fraud. So I trust in this future post we will be looking at how Chomsky discharged his obligations to clear the evidential bar. I am looking very forward to studying the novel evidence because from what I have learned so far, i am inclined to say:

    " given all the evidence to date, it strikes me that the charges of CHARLATANERY have very little evidential support, and that there are decent alternative explanations for what took place absent the intention to deceive. In fact, I would suggest that given the gravity of the charge, we should set a pretty high evidential bar before confidently declaring CHARLATANERY. As far as I can tell from reading what [was in Folha de Sao Paulo, Feb 1st, 2009], this bar has not been remotely approached, let alone cleared."

    ReplyDelete
    Replies
    1. This is a reply to Christina’s post about a question that came up in the Chronicle of Higher Education (CHE) discussion of 2012. At the time I asked if in fact Daniel Everett had been accused, as was suggested in the CHE, of (1) charlatanry, (2) fabrication of data and (3) racism in his fieldwork of many years with the indigenous community of Pirahã speakers, and especially whether there was evidence for (2) and (3).
      http://chronicle.com/blogs/linguafranca/2012/03/28/poisonous-dispute/
      The three accusations are related because they appear to be part of an effort to discredit Everett’s work. If, on the other hand, the accusations are based on evidence the profession should be provided with this evidence or a summary of it. In addition, (3) in particular has been reported to have been a consideration in evaluating whether Everett should continue to have access to the Pirahã community by the Brazilian government agency (FUNAI) charged with granting such access; and (1) and (2) have been pointed to as being relevant to this evaluation. A “charlatan” according to Webster’s Third Edition is a “fraud.”
      At the time, I asked if anyone could help me locate the published record of the accusation (1), attributed to Noam Chomsky, that Everett is a “charlatan.” The following is the account of an exchange, written by the editor(s) of the Science page of Folha de San Paulo appearing in the September 1, 2009 issue. The title of the report reads: "Ele virou um charlatão", diz Chomsky [“He became a charlatan,” says Chomsky].
      http://www1.folha.uol.com.br/fsp/ciencia/fe0102200904.htm
      The second paragraph begins:
      "Ele virou um charlatão puro, embora costumasse ser um bom linguista descritivo.”
      [He became a pure charlatan, even though he was once a good descriptive linguist]. The passage appears in quotes, suggesting that these are the words of Noam Chomsky.
      In the interest of fairness, it needs to be determined whether the quoted passage attributed to Chomsky was actually pronounced: (a) in an interview on-the-record, or if it was (b) off-the-record and then published without permission or knowledge of Chomsky, or that it is (c) a misquotation, in the clear case of (b) or (c) an irresponsible report by the author.
      Given that a version of the “became a pure charlatan” charge has been widely circulated on the internet, I am asking which of the three (a), (b) or (c) is the case, and if Professor Chomsky stands by the characterization of Everett being a “pure charlatan.”
      I’m still waiting to see if anyone can help me clarify accusations (2) and (3). Thank you in advance.

      Delete
  2. fraud does not seem to me to be the biggest source of data pollution, nor the hardest one to root out.

    This, of course, doesn't imply that nothing should be done about fraud, regardless of its relative size in the pool of data pollution sources. Folks like Gelman are also putting plenty of energy into dealing with other sources (e.g., search for Type M and Type S errors on Gelman's blog - he frequently writes about various problems with uncritical reliance on p-values, likely a much bigger source of pollution).

    Just to be clear, I agree that charges of fraud are serious and should not be leveled carelessly.

    ReplyDelete
  3. I'll quickly add that I know that you (Norbert) haven't said we shouldn't do anything about fraud. My point was just that multiple issues with scientific integrity can be, and are being, simultaneously addressed.

    ReplyDelete
    Replies
    1. Yes, multiple issues can be and are. I agree that fraud is bad and should not occur. What I worry about is that this "easy" case can dominate discussion and divert intellectual resources from the more serious problem. I would love to know how serious fraud is. I suspect, but do not know, that it is not that serious AS A PRACTICAL MATTER. In this respect, it MAY be a little like voter fraud, which we know is very low, though not nonexistent. So, I would personally like to know how big a practical deal it is. This would determine how seriously we need to be in combatting it.

      Second, I think that it is a serious moral charge that does have real consequences, e.g. Hauser resigned over this, Imanishi Kari and Baltimore spent 2-3 years defending themselves (successfully) against similar charges years ago. So, before one levels such charges one needs to have a very good case. That's what I found so odd when reading the reports: the charges were quite weak, at least they seemed so to me. And this is really bad: if the problem is not urgent AND the consequences on individuals so charged are then leveling such charges should be done only in the clearest possible circumstances. I don't deny that such exist. I am not nearly convinced that this is such a case.

      To end: yes I agree that it is bad etc. etc. The practical question is what to do about it and when. Thx for your judicious comments.

      Delete