More on experimental syntax

In a recent post (here), I discussed a really nice result that our experimental syntacticians (ES) have delivered wrt island phenomena. As I noted, they have shown that ES can find (and has found) a distinctive super additivity signature of island violations even in highly acceptable sentences in Scandinavian (i.e. independently of overall acceptability). This strongly challenges (IMO debunks) the long held view that islands are parametric, a view that makes little sense when viewed through the lens of the Poverty of Stimulus (PoS) argument. Given that a well-formed PoS argument is virtually apodictic, this is what we should have expected all along. Given ES tools, our expectations have been realized. Chalk this one up as a victory for PoS logic.

I also remarked on another nice feature of this application of ES methods; it has yielded a really novel result. We actually could not “see” island effects using more conventional binary judgment (* vs ok) methods and this misled syntacticians. In other words, opening one’s mind to the conceptually impossible (i.e. “results” that violate PoS logic), encourages (a comfortable, yet fraudulent) skepticism regarding our best-grounded insights (e.g. concerning the universality of island effects) and this in turn leads research in the wrong direction. Well-grounded results, like Ross’s islands should be fixed points in ongoing inquiry and they cannot serve this function if GGers doubt their standing. At any rate, hooray for Kush, Londahl, Sprouse and Almeida.  They have done something both valuable and new.

I wanted to offer up two other results based on ES methods that I believe are “novel” in the sense of being heretofore less easily investigated without ES tools.  I was involved with two of these and so I know something about them. Hence what follows has the whiff of self-promotion. Believe me when I tell you that my contribution to both efforts was very minimal.  I invite readers to inform us of other new kinds of results using ES methods.

The two I know about are both chapters in the book that Jon Sprouse (with a small amount of help from moi) edited (here). The first is the chapter by Johannes Jurka on subject islands in German.  This paper examines these uisng ES methods. It shows that extracting out of subjects is worse than extracting out of objects and (this is the fun part) that extracting out of non-agreeing specifiers lies midway between the two. Thus, extracting out of specifiers is harder than doing so out of complements even in the absence of much agreement or any evidence of displacement.[1] This effect appears to be independent of freezing, which Jurka notes seems to function as an independent factor. Indeed, extraction out of external arguments is always harder/worse than extracting out of complements regardless of movement or agreement. This is very much worth knowing for, if correct, it suggests that subject islands cannot be reduced entirely to freezing effects. In fact, one can go a little further: so far as I know there are not many current theories (as opposed to Barriers style accounts that might be able to accommodate this asymmetry via differential L/theta-marking) that predict that extraction out of a specifier per se should be harder than extraction out of a complement.[2] Thus, Jurka’s ES empirical work suggests that we need to think quite a bit harder about the subject side of CED effects.

The second paper is one that I co-authored with Brian Dillon (again, he did most of the work). The paper presents ES evidence that what islands are sensitive to is syntactic structure.  “Opposed to what?”, you might be wondering. Well, opposed to semantic structure in particular. There once was a time when people tried to reduce island effects to semantic ones (I faintly recall Rodman trying to do this in the early 80s). And given the close connection between semantic hierarchical structure and syntactic hierarchical structure it is very hard to tease apart whether island effects are actually syntactic.  In fact, doing this requires holding the semantics constant and manipulating only the syntactic form, and this is not easy to do while keeping all other factors more or less the same. The above paper does this by focusing on extraction form small clauses and their semantically (largely) identical nominal counterparts. Thus pairs like (1a,b) mean the same thing (both denote events with the same John/Mary participants) but it turns out that extracting out of the nominal complement is harder than extracting out of the small clause complement.

(1)  a. Mary heard John clumsily attempt to kiss Mary

b. Mary heard John’s clumsy attempt to kiss Mary

There are, of course, all sorts of manipulations required on this basic theme to control for all sorts of things (e.g. definiteness effects among others (btw, Brian did this)), but the basic result is that extracting out of nominal event denoting complements is harder than extracting out of their small clause counterparts and this seems to be entirely due to the fact that one is nominal and the other is not.  Again, many of the judgments are in the acceptabl-ish territory so we have a kind of subliminal island effect. At any rate, to my knowledge this is one of the first attempts to pin down the claim that islands are syntactic effects, i.e. effects sensitive to syntactic structure.  In this particular case what matters is the distinction between a nominal and a sentential complement (i.e. labels seem to make a difference here).  If this is correct, then island effects are syntactic phenomena at least in the sense that the relevant primitives need to allude to the syntactic features of constituents.

Again, I mention these papers because they try to do something new with ES methods, they try to find effects that are hard to spot using the easier more convenient ask-your- next-door-neighbor methods. Let me repeat, lest this be grossly misunderstood, that I am NOT endorsing the view that we all now do ES experiments to ground our data. This is not necessary in general (again as Sprouse and colleagues have argued successfully IMO). However, there seem to be times when ES methods can yield new insights, and when this is so, we should not be reluctant to use these (more expensive) methods. When might this be? 

I suspect that it is not an accident that ES has been most successfully applied to island phenomena. Why? Because we know a hell of a lot about islands, both empirically and theoretically. Much of this knowledge is based on data collected in the standard way, and the conventional methods have clearly proven to be very productive.  However, what the ESers have shown is that when we get down to more refined and filigree issues especially in areas that we know a lot about, it should not be surprising that we might need more careful empirical probes.[3] In fact, as I’ve mentioned elsewhere, what we should find surprising is that the very crude methods we have used till now have proven to be so robust and subtle. This surely tells us something about FL, namely that it leaves very deep footprints so that even slapdash methods suffice to probe it. However, we should never have expected this charmed state of affairs to continue forever. Happily, ES, which is pretty easy to deploy, provides another method for probing structure. Happily it largely leads to the same results in the well-understood cases. Happily, it sometimes delivers new insights.  All in all, this is all a very happy fact. So, be happy and be catholic in your choice of tools.

---

[1] An interesting feature of Johannes’ results is that they provide an argument for binary branching. How so? Well, IOs and DOs cannot both be complements given his results. This should be possible were non-binary branching possible. Though I believe that binary branching is in fact a condition on constituency, there are not all that many arguments in its favor, so far as I know. There are the Barss-Lasnik data, bit aside from that I don’t know of many others. Do you?

[2] The one that I do know of (and that Jurka cites) is Uriagereka’s version of multiple spell out, which, to my knowledge is not widely investigated or accepted.

[3] Akira Omaki has used ES methods to explore a Relativized Minimality approach to WH-islands (here). I hope to discuss his results in a forthcoming post on RM. But no reason for you to wait to read it.

Dreams of a unified theory; a great big juicy problem (Yay!!)

The intricacies of A’-syntax is one of the glories of GB.[1]  The unification of Ross’s islands in terms of subjacency and the discovery of ECP dependencies (especially the adjunct/argument distinction) coupled with wide ranging investigations of these effects in a large variety of different kinds of languages marked a high point in Generative Grammar. This all changed with the Minimalist (M) “Revolution” (yes; these are scare quotes). Thereafter, Island and ECP effects mostly fell from the hot topics list (compare post M work with that done in the 80s and early 90s where it seemed that every other paper/book was about A’-dependencies and their island/ECP restrictions). Moreover, though early M was chock full of discussions of Superiority, an A’-effect, it was mainly theoretically interesting for the light that it threw on Minimality and Shortest Move/Attract rather than how it bore on Islands or the ECP. Indeed, from where I sit, the bulk of the interesting work within M has been on A rather than A’ dependencies.[2]

Moreover, whereas there has been interesting research aiming to unify various grammatical modules, subjacency and ECP have resisted theoretical integration, at least interesting versions thereof. It is possible, indeed easy, to translate bounding theory or barriers into phase terminology.[3] However, there is nothing particularly insightful gained in doing this. It is also possible to unify Islands with Minimality given the right use of features placed in appropriate edge positions, but IMO little has been gained to date in so proceeding. So Island and ECP effects, once the pride of theoretical syntax have become a backwater and a slightly embarrassing one for three related reasons.

First, though it is pretty easy to translate Subjacency (viz. bounding theory) in phase terms, this translation simply duplicates the peccadillos of the earlier approaches (e.g. we stipulated bounding nodes, we now stipulate (strong) phases, we stipulated escape hatches (C yes, D no) we now stipulate phase edges (both which phases have any to use and how many they have)).

Second, ad hoc as this is, it’s good compared to the problems the ECP throws up. For example, the ECP is conceptually a trace licensing requirement. Where does this leave us when we replace traces with copies as M does? Do copies need licensing? Why if they are simply different occurrences of a single expression? Moreover, how do we code the difference between adjuncts versus arguments?  What makes the former so restricted when compared to the latter?

Last, the obvious redundancy between Subjacency and the ECP raises serious M questions. Both involve the same island like configurations yet they are entirely different licensing conditions. Talk of redundancy! One of Subjacency or the ECP is bad enough, but both? Argh!!

So, A’-syntax raises M issues and a natural hope is to dispose of these problems by placing them in someone else’s trash bin. And there have been several attempts, to do just this, e.g. Kluender & Kutas, Sag & Hoffmeister, Hawkins, among others. The idea has been to treat island effects as a reflection of processing complexity, the latter arising when parsers try to relate elements outside an island (fillers) to positions (gaps) within an island.  It is well known that filler/gap dependencies impose a memory/storage cost as the process of relating a filler to a gap requires keeping the filler “live” until it’s discharged in the appropriate position. Interestingly, there is independent psycho-ling evidence that the cost of keeping elements active can depend on the details of the parse quite independently of whether islands are involved (e.g. beginnings of finite clauses induce load, as does the parsing of definites).[4] Island effects, on this view, are just the sum total of these island-independent processing costs. In effect, Islands are just structures where these other costly independently manifested requirements converge. If true, this idea could, with some work, let M off the island hook.[5] Wouldn’t that be nice?

It would be, but I personally doubt that this strategy will work out.  The main problem is that it seems very hard to explain the unacceptability profiles of island effects in processing terms. A recent volume (of which I am co-editor though Jon Sprouse did all the really heavy lifting and deserves all the credit, Experimental Syntax and Island Effects) reviews the basic issues. The main take home message is that when considered in detail, the relevant cited complexity inducers (e.g. definiteness) do not eliminate the structural contributions of islands to the perceived acceptability, though they can modulate it (viz. the super-additive effects of islands remain even if the severity of the unacceptability can be manipulated). Many of the papers in the volume address these issues in detail (see especially those by Jon Sprouse, Matt Wagers, and Colin Phillips). The book also contains good representatives of the processing “complexity” alternative and the interested reader is encouraged to take a look at the papers (WARNING: being a co-editor forbids me in good conscience, from advocating purchase but I believe that many would consider this book a perfect holiday gift even for those with no interest in the relevant intellectual issues, e.g. it’s really heavy and would make a perfect paperweight or door stopper).

A nice companion piece to the papers in the above volume that I have recently read seconds the conclusion that Island Effects have a structural source.  The paper (here) is by Yoshida, Kazanina, Pablos and Sturt (YKPS) and it explores the problem in a very clever way. Here’s a quick review.

YKPS starts from the assumption that if the problem is one of the processing complexities of islands, then any dependency into an island that is computed online (as filler/gap dependencies are) should show island like properties even if these dependencies are not products of movement. They identify forward cataphora (e.g. his₁ managers revealed that [_island the studio that notified Jeffrey Stewart₁ about the new film] selected a novel for the script) as one such dependency. YKPS shows that the indicated referential dependency is calculated online just as filler/gap dependencies are (both are very greedy in fixing the dependency). However, in contrast to movement dependencies, pronoun resolution in forward cataphora does not exhibit island effects. The argument is easy to follow and the conclusion strikes me as pretty solid, but read it and judge for yourself. What I liked about it is that it is a classic example of a typical linguistic argument form: YKPS identifies a dog that doesn’t bark. If parsing complexity is the relevant variable then it needs to explain both why some dependencies exhibit island effects and, just as importantly, why some do not. In other words, negative data counts! The absence of island effects is as much a datum as its presence is, though it is often ignored.[6] As YKPS put it:

Complexity accounts, which attribute island effects to the effect of processing complexity of the online dependency formation process, need to explain why the same complexity does not affect (my emphasis, NH) the formation of cataphoric dependencies. (17)

So, it seems to me that islands are here to stay, even if their presence in UG embarrasses minimalists.

Three points and I end. First, the argument that YKPS presents is another nice example of how psycho-techniques can be used to advance syntactic ends.  How so? Well, it is critical to YKPS’s point that forward cataphora involves the same kind of processing strategies (active filler) as do regular filler/gap dependencies that one finds in movement despite the dependencies being entirely different grammatically. This is what makes it possible to compare the two kinds of processes and conclude from their different behavior wrt islands that structural effects cannot be reduced to parsing complexity (a prima facie very reasonable hypothesis and one that might even be nice were it true!).[7] 

Second, the complexity theory of islands pertains to Subjacency Effects. The far harder problem, as I mentioned earlier, involve ECP effects. Indeed, were Subjacency Effects reduced to complexity effects, the presence of ECP effects in the very same configurations would become even more puzzling, at least to me. At any rate, both problems remain, and await a decent M analysis.

Third, let me end with some personal intellectual history. I taught a course on the old GB A’ material with Howard Lasnik this semester (a great experience, thx Howard) and have become pretty convinced that finding a way to simply recapitulate ECP and Island effects in M terms is by no means trivial.  To see this, I invite you to simply try to translate the GB theories into an M acceptable idiom. Even this is pretty hard to do, and a simple translation still leaves one short of a M acceptable account. Conclusion? This is still a really juicy research topic for the unificationally inclined, i.e. a great Minimalist research topic.

---

[1] I take GB to be the logical culmination of work that first developed as the Extended Standard Theory. Moreover, I here, again, take GB to be one of several kissing cousins, such as GPSG, LFG, HPSG.

[2] This is a bird’s eye evaluation and there are notable exceptions to this coarse generalization. Here is one very conspicuous exception: how ellipsis obviates island effects. Lasnik and Merchant have turned this into a small very productive industry. The main theoretical effect has been to make us reconsider what makes an island islandy. The ellipsis effects have revived an interpretation that has some roots in Ross, that it is not the illicit dependency that matters but the phonological realization thereof that counts.  Islands, on this view, are PF rather than syntactic effects. At any rate, this is really interesting stuff which has led us to understand Island Effects in new ways.

[3] At least if one allows D to be a phase, something some (e.g. Chomsky) has only grudgingly accepted.

[4] Rick Lewis has some nice models of this based on empirical work by Gibson.

[5] Of course, more work needs doing. For example, one needs to explain, why, ellipsis obviates these processing effects (see note 2).

[6] Note 4 indicates another bit of negative data that needs explanation on the complexity account. One might think, for example, that having to infer structure would add to complexity and thus increase the unacceptability of island violations, contrary to what we in fact find.

[7] Very reasonable indeed as witnessed by Chomsky’s extensive efforts to argue against the supposition that island effects are simple complexity effects in On Wh Movement.

Bad Data

Gary Marcus picks up on a currently popular meme about shoddy empirical hygiene in science. He points to two problems. First, there have been busts of prominent scientists (I will return to this), flurries of retractions, and the emergence of a Blog (Retraction Watch) to monitor experimental malfeasance, which, apparently, is rampant, especially in the biomedical world.  Second, it appears that experiments are all too often unreplicable.  Together, Gary seems to believe, these two problems threaten to slow down the march of scientific understanding, despite the long run self correcting nature of the enterprise. As he puts it:

In the long run, science is self-correcting…Even if nothing changed, we would eventually achieve the deep understanding that all scientists strive for.  But there is no doubt that we can get there faster if we clean up our act.

This all sounds pretty dire. Gary sites one study of fifty-three medical studies and found that forty-seven did not replicate.  And this is the non-fraudulent stuff! At the risk of not being sufficiently panicked, I cannot help wondering how big a problem this really is and whether the meme reveals more about an implicit empiricist philosophy of science than it does a serious problem threatening to appreciably slow down research.

Before saying a bit more, let me shout out very loudly that I AM NOT CONDONING MALPRACTICE AND DISHONESTY. Of course, one should not lie or steal or cheat or practice bad statistical hygiene. However, there are times when problems that look serious are not worth worrying about, or even fixing. Think about the recent Republican hyperventilation about voter registration fraud.  Fixing even legitimate concerns can have undesired side effects. I will mention one below currently raising hackles in syntax. So, stipulating that we want everyone to act honestly and experiment carefully, are the problems Gary mentions really something we should be worried about, at least in our small part of the scientific universe?

Let’s take fraud first.  In case anyone hasn’t heard, Marc Hauser was accused of fabricatingdata. His case was reviewed both by Harvard and the NIH. He was forced to resign for scientific misconduct and though neither “admit[ting] nor deny[ing]  scientific misconduct” he did accept responsibility for “all errors made within the lab.” 

This fraud case always struck me as pretty much a tempest in a teapot.  Hauser was accused of mishandling data in three published papers. Of these, one on Cognition (2002) had to be retracted. The two others reconfirmed the earlier stated results when the data analysis was redone.  In addition, it seems that Hauser also misstated results in some papers that were corrected before publication. All in all, it seems that exactly one published paper proved to be seriously defective and it was pulled.

Curiously, in my opinion, the paper that was pulled had (what to a linguist would be) a pretty boring result.  It was based on other work by Gary Marcus (Marcus et. al in Science 1999) that showed that kids could think algebraically and abstract patterns that eluded standard connectionist devices. It also would have served as an interesting counterpoint to later work byMarcus (see Marcus et. al 2009) that provided evidence that a child’s capacity to “extract abstract rules and regularities from sequences” engaged “at least one learning mechanism that is specially tuned to language.” This latter is really cool for it appears to provide evidence for a linguistically dedicated learning component. Note that the 2002 Cognition piece would have provided evidence against this juicy conclusion. It argued that Tamarins (they don’t talk!) could do the same thing. Given that this paper has been retracted, it seems that the interesting result is still viable. From the little I can gather, the retracted paper had little influence on the direction of other research (e.g. it did not stop Gary from pursuing the interesting hypothesis noted above) and I doubt that it did much to impede the march of science, or the attractiveness of the modularity of learning thesis (or lack thereof, psychologists tend to dislike these kinds of dedicated language results).

So much for fraud. More interesting is the idea that most experiments are not replicable. Gary discusses several ways in which experimentalist troll for significant results and urges, reasonably enough, that these bad practices should be avoided. He also notes that there are institutional incentives that abet these unfortunate tendencies, including only publishing experiments that succeed.  At any rate, the points he makes are reasonable, though I suspect are not the real source of the slow pace of advance in many of the sciences.  Let me explain.

From my very restricted vantage point, the main problem in a lot of “scientific” work is the absence of any (even rough) understanding of the causal architecture of the problem domain. In short, the dearth of any reasonably articulated theory.  This theoretical lacuna arises not because of an absence of enough good data, but because we often have no idea what the underlying causal processes might be or how to generalize from the individual data points we collect.  Consequently, there are many beautifully crafted experiments whose point is completely obscure. Indeed, I often get the impression that psychology is the study of methodologically flawless experiments rather than the study of mental capacities. In this context, rigor is the only game in town and generating bad data the ultimate crime.  In areas where there is a modicum of interesting theory, bad data is not nearly so serious for it is easier to detect and weed out. Eddington’s dictum explains why: Never trust an experiment until it has been verified by theory! Theory serves to filter out experimental detritus.  Where such theory is absent bad data can confuse. But then the main problem with such a discipline is not the prevalence of bad data but the absence of even weak theory. 

Is there a bad data problem in Linguistics? Some seem to think there is, and they have recently again begun to chastise generativists for their irresponsible and errant ways. It has been asserted that the lax ways in which linguists (syntacticians are the cynosure here) collect judgment data, i.e. they consult the intuitions of a handful of native speakers, generate bad data, which consequently result in very poor theories. Indeed, many of the all too frequent pronouncements about the collapse of the generative enterprise often go hand in hand with lots of clucking about the shoddy data collection that is claimed to be endemic. Gibson is the most recent avatar of this meme (though there are others) and Jon Sprouse and Diogo Almeida (S&A) the most prominent ghost busters.

In a series of papers (see here, here, here and here), S&A eviscerate these claims. They do this by retesting the “badly collected” data using more refined testing techniques borrowed from our friends in psychology. They find that the informal methods exploited by linguists are more than good enough. Indeed comparing them to what we typically find in psych work, they are unbelievably reliable (95% of the data is reliably replicable, an unheard level of reliability in the mental sciences) and very sensitive (it takes only a few sentences asked of a few judgers to get this very reliable data). So the shoddy methods we know and love are more than good enough for most of what we do, at least if the more careful experimental methods that Gibson urges are the touchstones of adequacy. This does not mean to say that more careful methods may not be appropriate in some circumstances and for investigating different kinds of problems (c.f. Sprouse’s more recent work discusses examples. Not yet written up so try to go to a talk if he is speaking at a venue near you). These more prissy methods may be useful in the right contexts and linguists should not shy away from using them when appropriate.  However, S&A have demonstrated quite conclusively that the informal methods that are quick and easy to use (no small virtues I might add) are perfectly adequate, indeed surprisingly powerful, and that the theory developed using these methods if inadequate are not inadequate because the data the theory addresses is defective.

There is a popular picture of science that owes a lot to empiricist epistemology: scientists carefully collect data, cautiously develop theories to explain this data, extend these theories by yet more refined methods of data collection and build theories on these purer data points.  It is easy to understand the danger posed by bad data given this conception.  It pollutes the process, adds dirt to the gears of science thereby reducing its efficiency and threatening to derail it. However, this picture is false. Data IS important, but mainly for testing theory and even then how data and theory come together is a very complicated matter.  I am partial to the version of the scientific method urged by Percy Bridgeman: “Use your noodle and no holds barred.”  If there is some reasonable theory for your noodle to work with the bad data problem will annoy but not otherwise impede progress.  In my view, the most serious impediments are not hygienic. Rather, most of the time we just don’t have the foggiest idea what’s going on, and, sadly, that has no quick fix.